The application of MS Kinect is popular since it provides the user with color images with depth in real time. Related research in computer graphics has been focused on getting structured information out of the raw data. Although this is the first application of face reconstruction using Kinect, the ideas behind it are not as new: divide and conquer, linear shape model, and global smoothing. As long as it is done right, this application definitely boosts the value of Kinect.
As pointed out by the authors, the main challenges of recovering a smooth, high-vertex-count 3D mesh with facial structures are noise, corruption, low resolution, and large deformations. Instead of facing all of these difficulties at one time, it is wise to divide the input data into patches and carry out the best protocol locally before they are put together again. This is where active appearance model (AAM) face tracking comes into play. You may notice that the difficulty of modeling when there are large deformations is alleviated due to the low degree of deformation freedom that each local patch has. Local sparse coding introduces a linear model to reconstruct each patch, and because of this model, occlusion and noise are handled properly. Another interesting part of this work is that it not only recovers the facial geometry, but also builds the vertex correspondences between the result mesh and each set of training data. This makes final global smoothing possible by deforming a training mesh onto the stitched patches.
The experiments show some improvement over existing methods, especially when input data is challenging. They also generated some data using traditional structured lights for quantitative analysis, which gives readers an idea of how far Kinect can go.
Although the authors want to make further improvements to recover subtle facial details, this seems to be the end of the story until new hardware becomes available. Considering the processing time is over one minute for this method, improving its efficiency without sacrificing its good properties might be more useful.
