To segment a static foreground object from a highly cluttered background in an image can be tricky. But how about using an image sequence? Is it true that the more images we have, the better we can do? This paper shows that a considerable speed and quality improvement can be achieved when thousands of input frames are available to build a light field with high spatio-angular resolution.

The main idea is motivated by the observation that when the light field has a good resolution, the smooth trajectory of a point in the field can be used to distinguish the foreground against the background. Nevertheless, processing this big sequence data properly and efficiently requires several local-to-global simplification and consolidation steps. First, the trajectory of each point in the scene is calculated to give its probability of being the foreground. In order to approximate the locations of foreground and the background, a pair of proxy objects as simple as two planes with known distances from the camera has to be placed in the scene before capturing the light field, so that the trajectory of a point can be compared with that of the proxies. An additional proxy between the foreground and background is also suggested in the paper. The trajectory of a point is just the gradient of the local light field, which can be reused to speed up when the motion between two consecutive frames remains the same. Special care must be given when an image edge is aligned with the camera motion, in which case three confidence measures are suggested to reduce ambiguity. The trajectory comparison gives an initial set of the foreground with high confidence in some regions. A local refinement is then carried out to propagate this initial set to low-gradient areas. Finally, a global gathering refines the segmentation by combining the per-image segmentations from the previous step to ensure global geometrical consistency.

More details about the settings and performance of the algorithms can be found in section 4, which also contains many very illustrative visual examples that clearly demonstrate the superiority of the author’s techniques. To answer my first question, it is worth noting that although increasing the number of inputs can improve the segmentation quality, the underlying heuristics in the refinement steps can still break and produce artifacts and missing foreground as discussed in the paper. We also must remember that we are limited in the ways we can capture more images in a natural environment.

Finally, I had difficulty reading some figures that could have been more narrative. Some additional visual illustrations and more text explanations could have helped.