Generating 3D models of historical photographs, paintings, and line drawings using discriminative visual elements that align automatically is a recent trend in computer vision applications. These studies enhance the visualization of details, turning a 2D representation of an architectural site into a 3D model.

The authors present an automatic alignment technique that “aligns arbitrary 2D depictions of an architectural site with a 3D model of the site,” based on discriminative visual elements. According to the paper, they found that “appearance and scene structure in 2D depictions can be very different from the appearance and geometry of the 3D model ... due to the specific rendering style, drawing error, age, lighting, or change of seasons.” The authors redepict complex 3D models using a set of visually distinct mid-level scene elements that are possible matches “in 2D depictions of the scene despite large variations in rendering style (for example, watercolor, sketch, historical photograph) and structural changes (for example, missing scene parts, large occluders) of the scene.” The proposed approach adds to the automatic computational rephotography of “historical paintings and photographs with respect to a 3D model” approximating viewpoints.

Experimental data include a set of human-generated 3D models from Trimble 3D Warehouse, and “a 3D model of San Marco’s Square that was reconstructed from a set of photographs using dense multiview stereo.” A total of 85 historical photographs and 252 nonphotographic depictions of the sites--drawings (60 images), engravings (45 images), and paintings (147 images)--were collected from the Internet. The authors use Amazon Mechanical Turk to conduct a user study to validate the proposed approach, comparing it against four baseline approaches. In the set of sampled views, the authors observe failed cases with regard to large-scale symmetries, locally confusing image structures, and false viewpoint depiction.

The computational cost estimation for both offline and online discriminative visual element detection stages makes the paper worth reading. The authors suggest that their approach could lead to “the possibility of efficient indexing for visual search of paintings and historical photographs (for example, via hashing of the [histogram of oriented gradients (HOG)] features ..., or automatic fitting of complex nonperspective models used in historical imagery.”