Interactive 3D reconstruction from video.

Abstract

This thesis explores several approaches to 3D reconstruction from video sequences in which the reconstruction process is aided by information about the scene provided interactively by a user. This user-supplied information may describe, for example, the types of objects in the scene and their positions, the boundaries of faces of an object in one view, geometric properties of an object, or geometry which is not seen in the video sequence. By incorporating this information into the modelling process, we can reconstruct scenes which would be difficult or impossible to reconstruct with a fully automated process due to elements of the scene having minimal texture, or translucent or reflective surfaces, or due to significant parts of the scene being occluded or poorly visible in all views. These interactive methods allow the user to reconstruct the desired parts of the scene to the desired level of detail, with automated processing minimising the required interaction. We first propose a method in which individual objects in a video sequence are reconstructed by fitting pre-defined model types selected by a user. A novel fitting process is used to efficiently evaluate and optimise models sampled from the large space of possible models. Models are evaluated using a novel combination of user-supplied information, 2D image information, and 3D point cloud data recovered with a Structure from Motion process. A method is also presented for polygonal modelling of objects in video sequences. This method allows the user to define the faces of the model in a single frame through an intuitive sketch-based interface. An automated process generates a 3D model from this set of 2D faces. Interactive techniques are also described for generating a complete model from a partial model of an object, for fitting primitive shapes, and for incorporating geometric constraints into the modelling process. We demonstrate the use of this polygonal modelling method for rapidly generating models in Augmented Reality environments. We then describe an additional method for Augmented Reality applications in which the camera is used as the input device. In this method interaction with the camera is used first to select an object in a scene, and then to provide sufficient views of the object for a complete reconstruction.