Video matting is the process of pulling a high-quality alpha matte and foreground from a video sequence. Current techniques require either a known background (e.g., a blue screen) or extensive user interaction (e.g., to specify known foreground and background elements). The matting problem is generally under-constrained, since not enough information has been collected at capture time. We propose a novel, fully autonomous method for pulling a matte using multiple synchronized video streams that share a point of view but differ in their plane of focus. The solution is obtained by directly minimizing the error in filter-based image formation equations, which are over-constrained by our rich data stream. Our system solves the fully dynamic video matting problem without user assistance: both the foreground and background may be high frequency and have dynamic content, the foreground may resemble the background, and the scene is lit by natural (as opposed to polarized or collimated) illumination.

William Yerazunis and John Barnwell of MERL worked with us to design and manufacture the camera. We thank Shree Nayar (Columbia), Doug Roble (Digital Domain), Bill Freeman (MIT), and Michael Black (Brown) for their advice, and NVIDIA Corporation for Morgan's fellowship. The novel backgrounds are photographs by Shriram Krishnamurthi.

@article{mcguire05matting,
 author = {Morgan McGuire and Wojciech Matusik and Hanspeter Pfister and \
           John F. Hughes and Fr\'{e}do Durand},
 title = {Defocus video matting},
 journal = {ACM Trans. Graph.},
 volume = {24},
 number = {3},
 year = {2005},
 issn = {0730-0301},
 pages = {567--576},
 doi = {http://doi.acm.org/10.1145/1073204.1073231},
 publisher = {ACM Press},
 address = {New York, NY, USA},
 URL = {http://www.cs.brown.edu/people/morgan/DefocusVideoMatting}
 }