1	Announcements Project 2 artifacts—vote now!! Project 3 questions? Start thinking about final project ideas, partners
2	Recovering 3D from images So far, we’ve relied on a human to provide depth cues parallel lines, reference points, etc. How might we do this automatically? What cues in the image provide 3D information?
3	Visual cues Shading
4	Visual cues Shading Texture
5	Visual cues Shading Texture Focus
6	Visual cues Shading Texture Focus Motion
7	Visual cues Shading Texture Focus Motion
8	Stereo Readings Forsyth, Chapters 10.1, 11
9
10
11
12
13
14	Stereograms online UCR stereographs http://www.cmp.ucr.edu/site/exhibitions/stereo/ The Art of Stereo Photography http://www.photostuff.co.uk/stereo.htm History of Stereo Photography http://www.rpi.edu/~ruiz/stereo_history/text/historystereog.html Double Exposure http://home.centurytel.net/s3dcor/index.html Stereo Photography http://www.shortcourses.com/book01/chapter09.htm 3D Photography links http://www.studyweb.com/links/5243.html National Stereoscopic Association http://204.248.144.203/3dLibrary/welcome.html Books on Stereo Photography http://userwww.sfsu.edu/~hl/3d.biblio.html
15	Stereo
16	Stereo
17	Stereo correspondence Determine Pixel Correspondence Pairs of points that correspond to same scene point
18	Fundamental matrix Let p be a point in left image, p’ in right image Epipolar relation p maps to epipolar line l’ p’ maps to epipolar line l Epipolar mapping described by a 3x3 matrix F It follows that
19	Fundamental matrix This matrix F is called the “Essential Matrix” when image intrinsic parameters are known the “Fundamental Matrix” more generally (uncalibrated case) Can solve for F from point correspondences Each (p, p’) pair gives one linear equation in entries of F 8 points give enough to solve for F (8-point algorithm) see readings (Forsyth chapter 10.1) for more on this
20	Stereo image rectification
21	Stereo image rectification Image Reprojection reproject image planes onto common plane parallel to line between optical centers a homography (3x3 transform) applied to both input images pixel motion is horizontal after this transformation C. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo Vision. IEEE Conf. Computer Vision and Pattern Recognition, 1999.
22	Stereo matching algorithms Match Pixels in Conjugate Epipolar Lines Assume brightness constancy This is a tough problem Numerous approaches dynamic programming [Baker 81,Ohta 85] smoothness functionals more images (trinocular, N-ocular) [Okutomi 93] graph cuts [Boykov 00] A good survey and evaluation: http://www.middlebury.edu/stereo/
23	Your basic stereo algorithm
24	Window size Smaller window Larger window
25	Stereo results Data from University of Tsukuba Similar results on other images without ground truth
26	Results with window search
27	Better methods exist...
28	Stereo as energy minimization Matching Cost Formulated as Energy “data” term penalizing bad matches “neighborhood term” encouraging spatial smoothness
29	Stereo as a graph problem [Boykov, 1999] Pixels
30	Graph definition Initial state Each pixel connected to it’s immediate neighbors Each disparity label connected to all of the pixels
31	Stereo matching by graph cuts Graph Cut Delete enough edges so that each pixel is (transitively) connected to exactly one label node Cost of a cut: sum of deleted edge weights Finding min cost cut equivalent to finding global minimum of the energy function
32	Computing a multiway cut With two labels: classical min-cut problem Solvable by standard network flow algorithms polynomial time in theory, nearly linear in practice More than 2 labels: NP-hard [Dahlhaus et al., STOC ‘92] But efficient approximation algorithms exist Within a factor of 2 of optimal Computes local minimum in a strong sense even very large moves will not improve the energy Yuri Boykov, Olga Veksler and Ramin Zabih, Fast Approximate Energy Minimization via Graph Cuts, International Conference on Computer Vision, September 1999. Basic idea reduce to a series of 2-way-cut sub-problems, using one of: swap move: pixels with label l1 can change to l2, and vice-versa expansion move: any pixel can change it’s label to l1
33	Depth from disparity
34	Image-based rendering Render new views from raw disparity S. M. Seitz and C. R. Dyer, View Morphing, Proc. SIGGRAPH 96, 1996, pp. 21-30. L. McMillan and G. Bishop. Plenoptic Modeling: An Image-Based Rendering System, Proc. of SIGGRAPH 95, 1995, pp. 39-46.
35	Stereo reconstruction pipeline Steps Calibrate cameras Rectify images Compute disparity Estimate depth
36	Stereo matching Features vs. Pixels? Do we extract features prior to matching?
37	Active stereo with structured light Project “structured” light patterns onto the object simplifies the correspondence problem
38	Active stereo with structured light
39	Laser scanning Optical triangulation Project a single stripe of laser light Scan it across the surface of the object This is a very precise version of structured light scanning
40
41	Real-time stereo Used for robot navigation (and other tasks) Several software-based real-time stereo techniques have been developed (most based on simple discrete search)