1	Announcements more panorama slots available now you can sign up for a 2^nd time if you’d like
2	Motion Estimation Today’s Readings Szeliski Chapters 7.1, 7.2, 7.4
3	Why estimate motion? Lots of uses Track object behavior Correct for camera jitter (stabilization) Align images (mosaics) 3D shape reconstruction Special effects
4	Motion estimation Input: sequence of images Output: point correspondence Feature correspondence: “Feature Tracking” we’ve seen this already (e.g., SIFT) can modify this to be more accurate/efficient if the images are in sequence (e.g., video) Pixel (dense) correspondence: “Optical Flow” today’s lecture
5	Optical flow
6	Problem definition: optical flow How to estimate pixel motion from image H to image I?
7	Optical flow constraints (grayscale images) Let’s look at these constraints more closely
8	Optical flow equation Combining these two equations
9	Optical flow equation Combining these two equations
10	Optical flow equation Q: how many unknowns and equations per pixel?
11	Aperture problem
12	Aperture problem
13	Solving the aperture problem Basic idea: assume motion field is smooth Horn & Schunk: add smoothness term Lucas & Kanade: assume locally constant motion pretend the pixel’s neighbors have the same (u,v) Many other methods exist. Here’s an overview: S. Baker, M. Black, J. P. Lewis, S. Roth, D. Scharstein, and R. Szeliski. A database and evaluation methodology for optical flow. In Proc. ICCV, 2007 http://vision.middlebury.edu/flow/
14	Lucas-Kanade flow How to get more equations for a pixel? Basic idea: impose additional constraints most common is to assume that the flow field is smooth locally one method: pretend the pixel’s neighbors have the same (u,v) If we use a 5x5 window, that gives us 25 equations per pixel!
15	RGB version How to get more equations for a pixel? Basic idea: impose additional constraints most common is to assume that the flow field is smooth locally one method: pretend the pixel’s neighbors have the same (u,v) If we use a 5x5 window, that gives us 25*3 equations per pixel!
16	Lucas-Kanade flow Prob: we have more equations than unknowns
17	Conditions for solvability Optimal (u, v) satisfies Lucas-Kanade equation
18	Observation This is a two image problem BUT Can measure sensitivity by just looking at one of the images! This tells us which pixels are easy to track, which are hard very useful for feature tracking...
19	Errors in Lucas-Kanade What are the potential causes of errors in this procedure? Suppose A^TA is easily invertible Suppose there is not much noise in the image
20	Improving accuracy Recall our small motion assumption
21	Iterative Refinement
22	Revisiting the small motion assumption Is this motion small enough? Probably not—it’s much larger than one pixel (2^nd order terms dominate) How might we solve this problem?
23	Reduce the resolution!
24	Coarse-to-fine optical flow estimation
25	Coarse-to-fine optical flow estimation
26	Optical flow result
27	Robust methods L-K minimizes a sum-of-squares error metric least squares techniques overly sensitive to outliers
28	Robust optical flow Robust Horn & Schunk Robust Lucas-Kanade
29	Benchmarking optical flow algorithms Middlebury flow page http://vision.middlebury.edu/flow/
30	Discussion: features vs. flow? Features are better for: Flow is better for:
31	Advanced topics Particles: combining features and flow Peter Sand et al. http://rvsn.csail.mit.edu/pv/ State-of-the-art feature tracking/SLAM Georg Klein et al. http://www.robots.ox.ac.uk/~gk/