Announcements

more panorama slots available now

you can sign up for a 2^nd time if you’d like

Motion Estimation

Today’s Readings

Szeliski Chapters 7.1, 7.2, 7.4

Why estimate motion?

Lots of uses

Track object behavior

Correct for camera jitter (stabilization)

Align images (mosaics)

3D shape reconstruction

Special effects

Motion estimation

Input: sequence of images

Output: point correspondence

Feature correspondence: “Feature Tracking”

we’ve seen this already (e.g., SIFT)

can modify this to be more accurate/efficient if the images are in sequence (e.g., video)

Pixel (dense) correspondence: “Optical Flow”

today’s lecture

Optical flow

Problem definition: optical flow

How to estimate pixel motion from image H to image I?

Optical flow constraints (grayscale images)

Let’s look at these constraints more closely

Optical flow equation

Combining these two equations

Optical flow equation

Q: how many unknowns and equations per pixel?

Aperture problem

Solving the aperture problem

Basic idea: assume motion field is smooth

Horn & Schunk: add smoothness term

Lucas & Kanade: assume locally constant motion

pretend the pixel’s neighbors have the same (u,v)

Many other methods exist. Here’s an overview:

S. Baker, M. Black, J. P. Lewis, S. Roth, D. Scharstein, and R. Szeliski. A database and evaluation methodology for optical flow. In Proc. ICCV, 2007

http://vision.middlebury.edu/flow/

Lucas-Kanade flow

How to get more equations for a pixel?

Basic idea: impose additional constraints

most common is to assume that the flow field is smooth locally

one method: pretend the pixel’s neighbors have the same (u,v)

If we use a 5x5 window, that gives us 25 equations per pixel!

RGB version

How to get more equations for a pixel?

Basic idea: impose additional constraints

most common is to assume that the flow field is smooth locally

one method: pretend the pixel’s neighbors have the same (u,v)

If we use a 5x5 window, that gives us 25*3 equations per pixel!

Lucas-Kanade flow

Prob: we have more equations than unknowns

Conditions for solvability

Optimal (u, v) satisfies Lucas-Kanade equation

Observation

This is a two image problem BUT

Can measure sensitivity by just looking at one of the images!

This tells us which pixels are easy to track, which are hard

very useful for feature tracking...

Errors in Lucas-Kanade

What are the potential causes of errors in this procedure?

Suppose A^TA is easily invertible

Suppose there is not much noise in the image

Improving accuracy

Recall our small motion assumption

Iterative Refinement

Revisiting the small motion assumption

Is this motion small enough?

Probably not—it’s much larger than one pixel (2^nd order terms dominate)

How might we solve this problem?

Reduce the resolution!

Coarse-to-fine optical flow estimation

Optical flow result

Robust methods

L-K minimizes a sum-of-squares error metric

least squares techniques overly sensitive to outliers

Robust optical flow

Robust Horn & Schunk

Robust Lucas-Kanade

Benchmarking optical flow algorithms

Middlebury flow page

http://vision.middlebury.edu/flow/

Discussion: features vs. flow?

Features are better for:

Flow is better for:

Advanced topics

Particles: combining features and flow

Peter Sand et al.

http://rvsn.csail.mit.edu/pv/

State-of-the-art feature tracking/SLAM

Georg Klein et al.

http://www.robots.ox.ac.uk/~gk/


	more panorama slots available now
		you can sign up for a 2^nd time if you’d like


	Lots of uses
		Track object behavior
		Correct for camera jitter (stabilization)
		Align images (mosaics)
		3D shape reconstruction
		Special effects


	Input: sequence of images
	Output: point correspondence

	Feature correspondence: “Feature Tracking”
		we’ve seen this already (e.g., SIFT)
		can modify this to be more accurate/efficient if the images are in sequence (e.g., video)

	Pixel (dense) correspondence: “Optical Flow”
		today’s lecture


	Basic idea: assume motion field is smooth

	Horn & Schunk: add smoothness term


	Lucas & Kanade: assume locally constant motion
		pretend the pixel’s neighbors have the same (u,v)


	Many other methods exist. Here’s an overview:
		S. Baker, M. Black, J. P. Lewis, S. Roth, D. Scharstein, and R. Szeliski. A database and evaluation methodology for optical flow. In Proc. ICCV, 2007
		http://vision.middlebury.edu/flow/


How to get more equations for a pixel?
	Basic idea: impose additional constraints
		most common is to assume that the flow field is smooth locally
		one method: pretend the pixel’s neighbors have the same (u,v)
			If we use a 5x5 window, that gives us 25 equations per pixel!


This is a two image problem BUT
	Can measure sensitivity by just looking at one of the images!
	This tells us which pixels are easy to track, which are hard
		very useful for feature tracking...


	What are the potential causes of errors in this procedure?
		Suppose A^TA is easily invertible
		Suppose there is not much noise in the image


	Is this motion small enough?
		Probably not—it’s much larger than one pixel (2^nd order terms dominate)
		How might we solve this problem?


	L-K minimizes a sum-of-squares error metric
		least squares techniques overly sensitive to outliers


	Particles: combining features and flow
		Peter Sand et al.
		http://rvsn.csail.mit.edu/pv/

	State-of-the-art feature tracking/SLAM
		Georg Klein et al.
		http://www.robots.ox.ac.uk/~gk/