Motion Estimation

Today’s Readings

Trucco & Verri, 8.3 – 8.4 (skip 8.3.3, read only top half of p. 199)

Numerical Recipes (Newton-Raphson), 9.4 (first four pages)

http://www.library.cornell.edu/nr/bookcpdf/c9-4.pdf

Why estimate motion?

Lots of uses

Track object behavior

Correct for camera jitter (stabilization)

Align images (mosaics)

3D shape reconstruction

Special effects

Optical flow

Problem definition: optical flow

How to estimate pixel motion from image H to image I?

Optical flow constraints (grayscale images)

Let’s look at these constraints more closely

Optical flow equation

Combining these two equations

Optical flow equation

Q: how many unknowns and equations per pixel?

Aperture problem

Solving the aperture problem

How to get more equations for a pixel?

Basic idea: impose additional constraints

most common is to assume that the flow field is smooth locally

one method: pretend the pixel’s neighbors have the same (u,v)

If we use a 5x5 window, that gives us 25 equations per pixel!

RGB version

How to get more equations for a pixel?

Basic idea: impose additional constraints

most common is to assume that the flow field is smooth locally

one method: pretend the pixel’s neighbors have the same (u,v)

If we use a 5x5 window, that gives us 25*3 equations per pixel!

Lukas-Kanade flow

Prob: we have more equations than unknowns

Conditions for solvability

Optimal (u, v) satisfies Lucas-Kanade equation

Eigenvectors of A^TA

Edge

Low texture region

High textured region

Observation

This is a two image problem BUT

Can measure sensitivity by just looking at one of the images!

This tells us which pixels are easy to track, which are hard

very useful later on when we do feature tracking...

Errors in Lukas-Kanade

What are the potential causes of errors in this procedure?

Suppose A^TA is easily invertible

Suppose there is not much noise in the image

Improving accuracy

Recall our small motion assumption

Iterative Refinement

Revisiting the small motion assumption

Is this motion small enough?

Probably not—it’s much larger than one pixel (2^nd order terms dominate)

How might we solve this problem?

Reduce the resolution!

Coarse-to-fine optical flow estimation

Optical flow result

Motion tracking

Suppose we have more than two images

How to track a point through all of the images?

Feature Detection

Tracking features

Feature tracking

Compute optical flow for that feature for each consecutive H, I

Handling large motions

L-K requires small motion

If the motion is much more than a pixel, use discrete search instead

Given feature window W in H, find best matching window in I

Minimize sum squared difference (SSD) of pixels in window

Solve by doing a search over a specified range of (u,v) values

this (u,v) range defines the search window

Tracking Over Many Frames

Feature tracking with m frames

Select features in first frame

Given feature in frame i, compute position in i+1

Select more features if needed

i = i + 1

If i < m, go to step 2

Incorporating Dynamics

Idea

Can get better performance if we know something about the way points move

Most approaches assume constant velocity

or constant acceleration

Use above to predict position in next frame, initialize search

Feature tracking demo

Image alignment

Goal: estimate single (u,v) translation for entire image

Easier subcase: solvable by pyramid-based Lukas-Kanade

Application: Rotoscoping (demo)


Today’s Readings
	Trucco & Verri, 8.3 – 8.4 (skip 8.3.3, read only top half of p. 199)
	Numerical Recipes (Newton-Raphson), 9.4 (first four pages)
		http://www.library.cornell.edu/nr/bookcpdf/c9-4.pdf


	Lots of uses
		Track object behavior
		Correct for camera jitter (stabilization)
		Align images (mosaics)
		3D shape reconstruction
		Special effects


How to get more equations for a pixel?
	Basic idea: impose additional constraints
		most common is to assume that the flow field is smooth locally
		one method: pretend the pixel’s neighbors have the same (u,v)
			If we use a 5x5 window, that gives us 25 equations per pixel!


This is a two image problem BUT
	Can measure sensitivity by just looking at one of the images!
	This tells us which pixels are easy to track, which are hard
		very useful later on when we do feature tracking...


	What are the potential causes of errors in this procedure?
		Suppose A^TA is easily invertible
		Suppose there is not much noise in the image


	Is this motion small enough?
		Probably not—it’s much larger than one pixel (2^nd order terms dominate)
		How might we solve this problem?


	Suppose we have more than two images
		How to track a point through all of the images?


	Feature tracking
		Compute optical flow for that feature for each consecutive H, I


L-K requires small motion
	If the motion is much more than a pixel, use discrete search instead






	Given feature window W in H, find best matching window in I
	Minimize sum squared difference (SSD) of pixels in window




	Solve by doing a search over a specified range of (u,v) values
		this (u,v) range defines the search window


	Feature tracking with m frames
		Select features in first frame
		Given feature in frame i, compute position in i+1
		Select more features if needed
		i = i + 1
		If i < m, go to step 2


Idea
	Can get better performance if we know something about the way points move
	Most approaches assume constant velocity



		or constant acceleration



	Use above to predict position in next frame, initialize search


	Goal: estimate single (u,v) translation for entire image
		Easier subcase: solvable by pyramid-based Lukas-Kanade