Announcements

Project1 artifact reminder

counts towards your grade

Demos this Thursday, 12-2:30

sign up!

Extra office hours this week

David (T 12-1, W/F 1:30-2:30)

Jiwon (T 2:30-3:30, W 3:30-4:30)

Steve (T 1:30-2:30)

Q’s from last lecture

convolution and derivatives

laplacian scale factor

Motion Estimation

Today’s Readings

Trucco & Verri, 8.3 – 8.4 (skip 8.3.3, read only top half of p. 199)

Numerical Recipes (Newton-Raphson), 9.4 (first four pages)

http://www.ulib.org/webRoot/Books/Numerical_Recipes/bookcpdf/c9-4.pdf

Why estimate motion?

Lots of uses

Track object behavior

Correct for camera jitter (stabilization)

Align images (mosaics)

3D shape reconstruction

Special effects

Optical flow

Problem definition: optical flow

How to estimate pixel motion from image H to image I?

Optical flow constraints (grayscale images)

Let’s look at these constraints more closely

Optical flow equation

Combining these two equations

Optical flow equation

Q: how many unknowns and equations per pixel?

Aperture problem

Solving the aperture problem

How to get more equations for a pixel?

Basic idea: impose additional constraints

most common is to assume that the flow field is smooth locally

one method: pretend the pixel’s neighbors have the same (u,v)

If we use a 5x5 window, that gives us 25 equations per pixel!

RGB version

How to get more equations for a pixel?

Basic idea: impose additional constraints

most common is to assume that the flow field is smooth locally

one method: pretend the pixel’s neighbors have the same (u,v)

If we use a 5x5 window, that gives us 25*3 equations per pixel!

Lukas-Kanade flow

Prob: we have more equations than unknowns

Conditions for solvability

Optimal (u, v) satisfies Lucas-Kanade equation

Eigenvectors of A^TA

Edge

Low texture region

High textured region

Observation

This is a two image problem BUT

Can measure sensitivity by just looking at one of the images!

This tells us which pixels are easy to track, which are hard

very useful later on when we do feature tracking...

Errors in Lukas-Kanade

What are the potential causes of errors in this procedure?

Suppose A^TA is easily invertible

Suppose there is not much noise in the image

Improving accuracy

Recall our small motion assumption

Iterative Refinement

Revisiting the small motion assumption

Is this motion small enough?

Probably not—it’s much larger than one pixel (2^nd order terms dominate)

How might we solve this problem?

Reduce the resolution!

Coarse-to-fine optical flow estimation

Optical flow result

Motion tracking

Suppose we have more than two images

How to track a point through all of the images?

Feature Detection

Tracking features

Feature tracking

Compute optical flow for that feature for each consecutive H, I

Handling large motions

L-K requires small motion

If the motion is much more than a pixel, use discrete search instead

Given feature window W in H, find best matching window in I

Minimize sum squared difference (SSD) of pixels in window

Solve by doing a search over a specified range of (u,v) values

this (u,v) range defines the search window

Tracking Over Many Frames

Feature tracking with m frames

Select features in first frame

Given feature in frame i, compute position in i+1

Select more features if needed

i = i + 1

If i < m, go to step 2

Incorporating Dynamics

Idea

Can get better performance if we know something about the way points move

Most approaches assume constant velocity

or constant acceleration

Use above to predict position in next frame, initialize search

Feature tracking demo

http://www.toulouse.ca/?/CamTracker/?/CamTracker/FeatureTracking.html

Image alignment

Goal: estimate single (u,v) translation for entire image

Easier subcase: solvable by pyramid-based Lukas-Kanade

Summary

Things to take away from this lecture

Optical flow problem definition

Aperture problem and how it arises

Assumptions

Brightness constancy, small motion, smoothness

Derivation of optical flow constraint equation

Lukas-Kanade equation

Derivation

Conditions for solvability

meanings of eigenvalues and eigenvectors

Iterative refinement

Newton’s method

Coarse-to-fine flow estimation

Feature tracking

Harris feature detector

L-K vs. discrete search method

Tracking over many frames

Prediction using dynamics

Applications

MPEG video compression

Image alignment


	Project1 artifact reminder
		counts towards your grade
	Demos this Thursday, 12-2:30
		sign up!
	Extra office hours this week
		David (T 12-1, W/F 1:30-2:30)
		Jiwon (T 2:30-3:30, W 3:30-4:30)
		Steve (T 1:30-2:30)
	Q’s from last lecture
		convolution and derivatives
		laplacian scale factor


Today’s Readings
	Trucco & Verri, 8.3 – 8.4 (skip 8.3.3, read only top half of p. 199)
	Numerical Recipes (Newton-Raphson), 9.4 (first four pages)
		http://www.ulib.org/webRoot/Books/Numerical_Recipes/bookcpdf/c9-4.pdf


	Lots of uses
		Track object behavior
		Correct for camera jitter (stabilization)
		Align images (mosaics)
		3D shape reconstruction
		Special effects


How to get more equations for a pixel?
	Basic idea: impose additional constraints
		most common is to assume that the flow field is smooth locally
		one method: pretend the pixel’s neighbors have the same (u,v)
			If we use a 5x5 window, that gives us 25 equations per pixel!


This is a two image problem BUT
	Can measure sensitivity by just looking at one of the images!
	This tells us which pixels are easy to track, which are hard
		very useful later on when we do feature tracking...


	What are the potential causes of errors in this procedure?
		Suppose A^TA is easily invertible
		Suppose there is not much noise in the image


	Is this motion small enough?
		Probably not—it’s much larger than one pixel (2^nd order terms dominate)
		How might we solve this problem?


	Suppose we have more than two images
		How to track a point through all of the images?


	Feature tracking
		Compute optical flow for that feature for each consecutive H, I


L-K requires small motion
	If the motion is much more than a pixel, use discrete search instead






	Given feature window W in H, find best matching window in I
	Minimize sum squared difference (SSD) of pixels in window




	Solve by doing a search over a specified range of (u,v) values
		this (u,v) range defines the search window


	Feature tracking with m frames
		Select features in first frame
		Given feature in frame i, compute position in i+1
		Select more features if needed
		i = i + 1
		If i < m, go to step 2


Idea
	Can get better performance if we know something about the way points move
	Most approaches assume constant velocity



		or constant acceleration



	Use above to predict position in next frame, initialize search


	http://www.toulouse.ca/?/CamTracker/?/CamTracker/FeatureTracking.html


	Goal: estimate single (u,v) translation for entire image
		Easier subcase: solvable by pyramid-based Lukas-Kanade


Things to take away from this lecture
	Optical flow problem definition
	Aperture problem and how it arises
	Assumptions
		Brightness constancy, small motion, smoothness
	Derivation of optical flow constraint equation
	Lukas-Kanade equation
		Derivation
		Conditions for solvability
		meanings of eigenvalues and eigenvectors
	Iterative refinement
		Newton’s method
		Coarse-to-fine flow estimation
	Feature tracking
		Harris feature detector
		L-K vs. discrete search method
		Tracking over many frames
		Prediction using dynamics
	Applications
		MPEG video compression
		Image alignment