Announcements

Project 2 due today

Project 3 out today

help session today

Recognition

Readings

C. Bishop, “Neural Networks for Pattern Recognition”, Oxford University Press, 1998, Chapter 1.

Forsyth and Ponce, 22.3 (eigenfaces)

Recognition problems

What is it?

Object detection

Who is it?

Recognizing identity

What are they doing?

Activities

All of these are classification problems

Choose one class from a list of possible candidates

Face detection

How to tell if a face is present?

One simple method: skin detection

Skin pixels have a distinctive range of colors

Corresponds to region(s) in RGB color space

for visualization, only R and G components are shown above

Skin detection

Learn the skin region from examples

Manually label pixels in one or more “training images” as skin or not skin

Plot the training data in RGB space

skin pixels shown in orange, non-skin pixels shown in blue

some skin pixels may be outside the region, non-skin pixels inside. Why?

Skin classification techniques

Probability

Basic probability

X is a random variable

P(X) is the probability that X achieves a certain value

or

Conditional probability: P(X | Y)

probability of X given that we already know Y

Probabilistic skin classification

Now we can model uncertainty

Each pixel has a probability of being skin or not skin

Learning conditional PDF’s

We can calculate P(R | skin) from a set of training images

It is simply a histogram over the pixels in the training images

each bin R_i contains the proportion of skin pixels with color R_i

Bayes rule

In terms of our problem:

Bayesian estimation

Bayesian estimation

Goal is to choose the label (skin or ~skin) that maximizes the posterior

this is called Maximum A Posteriori (MAP) estimation

Skin detection results

General classification

This same procedure applies in more general circumstances

More than two classes

More than one dimension

Linear subspaces

Classification can be expensive

Must either search (e.g., nearest neighbors) or store large PDF’s

Dimensionality reduction

Linear subspaces

Principal component analysis

Suppose each data point is N-dimensional

Same procedure applies:

The eigenvectors of A define a new coordinate system

eigenvector with largest eigenvalue captures the most variation among training vectors x

eigenvector with smallest eigenvalue has least variation

We can compress the data by only using the top few eigenvectors

corresponds to choosing a “linear subspace”

represent points on a line, plane, or “hyper-plane”

these eigenvectors are known as the principal components

The space of faces

An image is a point in a high dimensional space

An N x M image is a point in R^NM

We can define vectors in this space as we did in the 2D case

Dimensionality reduction

The set of faces is a “subspace” of the set of images

Suppose it is K dimensional

We can find the best subspace using PCA

This is like fitting a “hyper-plane” to the set of faces

spanned by vectors v₁, v₂, ..., v_K

any face

Eigenfaces

PCA extracts the eigenvectors of A

Gives a set of vectors v₁, v₂, v₃, ...

Each one of these vectors is a direction in face space

what do these look like?

Projecting onto the eigenfaces

The eigenfaces v₁, ..., v_K span the space of faces

A face is converted to eigenface coordinates by

Recognition with eigenfaces

Algorithm

Process the image database (set of images with labels)

Run PCA—compute eigenfaces

Calculate the K coefficients for each image

Given a new image (to be recognized) x, calculate K coefficients

Detect if x is a face

If it is a face, who is it?

Choosing the dimension K

How many eigenfaces to use?

Look at the decay of the eigenvalues

the eigenvalue tells you the amount of variance “in the direction” of that eigenface

ignore eigenfaces with low variance

Issues: metrics

What’s the best way to compare images?

need to define appropriate features

depends on goal of recognition task

Metrics

Lots more feature types that we haven’t mentioned

moments, statistics

metrics: Earth mover’s distance, ...

edges, curves

metrics: Hausdorff, shape context, ...

3D: surfaces, spin images

metrics: chamfer (ICP)

...

Issues: feature selection

Issues: data modeling

Generative methods

model the “shape” of each class

histograms, PCA, mixtures of Gaussians

graphical models (HMM’s, belief networks, etc.)

...

Discriminative methods

model boundaries between classes

perceptrons, neural networks

support vector machines (SVM’s)

Generative vs. Discriminative

Issues: dimensionality

What if your space isn’t flat?

PCA may not help

Other Issues

Some other factors

Prior information, context

Classification vs. inference

Representation

Other recognition problems

individuals

classes

activities

low-level properties

materials, super-resolution, edges, circles, etc...

Issues: speed

Case study: Viola Jones face detector

Exploits three key strategies:

simple, super-efficient features

image pyramids

pruning (cascaded classifiers)

Viola/Jones: features

Integral Image (aka. summed area table)

Define the Integral Image

Any rectangular sum can be computed in constant time:

Rectangle features can be computed as differences between rectangles

Viola/Jones: handling scale

Viola/Jones: cascaded classifiers

Given a nested set of classifier hypothesis classes

Computational Risk Minimization

Cascaded Classifier

first classifier: 100% detection, 50% false positives.

second classifier: 100% detection, 40% false positives

(20% cumulative)

using data from previous stage.

third classifier: 100% detection,10% false positive rate

(2% cumulative)

Put cheaper classifiers up front

Viola/Jones results:


	Readings
		C. Bishop, “Neural Networks for Pattern Recognition”, Oxford University Press, 1998, Chapter 1.
		Forsyth and Ponce, 22.3 (eigenfaces)


	What is it?
		Object detection

	Who is it?
		Recognizing identity

	What are they doing?
		Activities

	All of these are classification problems
		Choose one class from a list of possible candidates


Skin pixels have a distinctive range of colors
	Corresponds to region(s) in RGB color space
		for visualization, only R and G components are shown above


Learn the skin region from examples
	Manually label pixels in one or more “training images” as skin or not skin
	Plot the training data in RGB space
		skin pixels shown in orange, non-skin pixels shown in blue
		some skin pixels may be outside the region, non-skin pixels inside. Why?


Basic probability
	X is a random variable
	P(X) is the probability that X achieves a certain value








	or


	Conditional probability: P(X \| Y)
		probability of X given that we already know Y


	Now we can model uncertainty
		Each pixel has a probability of being skin or not skin


We can calculate P(R \| skin) from a set of training images
	It is simply a histogram over the pixels in the training images
		each bin R_i contains the proportion of skin pixels with color R_i


Bayesian estimation
	Goal is to choose the label (skin or ~skin) that maximizes the posterior
		this is called Maximum A Posteriori (MAP) estimation


	This same procedure applies in more general circumstances
		More than two classes
		More than one dimension


	Classification can be expensive
		Must either search (e.g., nearest neighbors) or store large PDF’s


Suppose each data point is N-dimensional
	Same procedure applies:



	The eigenvectors of A define a new coordinate system
		eigenvector with largest eigenvalue captures the most variation among training vectors x
		eigenvector with smallest eigenvalue has least variation
	We can compress the data by only using the top few eigenvectors
		corresponds to choosing a “linear subspace”
			represent points on a line, plane, or “hyper-plane”
		these eigenvectors are known as the principal components


	An image is a point in a high dimensional space
		An N x M image is a point in R^NM
		We can define vectors in this space as we did in the 2D case


The set of faces is a “subspace” of the set of images
	Suppose it is K dimensional
	We can find the best subspace using PCA
	This is like fitting a “hyper-plane” to the set of faces
		spanned by vectors v₁, v₂, ..., v_K
		any face


PCA extracts the eigenvectors of A
	Gives a set of vectors v₁, v₂, v₃, ...
	Each one of these vectors is a direction in face space
		what do these look like?


	The eigenfaces v₁, ..., v_K span the space of faces
		A face is converted to eigenface coordinates by


Algorithm
	Process the image database (set of images with labels)
		Run PCA—compute eigenfaces
		Calculate the K coefficients for each image
	Given a new image (to be recognized) x, calculate K coefficients
	Detect if x is a face


	If it is a face, who is it?


	How many eigenfaces to use?
	Look at the decay of the eigenvalues
		the eigenvalue tells you the amount of variance “in the direction” of that eigenface
		ignore eigenfaces with low variance


	What’s the best way to compare images?
		need to define appropriate features
		depends on goal of recognition task


Lots more feature types that we haven’t mentioned
	moments, statistics
		metrics: Earth mover’s distance, ...
	edges, curves
		metrics: Hausdorff, shape context, ...
	3D: surfaces, spin images
		metrics: chamfer (ICP)
	...


Generative methods
	model the “shape” of each class
		histograms, PCA, mixtures of Gaussians
		graphical models (HMM’s, belief networks, etc.)
		...

Discriminative methods
	model boundaries between classes
		perceptrons, neural networks
		support vector machines (SVM’s)


Some other factors
	Prior information, context
	Classification vs. inference
	Representation
	Other recognition problems
		individuals
		classes
		activities
		low-level properties
			materials, super-resolution, edges, circles, etc...


	Case study: Viola Jones face detector
	Exploits three key strategies:
		simple, super-efficient features
		image pyramids
		pruning (cascaded classifiers)


	Define the Integral Image



	Any rectangular sum can be computed in constant time:




	Rectangle features can be computed as differences between rectangles


	Given a nested set of classifier hypothesis classes





	Computational Risk Minimization


	first classifier: 100% detection, 50% false positives.
	second classifier: 100% detection, 40% false positives
	(20% cumulative)
		using data from previous stage.
	third classifier: 100% detection,10% false positive rate
	(2% cumulative)

	Put cheaper classifiers up front