Announcements

Project 4 questions

Evaluations

Image Segmentation

Today’s Readings

Forsyth chapter 14

http://www.dai.ed.ac.uk/HIPR2/morops.htm

Dilation, erosion, opening, closing

From images to objects

What Defines an Object?

Subjective problem, but has been well-studied

Gestalt Laws seek to formalize this

proximity, similarity, continuation, closure, common fate

see notes by Steve Joordens, U. Toronto

Image Segmentation

We will consider different methods

Already covered:

Intelligent Scissors (contour-based)

Hough transform (model-based)

Today:

K-means clustering (color-based)

Normalized Cuts (region-based)

Image histograms

How many “orange” pixels are in this image?

This type of question answered by looking at the histogram

A histogram counts the number of occurrences of each color

Given an image

The histogram is defined to be

What is the dimension of the histogram of an NxN RGB image?

What do histograms look like?

Photoshop demo

Histogram-based segmentation

Goal

Break the image into K regions (segments)

Solve this by reducing the number of colors to K and mapping each pixel to the closest color

photoshop demo

Clustering

How to choose the representative colors?

This is a clustering problem!

Break it down into subproblems

Suppose I tell you the cluster centers c_i

Q: how to determine which points to associate with each c_i?

K-means clustering

K-means clustering algorithm

Randomly initialize the cluster centers, c₁, ..., c_K

Given cluster centers, determine points in each cluster

For each point p, find the closest c_i. Put p into cluster i

Given points in each cluster, solve for c_i

Set c_i to be the mean of points in cluster i

If c_i have changed, repeat Step 2

Java demo: http://www.elet.polimi.it/upload/matteucc/Clustering/tutorial_html/AppletKM.html

Properties

Will always converge to some solution

Can be a “local minimum”

does not always find the global minimum of objective function:

Probabilistic clustering

Basic questions

what’s the probability that a point x is in cluster m?

what’s the shape of each cluster?

K-means doesn’t answer these questions

Probabilistic clustering (basic idea)

Treat each cluster as a Gaussian density function

Expectation Maximization (EM)

A probabilistic variant of K-means:

E step: “soft assignment” of points to clusters

estimate probability that a point is in a cluster

M step: update cluster parameters

mean and variance info (covariance matrix)

maximizes the likelihood of the points given the clusters

Forsyth Chapter 16 (optional)

EM demo

http://www.cs.ucsd.edu/users/ibayrakt/java/em/

Applications of EM

Turns out this is useful for all sorts of problems

any clustering problem

model estimation with missing/hidden data

finding outliers

segmentation problems

segmentation based on color

segmentation based on motion

foreground/background separation

...

Cleaning up the result

Problem:

Histogram-based segmentation can produce messy regions

segments do not have to be connected

may contain holes

How can these be fixed?

Dilation operator:

Dilation operator

Demo

http://www.cs.bris.ac.uk/~majid/mengine/morph.html

Erosion operator:

Erosion operator

Demo

http://www.cs.bris.ac.uk/~majid/mengine/morph.html

Nested dilations and erosions

What does this operation do?

Graph-based segmentation?

Images as graphs

Fully-connected graph

node for every pixel

link between every pair of pixels, p,q

cost c_pq for each link

c_pq measures similarity

similarity is inversely proportional to difference in color and position

this is different than the costs for intelligent scissors

Segmentation by Graph Cuts

Break Graph into Segments

Delete links that cross between segments

Easiest to break links that have low cost (similarity)

similar pixels should be in the same segments

dissimilar pixels should be in different segments

Cuts in a graph

Link Cut

set of links whose removal makes a graph disconnected

cost of a cut:

But min cut is not always the best cut...

Cuts in a graph

Interpretation as a Dynamical System

Treat the links as springs and shake the system

elasticity proportional to cost

vibration “modes” correspond to segments

can compute these by solving an eigenvector problem

Forsyth chapter 14.5

Color Image Segmentation


Today’s Readings
	Forsyth chapter 14
	http://www.dai.ed.ac.uk/HIPR2/morops.htm
		Dilation, erosion, opening, closing


What Defines an Object?
	Subjective problem, but has been well-studied
	Gestalt Laws seek to formalize this
		proximity, similarity, continuation, closure, common fate
		see notes by Steve Joordens, U. Toronto


	We will consider different methods
	Already covered:
		Intelligent Scissors (contour-based)
		Hough transform (model-based)
	Today:
		K-means clustering (color-based)
		Normalized Cuts (region-based)


How many “orange” pixels are in this image?
	This type of question answered by looking at the histogram
	A histogram counts the number of occurrences of each color
		Given an image

		The histogram is defined to be


		What is the dimension of the histogram of an NxN RGB image?


Goal
	Break the image into K regions (segments)
	Solve this by reducing the number of colors to K and mapping each pixel to the closest color
		photoshop demo


	How to choose the representative colors?
		This is a clustering problem!


	Suppose I tell you the cluster centers c_i
		Q: how to determine which points to associate with each c_i?


K-means clustering algorithm
	Randomly initialize the cluster centers, c₁, ..., c_K
	Given cluster centers, determine points in each cluster
		For each point p, find the closest c_i. Put p into cluster i
	Given points in each cluster, solve for c_i
		Set c_i to be the mean of points in cluster i
	If c_i have changed, repeat Step 2

Java demo: http://www.elet.polimi.it/upload/matteucc/Clustering/tutorial_html/AppletKM.html

Properties
	Will always converge to some solution
	Can be a “local minimum”
		does not always find the global minimum of objective function:


	Basic questions
		what’s the probability that a point x is in cluster m?
		what’s the shape of each cluster?
	K-means doesn’t answer these questions

	Probabilistic clustering (basic idea)
		Treat each cluster as a Gaussian density function


A probabilistic variant of K-means:
	E step: “soft assignment” of points to clusters
		estimate probability that a point is in a cluster
	M step: update cluster parameters
		mean and variance info (covariance matrix)
	maximizes the likelihood of the points given the clusters
	Forsyth Chapter 16 (optional)


Turns out this is useful for all sorts of problems
	any clustering problem
	model estimation with missing/hidden data
	finding outliers
	segmentation problems
		segmentation based on color
		segmentation based on motion
		foreground/background separation
	...


Problem:
	Histogram-based segmentation can produce messy regions
		segments do not have to be connected
		may contain holes

How can these be fixed?


Fully-connected graph
	node for every pixel
	link between every pair of pixels, p,q
	cost c_pq for each link
		c_pq measures similarity
			similarity is inversely proportional to difference in color and position
			this is different than the costs for intelligent scissors


Break Graph into Segments
	Delete links that cross between segments
	Easiest to break links that have low cost (similarity)
		similar pixels should be in the same segments
		dissimilar pixels should be in different segments


	Link Cut
		set of links whose removal makes a graph disconnected
		cost of a cut:


Treat the links as springs and shake the system
	elasticity proportional to cost
	vibration “modes” correspond to segments
		can compute these by solving an eigenvector problem
		Forsyth chapter 14.5