Announcements

Photos right now

Project 3 questions

Midterms back at the end of class

Image Segmentation

Today’s Readings

Shapiro, pp. 279-289

http://www.dai.ed.ac.uk/HIPR2/morops.htm

Dilation, erosion, opening, closing

From images to objects

What Defines an Object?

Subjective problem, but has been well-studied

Gestalt Laws seek to formalize this

proximity, similarity, continuation, closure, common fate

see notes by Steve Joordens, U. Toronto

Image Segmentation

We will consider different methods

Already covered:

Intelligent Scissors (contour-based, manual)

Today—automatic methods:

K-means clustering (color-based)

Normalized Cuts (region-based)

Image histograms

How many “orange” pixels are in this image?

This type of question answered by looking at the histogram

A histogram counts the number of occurrences of each color

Given an image

The histogram is

i.e., for each color value c (x-axis), plot # of pixels with that color (y-axis)

What is the dimension of the histogram of an NxN RGB image?

What do histograms look like?

Photoshop demo

Histogram-based segmentation

Goal

Break the image into K regions (segments)

Solve this by reducing the number of colors to K and mapping each pixel to the closest color

photoshop demo

Clustering

How to choose the representative colors?

This is a clustering problem!

Break it down into subproblems

Suppose I tell you the cluster centers c_i

Q: how to determine which points to associate with each c_i?

K-means clustering

K-means clustering algorithm

Randomly initialize the cluster centers, c₁, ..., c_K

Given cluster centers, determine points in each cluster

For each point p, find the closest c_i. Put p into cluster i

Given points in each cluster, solve for c_i

Set c_i to be the mean of points in cluster i

If c_i have changed, repeat Step 2

Java demo: http://home.dei.polimi.it/matteucc/Clustering/tutorial_html/AppletKM.html

Properties

Will always converge to some solution

Can be a “local minimum”

does not always find the global minimum of objective function:

Cleaning up the result

Problem:

Histogram-based segmentation can produce messy regions

segments do not have to be connected

may contain holes

How can these be fixed?

Dilation operator:

Dilation operator

Demo

http://www.cs.bris.ac.uk/~majid/mengine/morph.html

Erosion operator:

Erosion operator

Demo

http://www.cs.bris.ac.uk/~majid/mengine/morph.html

Nested dilations and erosions

What does this operation do?

Graph-based segmentation?

Images as graphs

Fully-connected graph

node for every pixel

link between every pair of pixels, p,q

cost c_pq for each link

c_pq measures similarity

similarity is inversely proportional to difference in color and position

this is different than the costs for intelligent scissors

Segmentation by Graph Cuts

Break Graph into Segments

Delete links that cross between segments

Easiest to break links that have low cost (low similarity)

similar pixels should be in the same segments

dissimilar pixels should be in different segments

Cuts in a graph

Link Cut

set of links whose removal makes a graph disconnected

cost of a cut:

But min cut is not always the best cut...

Cuts in a graph

Interpretation as a Dynamical System

Treat the links as springs and shake the system

elasticity proportional to cost

vibration “modes” correspond to segments

can compute these by solving an eigenvector problem

for more details, see

J. Shi and J. Malik, Normalized Cuts and Image Segmentation, CVPR, 1997

Interpretation as a Dynamical System

Color Image Segmentation


	Photos right now
	Project 3 questions
	Midterms back at the end of class


Today’s Readings
	Shapiro, pp. 279-289
		http://www.dai.ed.ac.uk/HIPR2/morops.htm
		Dilation, erosion, opening, closing


What Defines an Object?
	Subjective problem, but has been well-studied
	Gestalt Laws seek to formalize this
		proximity, similarity, continuation, closure, common fate
		see notes by Steve Joordens, U. Toronto


	We will consider different methods
	Already covered:
		Intelligent Scissors (contour-based, manual)
	Today—automatic methods:
		K-means clustering (color-based)
		Normalized Cuts (region-based)


How many “orange” pixels are in this image?
	This type of question answered by looking at the histogram
	A histogram counts the number of occurrences of each color
		Given an image

		The histogram is
			i.e., for each color value c (x-axis), plot # of pixels with that color (y-axis)
		What is the dimension of the histogram of an NxN RGB image?


Goal
	Break the image into K regions (segments)
	Solve this by reducing the number of colors to K and mapping each pixel to the closest color
		photoshop demo


	How to choose the representative colors?
		This is a clustering problem!


	Suppose I tell you the cluster centers c_i
		Q: how to determine which points to associate with each c_i?


K-means clustering algorithm
	Randomly initialize the cluster centers, c₁, ..., c_K
	Given cluster centers, determine points in each cluster
		For each point p, find the closest c_i. Put p into cluster i
	Given points in each cluster, solve for c_i
		Set c_i to be the mean of points in cluster i
	If c_i have changed, repeat Step 2

Java demo: http://home.dei.polimi.it/matteucc/Clustering/tutorial_html/AppletKM.html


Properties
	Will always converge to some solution
	Can be a “local minimum”
		does not always find the global minimum of objective function:


Problem:
	Histogram-based segmentation can produce messy regions
		segments do not have to be connected
		may contain holes

How can these be fixed?


Fully-connected graph
	node for every pixel
	link between every pair of pixels, p,q
	cost c_pq for each link
		c_pq measures similarity
			similarity is inversely proportional to difference in color and position
			this is different than the costs for intelligent scissors


Break Graph into Segments
	Delete links that cross between segments
	Easiest to break links that have low cost (low similarity)
		similar pixels should be in the same segments
		dissimilar pixels should be in different segments


	Link Cut
		set of links whose removal makes a graph disconnected
		cost of a cut:


Treat the links as springs and shake the system
	elasticity proportional to cost
	vibration “modes” correspond to segments
		can compute these by solving an eigenvector problem
		for more details, see
			J. Shi and J. Malik, Normalized Cuts and Image Segmentation, CVPR, 1997