Notes
Slide Show
Outline
1
Announcements
  • Project 2
    • more signup slots
    • questions
  • Picture taking at end of class
2
Image Segmentation
  • Today’s Readings
    • Forsyth chapter 14, 16
3
From images to objects
  • What Defines an Object?
    • Subjective problem, but has been well-studied
    • Gestalt Laws seek to formalize this
      • proximity, similarity, continuation, closure, common fate
      • see notes by Steve Joordens, U. Toronto
4
Image Segmentation
  • We will consider different methods


  • Already covered:
    • Intelligent Scissors (contour-based)
    • Hough transform (model-based)

  • This week:
    • K-means clustering (color-based)
    • EM
    • Mean-shift
    • Normalized Cuts (region-based)
5
Image histograms
  • How many “orange” pixels are in this image?
    • This type of question answered by looking at the histogram
    • A histogram counts the number of occurrences of each color
      • Given an image


      • The histogram is defined to be



6
What do histograms look like?
7
Histogram-based segmentation
  • Goal
    • Break the image into K regions (segments)
    • Solve this by reducing the number of colors to K and mapping each pixel to the closest color
      • photoshop demo
8
Histogram-based segmentation
  • Goal
    • Break the image into K regions (segments)
    • Solve this by reducing the number of colors to K and mapping each pixel to the closest color
      • photoshop demo
9
Clustering
  • How to choose the representative colors?
    • This is a clustering problem!
10
Break it down into subproblems
  • Suppose I tell you the cluster centers ci
    • Q:  how to determine which points to associate with each ci?
11
K-means clustering
  • K-means clustering algorithm
    • Randomly initialize the cluster centers, c1, ..., cK
    • Given cluster centers, determine points in each cluster
      • For each point p, find the closest ci.  Put p into cluster i
    • Given points in each cluster, solve for ci
      • Set ci to be the mean of points in cluster i
    • If ci have changed, repeat Step 2


  • Java demo:  http://www.cs.mcgill.ca/~bonnef/project.html


  • Properties
    • Will always converge to some solution
    • Can be a “local minimum”
      • does not always find the global minimum of objective function:

12
Probabilistic clustering
  • Basic questions
    • what’s the probability that a point x is in cluster m?
    • what’s the shape of each cluster?
  • K-means doesn’t answer these questions


  • Basic idea
    • instead of treating the data as a bunch of points, assume that they are all generated by sampling a continuous function
    • This function is called a generative model
      • defined by a vector of parameters θ

13
Mixture of Gaussians
  • One generative model is a mixture of Gaussians (MOG)
    • K Gaussian blobs with means μb covariance matrices Vb, dimension d
      • blob b defined by:


    • blob b is selected with probability
    • the likelihood of observing x is a weighted mixture of Gaussians



    • where
14
Expectation maximization (EM)
  • Goal
    • find blob parameters θ that maximize the likelihood function:

  • Approach:
    • E step:  given current guess of blobs, compute ownership of each point
    • M step:  given ownership probabilities, update blobs to maximize likelihood function
    • repeat until convergence


15
EM details
  • E-step
    • compute probability that point x is in blob i, given current guess of θ




  • M-step
    • compute probability that blob b is selected



    • mean of blob b



    • covariance of blob b
16
EM demos




  • http://www.cs.ucsd.edu/users/ibayrakt/java/em/



  • http://www.dreier.cc/index.php?topic=downloads&sub=em
17
Applications of EM
  • Turns out this is useful for all sorts of problems
    • any clustering problem
    • any model estimation problem
    • missing data problems
    • finding outliers
    • segmentation problems
      • segmentation based on color
      • segmentation based on motion
      • foreground/background separation
    • ...


18
Problems with EM
  • Local minima



  • Need to know number of segments



  • Need to choose generative model
19
Finding Modes in a Histogram
  • How Many Modes Are There?
    • Easy to see, hard to compute


20
Mean Shift [Comaniciu & Meer]
  • Iterative Mode Search
    • Initialize random seed, and window W
    • Calculate center of gravity (the “mean”) of W:
    • Translate the search window to the mean
    • Repeat Step 2 until convergence
21
Mean-Shift
  • Approach
    • Initialize a window around each point
    • See where it shifts—this determines which segment it’s in
    • Multiple points will shift to the same segment
22
Mean-shift for image segmentation
  • Useful to take into account spatial information
    • instead of (R, G, B), run in (R, G, B, x, y) space
    • D. Comaniciu, P. Meer, Mean shift analysis and applications, 7th International Conference on Computer Vision, Kerkyra, Greece, September 1999, 1197-1203.
      • http://www.caip.rutgers.edu/riul/research/papers/pdf/spatmsft.pdf
23
Region-based segmentation
  • Color histograms don’t take into account spatial info
    • Gestalt laws point out importance of spatial grouping
      • proximity, similarity, continuation, closure, common fate
    • Suggests that regions are important
24
Images as graphs
  • Fully-connected graph
    • node for every pixel
    • link between every pair of pixels, p,q
    • cost cpq for each link
      • cpq measures similarity
        • similarity is inversely proportional to difference in color and position
        • this is different than the costs for intelligent scissors
25
Segmentation by Graph Cuts
  • Break Graph into Segments
    • Delete links that cross between segments
    • Easiest to break links that have high cost
      • similar pixels should be in the same segments
      • dissimilar pixels should be in different segments

26
Cuts in a graph
  • Link Cut
    • set of links whose removal makes a graph disconnected
    • cost of a cut:
27
But min cut is not always the best cut...
28
Cuts in a graph
29
Interpretation as a Dynamical System
  • Treat the links as springs and shake the system
    • elasticity proportional to cost
    • vibration “modes” correspond to segments
30
Interpretation as a Dynamical System
  • Treat the links as springs and shake the system
    • elasticity proportional to cost
    • vibration “modes” correspond to segments
31
Color Image Segmentation
32
Normalize Cut in Matrix Form
  • Can write normalized cut as:
33
Cleaning up the result
  • Problem:
    • Histogram-based segmentation can produce messy regions
      • segments do not have to be connected
      • may contain holes


  • How can these be fixed?
34
Dilation operator:
35
Dilation operator
  • Demo
    • http://www.cs.bris.ac.uk/~majid/mengine/morph.html


36
Erosion operator:
37
Erosion operator
  • Demo
    • http://www.cs.bris.ac.uk/~majid/mengine/morph.html


38
Nested dilations and erosions
  • What does this operation do?
39
Nested dilations and erosions
  • What does this operation do?
40
Nested dilations and erosions
  • What does this operation do?