Announcements

Project 3 extension: Wednesday at noon

Final project proposal extension: Friday at noon

consult with Steve, Rick, and/or Ian now!

Project 2 artifact winners...

Active stereo with structured light

Project “structured” light patterns onto the object

simplifies the correspondence problem

Laser scanning

Optical triangulation

Project a single stripe of laser light

Scan it across the surface of the object

This is a very precise version of structured light scanning

3D cameras

Multiview stereo

Choosing the stereo baseline

What’s the optimal baseline?

Too small: large depth error

Too large: difficult search problem

The Effect of Baseline on Depth Estimation

Slide 9

Slide 10

Multibaseline Stereo

Basic Approach

Choose a reference view

Use your favorite stereo algorithm BUT

replace two-view SSD with SSD over all baselines

Limitations

Must choose a reference view (bad)

Visibility!

CMU’s 3D Room Video

The visibility problem

Volumetric stereo

Discrete formulation: Voxel Coloring

Complexity and computability

Issues

Theoretical Questions

Identify class of all photo-consistent scenes

Practical Questions

How do we compute photo-consistent models?

Slide 17

Reconstruction from Silhouettes (C = 2)

Volume intersection

Reconstruction Contains the True Scene

But is generally not the same

In the limit (all views) get visual hull

Complement of all lines that don’t intersect S

Voxel algorithm for volume intersection

Color voxel black if on silhouette in every image

for M images, N³ voxels

Don’t have to search 2^N³ possible scenes!

Properties of Volume Intersection

Pros

Easy to implement, fast

Accelerated via octrees [Szeliski 1993] or interval techniques [Matusik 2000]

Cons

No concavities

Reconstruction is not photo-consistent

Requires identification of silhouettes

Slide 22

Voxel Coloring Approach

Depth Ordering: visit occluders first!

Panoramic Depth Ordering

Cameras oriented in many different directions

Planar depth ordering does not apply

Panoramic Depth Ordering

Panoramic Layering

Compatible Camera Configurations

Calibrated Image Acquisition

Calibrated Turntable

360° rotation (21 images)

Voxel Coloring Results (Video)

Limitations of Depth Ordering

A view-independent depth order may not exist

Slide 33

Space Carving Algorithm

Space Carving Algorithm

Which shape do you get?

The Photo Hull is the UNION of all photo-consistent scenes in V

It is a photo-consistent scene reconstruction

Tightest possible bound on the true scene

Space Carving Algorithm

The Basic Algorithm is Unwieldy

Complex update procedure

Alternative: Multi-Pass Plane Sweep

Efficient, can use texture-mapping hardware

Converges quickly in practice

Easy to implement

Multi-Pass Plane Sweep

Sweep plane in each of 6 principle directions

Consider cameras on only one side of plane

Repeat until convergence

Space Carving Results: African Violet

Space Carving Results: Hand

Properties of Space Carving

Pros

Voxel coloring version is easy to implement, fast

Photo-consistent results

No smoothness prior

Cons

Bulging

No smoothness prior

Alternatives to space carving

Optimizing space carving

recent surveys

Slabaugh et al., 2001

Dyer et al., 2001

many others...

Graph cuts

Kolmogorov & Zabih

Level sets

introduce smoothness term

surface represented as an implicit function in 3D volume

optimize by solving PDE’s

Alternatives to space carving

Optimizing space carving

recent surveys

Slabaugh et al., 2001

Dyer et al., 2001

many others...

Graph cuts

Ramin Zabih’s lecture

Level sets

introduce smoothness term

surface represented as an implicit function in 3D volume

optimize by solving PDE’s

Level sets vs. space carving

Advantages of level sets

optimizes consistency with images + smoothness term

excellent results for smooth things

does not require as many images

Advantages of space carving

much simpler to implement

runs faster (orders of magnitude)

works better for thin structures, discontinuities

For more info on level set stereo:

Renaud Keriven’s page:

http://cermics.enpc.fr/~keriven/stereo.html

Current/Future Trends

Optimizing with visibility

Kolmogorov & Zabih

Current/Future Trends

Real-time algorithms

e.g., Buehler et al., image-based visual hulls, SIGGRAPH 2000

Current/Future Trends

Modeling shiny things (BRDF’s and materials)

e.g., Zickler et al., Helmholtz Stereopsis

References

Volume Intersection

Martin & Aggarwal, “Volumetric description of objects from multiple views”, Trans. Pattern Analysis and Machine Intelligence, 5(2), 1991, pp. 150-158.

Szeliski, “Rapid Octree Construction from Image Sequences”, Computer Vision, Graphics, and Image Processing: Image Understanding, 58(1), 1993, pp. 23-32.

Matusik, Buehler, Raskar, McMillan, and Gortler , “Image-Based Visual Hulls”, Proc. SIGGRAPH 2000, pp. 369-374.

Voxel Coloring and Space Carving

Seitz & Dyer, “Photorealistic Scene Reconstruction by Voxel Coloring”, Intl. Journal of Computer Vision (IJCV), 1999, 35(2), pp. 151-173.

Kutulakos & Seitz, “A Theory of Shape by Space Carving”, International Journal of Computer Vision, 2000, 38(3), pp. 199-218.

Recent surveys

Slabaugh, Culbertson, Malzbender, & Schafer, “A Survey of Volumetric Scene Reconstruction Methods from Photographs”, Proc. workshop on Volume Graphics 2001, pp. 81-100. http://users.ece.gatech.edu/~slabaugh/personal/publications/vg01.pdf

Dyer, “Volumetric Scene Reconstruction from Multiple Views”, Foundations of Image Understanding, L. S. Davis, ed., Kluwer, Boston, 2001, 469-489.
ftp://ftp.cs.wisc.edu/computer-vision/repository/PDF/dyer.2001.fia.pdf

References

Other references from this talk

Multibaseline Stereo: Masatoshi Okutomi and Takeo Kanade. A multiple-baseline stereo. IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), 15(4), 1993, pp. 353--363.

Level sets: Faugeras & Keriven, “Variational principles, surface evolution, PDE's, level set methods and the stereo problem", IEEE Trans. on Image Processing, 7(3), 1998, pp. 336-344.

Mesh based: Fua & Leclerc, “Object-centered surface reconstruction: Combining multi-image stereo and shading", IJCV, 16, 1995, pp. 35-56.

3D Room: Narayanan, Rander, & Kanade, “Constructing Virtual Worlds Using Dense Stereo”, Proc. ICCV, 1998, pp. 3-10.

Graph-based: Kolmogorov & Zabih, “Multi-Camera Scene Reconstruction via Graph Cuts”, Proc. European Conf. on Computer Vision (ECCV), 2002.

Helmholtz Stereo: Zickler, Belhumeur, & Kriegman, “Helmholtz Stereopsis: Exploiting Reciprocity for Surface Reconstruction”, IJCV, 49(2-3), 2002, pp. 215-227.


		Project 3 extension: Wednesday at noon
		Final project proposal extension: Friday at noon
			consult with Steve, Rick, and/or Ian now!
		Project 2 artifact winners...


	Project “structured” light patterns onto the object
		simplifies the correspondence problem


	Optical triangulation
		Project a single stripe of laser light
		Scan it across the surface of the object
		This is a very precise version of structured light scanning


	What’s the optimal baseline?
		Too small: large depth error
		Too large: difficult search problem


Basic Approach
	Choose a reference view
	Use your favorite stereo algorithm BUT
		replace two-view SSD with SSD over all baselines

Limitations
	Must choose a reference view (bad)
	Visibility!

CMU’s 3D Room Video


	Theoretical Questions
		Identify class of all photo-consistent scenes

	Practical Questions
		How do we compute photo-consistent models?


Reconstruction Contains the True Scene
	But is generally not the same
	In the limit (all views) get visual hull
		Complement of all lines that don’t intersect S


	Color voxel black if on silhouette in every image
		for M images, N³ voxels
		Don’t have to search 2^N³ possible scenes!


	Pros
		Easy to implement, fast
		Accelerated via octrees [Szeliski 1993] or interval techniques [Matusik 2000]

	Cons
		No concavities
		Reconstruction is not photo-consistent
		Requires identification of silhouettes


		Cameras oriented in many different directions
		Planar depth ordering does not apply


	The Photo Hull is the UNION of all photo-consistent scenes in V
		It is a photo-consistent scene reconstruction
		Tightest possible bound on the true scene


	The Basic Algorithm is Unwieldy
		Complex update procedure

	Alternative: Multi-Pass Plane Sweep
		Efficient, can use texture-mapping hardware
		Converges quickly in practice
		Easy to implement


		Sweep plane in each of 6 principle directions
		Consider cameras on only one side of plane
		Repeat until convergence


	Pros
		Voxel coloring version is easy to implement, fast
		Photo-consistent results
		No smoothness prior

	Cons
		Bulging
		No smoothness prior


Optimizing space carving
	recent surveys
		Slabaugh et al., 2001
		Dyer et al., 2001
	many others...
Graph cuts
	Kolmogorov & Zabih
Level sets
	introduce smoothness term
	surface represented as an implicit function in 3D volume
	optimize by solving PDE’s


Advantages of level sets
	optimizes consistency with images + smoothness term
	excellent results for smooth things
	does not require as many images

Advantages of space carving
	much simpler to implement
	runs faster (orders of magnitude)
	works better for thin structures, discontinuities

For more info on level set stereo:
	Renaud Keriven’s page:
		http://cermics.enpc.fr/~keriven/stereo.html


	Real-time algorithms
		e.g., Buehler et al., image-based visual hulls, SIGGRAPH 2000


	Modeling shiny things (BRDF’s and materials)
		e.g., Zickler et al., Helmholtz Stereopsis


Volume Intersection
	Martin & Aggarwal, “Volumetric description of objects from multiple views”, Trans. Pattern Analysis and Machine Intelligence, 5(2), 1991, pp. 150-158.
	Szeliski, “Rapid Octree Construction from Image Sequences”, Computer Vision, Graphics, and Image Processing: Image Understanding, 58(1), 1993, pp. 23-32.
	Matusik, Buehler, Raskar, McMillan, and Gortler , “Image-Based Visual Hulls”, Proc. SIGGRAPH 2000, pp. 369-374.
Voxel Coloring and Space Carving
	Seitz & Dyer, “Photorealistic Scene Reconstruction by Voxel Coloring”, Intl. Journal of Computer Vision (IJCV), 1999, 35(2), pp. 151-173.
	Kutulakos & Seitz, “A Theory of Shape by Space Carving”, International Journal of Computer Vision, 2000, 38(3), pp. 199-218.
	Recent surveys
		Slabaugh, Culbertson, Malzbender, & Schafer, “A Survey of Volumetric Scene Reconstruction Methods from Photographs”, Proc. workshop on Volume Graphics 2001, pp. 81-100. http://users.ece.gatech.edu/~slabaugh/personal/publications/vg01.pdf
		Dyer, “Volumetric Scene Reconstruction from Multiple Views”, Foundations of Image Understanding, L. S. Davis, ed., Kluwer, Boston, 2001, 469-489. ftp://ftp.cs.wisc.edu/computer-vision/repository/PDF/dyer.2001.fia.pdf


	Other references from this talk
		Multibaseline Stereo: Masatoshi Okutomi and Takeo Kanade. A multiple-baseline stereo. IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), 15(4), 1993, pp. 353--363.
		Level sets: Faugeras & Keriven, “Variational principles, surface evolution, PDE's, level set methods and the stereo problem", IEEE Trans. on Image Processing, 7(3), 1998, pp. 336-344.
		Mesh based: Fua & Leclerc, “Object-centered surface reconstruction: Combining multi-image stereo and shading", IJCV, 16, 1995, pp. 35-56.
		3D Room: Narayanan, Rander, & Kanade, “Constructing Virtual Worlds Using Dense Stereo”, Proc. ICCV, 1998, pp. 3-10.
		Graph-based: Kolmogorov & Zabih, “Multi-Camera Scene Reconstruction via Graph Cuts”, Proc. European Conf. on Computer Vision (ECCV), 2002.
		Helmholtz Stereo: Zickler, Belhumeur, & Kriegman, “Helmholtz Stereopsis: Exploiting Reciprocity for Surface Reconstruction”, IJCV, 49(2-3), 2002, pp. 215-227.