Image-Based Rendering
|
|
|
Computer Vision
CSE576, Spring 2005
Richard Szeliski |
Today’s lecture
|
|
|
Image–Based Rendering |
|
Light Fields and Lumigraphs |
|
Panoramas and Concentric
Mosaics |
|
Environment Matting |
|
Image-Based models |
Today’s lecture
|
|
|
Video-Based Rendering |
|
Facial animation |
|
Video matting and shadow
matting |
|
Video Textures and Animating
Stills |
|
Video-based tours |
Readings
|
|
|
S. J. Gortler , R. Grzeszczuk ,
R. Szeliski and M. F. Cohen, The Lumigraph, SIGGRAPH'96. |
|
M. Levoy and P. Hanrahan, Light
field rendering, SIGGRAPH'96. |
|
H.-Y. Shum and L.-W. He.
Rendering with concentric mosaics, SIGGRAPH’99. |
Readings
|
|
|
D. E. Zongker et al. Environment
matting and compositing, SIGGRAPH'99. |
|
Y.-Y. Chuang et al. Environment
matting extensions: Towards higher accuracy and real-time capture.
SIGGRAPH'2000, pp.121-130, 2000. |
|
P. E. Debevec , C. J. Taylor
and J. Malik, Modeling and rendering architecture from
photographs:…, SIGGRAPH'96. |
Readings
|
|
|
Y.-Y. Chuang et al. Video
matting of complex scenes. ACM Trans. on Graphics, 21(3):243-248, July 2002 |
|
Y.-Y. Chuang et al. Shadow
matting. ACM Transactions on Graphics, 22(3):494-500, July 2003. |
|
A. Schödl et al., Video
textures. SIGGRAPH'2000, pp. 489-498, 2000. |
|
M. Uyttendaele et al.
Image-based interactive exploration of real-world environments. IEEE Comp.
Graphics and Applications, 24(3), May/June 2004. |
Lightfields and
Lumigraphs
|
|
|
(with lots of slides from
Michael Cohen) |
Modeling light
|
|
|
|
How do we generate new scenes
and animations from existing ones? |
|
Classic “3D Vision + Graphics”: |
|
take (lots of) pictures |
|
recover camera pose |
|
build 3D model |
|
extract texture maps / BRDFs |
|
synthesize new views |
Computer Graphics
Computer Vision
Combined
But, vision technology
falls short
… and so does graphics.
Image Based Rendering
Ray
|
|
|
|
Constant radiance |
|
time is fixed |
|
|
|
|
|
|
|
5D |
|
3D position |
|
2D direction |
All Rays
|
|
|
|
Plenoptic Function: |
|
all possible images |
|
too much stuff! |
Line
|
|
|
|
Infinite line |
|
|
|
|
|
|
|
|
|
4D |
|
2D direction |
|
2D position |
|
non-dispersive medium |
Ray
|
|
|
|
Discretize, then interpolate |
|
|
|
|
|
|
|
|
|
|
|
Distance between 2 rays |
|
Which is closer together? |
Image
|
|
|
|
What is an image? |
|
|
|
|
|
|
|
|
|
|
|
All rays through a point |
|
Panorama |
Panoramic Mosaics
|
|
|
Convert panoramic image
sequence into a cylindrical image |
|
|
|
|
|
+ + … +
= |
Image
|
|
|
|
Image plane |
|
|
|
|
|
|
|
|
|
2D |
|
position in plane |
Object
|
|
|
|
Light leaving towards “eye” |
|
|
|
|
|
|
|
|
|
2D |
|
just dual of image |
Object
Slide 24
Object
Lumigraph / Lightfield
Slide 27
Lumigraph
|
|
|
|
|
|
How to ? |
|
organize |
|
capture |
|
render |
|
|
Lumigraph - Organization
Slide 30
Slide 31
Slide 32
Slide 33
Slide 34
Slide 35
Lumigraph - Rendering
|
|
|
|
|
For each output pixel |
|
determine s,t,u,v |
|
either |
|
find closest discrete RGB |
|
interpolate near values |
Slide 37
Lumigraph - Rendering
|
|
|
|
Nearest |
|
closest s |
|
closest u |
|
draw it |
|
|
|
|
|
Blend 16 nearest |
|
quadrilinear interpolation |
Lumigraph - Rendering
|
|
|
|
Depth Correction |
|
closest s |
|
intersection with “object” |
|
best u |
|
closest u |
|
|
Lumigraph - Rendering
|
|
|
|
Depth Correction |
|
quadralinear interpolation |
|
new “closest” |
|
like focus |
|
[Dynamically |
|
Reparameterized |
|
Light Fields, |
|
Isaksen, SG’2000] |
Lumigraph - Rendering
|
|
|
|
Fast s,t,u,v finding |
|
scanline interpolate |
|
texture mapping |
|
shear warp |
Lumigraph - Ray Space
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
3D space ray space |
|
surface depth Û slope in ray space |
Lumigraph - Ray Space
|
|
|
Image effects: |
|
parallax |
|
occlusion |
|
transparency |
|
highlights |
Lumigraph - Demo
|
|
|
|
Lumigraph |
|
Lion, Fruit Bowl |
Complex Light Field
acquisition
|
|
|
|
|
Digital Michelangelo Project |
|
Marc Levoy, Stanford University |
|
Lightfield (“night”) assembled
by Jon Shade |
|
|
Unstructured Lumigraph
|
|
|
What if the images aren’t
sampled on a regular 2D grid? |
|
can still re-sample rays |
|
ray weighting becomes more
complex
[Buehler et al., SIGGRAPH’2000] |
Surface Light Fields
|
|
|
Turn 4D parameterization
around: |
|
image @ every surface pt. |
|
Leverage coherence: |
|
compress radiance fn
(BRDF * illumination)
after rotation by n |
Surface Light Fields
|
|
|
[Wood et al, SIGGRAPH 2000] |
3D Representations
|
|
|
|
Image (and panoramas) are 2D |
|
Lumigraph is 4D |
|
What happened to 3D? |
|
3D Lumigraph subset |
|
Concentric mosaics |
3D Lumigraph
|
|
|
|
One row of s,t plane |
|
i.e., hold t constant |
3D Lumigraph
|
|
|
|
One row of s,t plane |
|
i.e., hold t constant |
|
thus s,u,v |
|
a “row of images” |
|
|
|
|
|
|
|
|
|
|
|
[Sloan et al., Symp. I3DG 97] |
Concentric Mosaics
|
|
|
Replace “row” with “circle” of
images |
|
[Shum & He, SIGGRAPH’97] |
Concentric Mosaics
Concentric Mosaics
|
|
|
Rendering |
|
|
|
( as seen
from above ) |
Slide 55
Slide 56
2.5D Representations
|
|
|
|
Image is 2D |
|
Lumigraph is 4D |
|
3D |
|
3D Lumigraph subset |
|
Concentric mosaics |
|
2.5D |
|
Layered Depth Images |
|
Sprites with Depth (impostors) |
|
View Dependent Surfaces (see Façade) |
Layered Depth Image
Slide 59
Slide 60
Sprites with Depth
|
|
|
Represent scene as collection
of cutouts with depth (planes + parallax) |
|
Render back to front with
fwd/inverse warping [Shade et al., SIGGRAPH’98] |
Environment
matting
and compositing
|
|
|
D. E. Zongker, D. M.
Werner,
B. Curless and D. H. Salesin. SIGGRAPH'99 |
Environment Matting
|
|
|
Capture the reflections and refractions
of a real-world object |
|
Composite object over a novel
background |
Environment Matting -
examples
Environment Matting
|
|
|
Capture the mapping from
each
image pixel to a real-world
ray direction(s) |
Acquisition setup
|
|
|
Use several monitors with
stripes |
Environment matting
equation
|
|
|
Captures foreground color
and background directions |
Environment matting -
result
Environment matting
extensions
|
|
|
|
[Chuang et al., SIGGRAPH’2001] |
|
accurate (multiple
refractions): |
|
soft stripes |
|
fast (video rate): |
|
color ramp |
Image-Based Modeling
Image Based Models
Image-Based Modeling
|
|
|
|
Create 3D model (and texture
maps) from images |
|
automated |
|
(structure from motion,
stereo) |
|
interactive |
|
Façade system |
Façade
|
|
|
Select building blocks |
|
Align them in each image |
|
Solve for camera pose
and block parameters
(using constraints) |
View-dependent texture
mapping
|
|
|
Determine visible cameras for
each surface element |
|
Blend textures (images)
depending on distance between original camera and novel viewpoint |
Model-based stereo
|
|
|
Compute offset from block model |
|
|
|
|
|
|
|
|
|
Some more results: |
Image-Based Faces
|
|
|
|
Estimate shape from images |
|
Match metrics to shape |
|
Project video onto shape |
|
Texture map |
|
Animate |
|
[Z. Liu et al., MSR-TR-2000-11] |
Hierarchy of Light Fields
[Levoy]
|
|
|
8D: Refractive/reflective
environment |
|
5D: Plenoptic Function (Ray) |
|
4D: Lumigraph / Lightfield |
|
4D*: Environment Matte (single
view) |
|
3D: Lumigraph Subset |
|
3D: Concentric Mosaics |
|
2.5D: Layered Depth Image |
|
2.5D: Image Based Models |
|
2D: Images and Panoramas |
Graphics/Imaging
Continuum
What lies beyond
Image-Based Rendering?
Video-Based Rendering
|
|
|
|
Image-Based Rendering: |
|
render from (real-world) images
for efficiency, quality, and photo-realism |
|
Video-Based Rendering |
|
use video instead of still
images for dynamic elements and source footage |
|
generate computer video instead
of
computer graphics |
VBR Examples
|
|
|
|
Facial animation |
|
Video Rewrite, … |
|
Layer/matte extraction |
|
Video Matting, … |
|
Dynamic (stochastic) elements |
|
Video Textures, … |
|
3-D world navigation |
|
Image-Based Realities, … |
Facial animation
|
|
|
|
Modeling from still images |
|
[Pighin et al., SG’98] |
|
Lip-synching from video |
|
Video Rewrite
[Bregler et al., SG’97] |
|
[Ezzat et al., SG’02] |
Matting and Compositing
Slide 84
Video Matting
|
|
|
Pull dynamic a-matte
from video with
complex backgrounds
[Chuang et al. @ UW, SIGGRAPH’2002] |
Video Matting
Shadow Matting
|
|
|
Transfer a shadow from one
background to another: |
|
Extract and model photometry
(darkening) |
|
Extract and model geometry
(deformation) |
Shadow Matting
Shadow Matting
Shadow Matting
Video Textures
Video Textures
|
|
|
|
How can we turn a short video
clip
into an ¥
amount of continuous video? |
|
dynamic elements in 3D games
and presentations |
|
alternative to 3D graphics
animation? |
|
[Schödl, Szeliski, Salesin,
Essa, SG’2000] |
Video Textures
|
|
|
Find cyclic structure in the
video |
|
(Optional) region-based
analysis |
|
Play frames with random shuffle |
|
Smooth over discontinuities
(morph) |
Region-based analysis
Crossfading and morphing
Video portrait
Dynamic scene element
|
|
|
Live waterfall in static
panorama |
Interactive fish
A complete animation
Video-Based Tours
Video-Based Walkthroughs
|
|
|
|
Move camera along a rail
(“dolly track”) and play back a 360° video |
|
Applications: |
|
Homes and architecture |
|
Outdoor locations
(tourist destinations) |
Surround video
acquisition system
OmniCam
|
|
|
Built by Point Grey Research (Ladybug) |
|
Six camera head |
|
Portable hard drives,
fiber-optic link |
|
Resolution per image: 1024 x
768 |
|
FOV: ~100o x ~80o |
|
Acquisition speed: 15 fps
uncompressed |
Acquisition platforms
Demo
A Virtual Home Tour
Open issues
|
|
|
How to best sample and
interpolate Light Field |
|
(sub-?) pixel accurate stereo |
|
reflections, refractions, … |
|
Compositing |
|
how to insert Light Field into
new environment |
|
relighting |
|
…? |
Summary
|
|
|
Image–Based Rendering |
|
Light Fields and Lumigraphs |
|
Panoramas and Concentric
Mosaics |
|
Matting: natural, environment,
and shadows |
|
Image-Based models |
|
Video-Based Rendering |
|
Facial animation |
|
Video Textures and Animating
Stills |
|
Video-based tours |
Summary
|
|
|
Image–Based Rendering |
|
Light Fields and Lumigraphs |
|
Panoramas and Concentric
Mosaics |
|
Environment Matting |
|
Image-Based models |
|
Video-Based Rendering |
|
Facial animation |
|
Video matting |
|
Video Textures and Animating
Stills |
|
Video-based tours |