1	Image-Based Rendering Computer Vision CSE576, Spring 2005 Richard Szeliski
2	Today’s lecture Image–Based Rendering Light Fields and Lumigraphs Panoramas and Concentric Mosaics Environment Matting Image-Based models
3	Today’s lecture Video-Based Rendering Facial animation Video matting and shadow matting Video Textures and Animating Stills Video-based tours
4	Readings S. J. Gortler , R. Grzeszczuk , R. Szeliski and M. F. Cohen, The Lumigraph, SIGGRAPH'96. M. Levoy and P. Hanrahan, Light field rendering, SIGGRAPH'96. H.-Y. Shum and L.-W. He. Rendering with concentric mosaics, SIGGRAPH’99.
5	Readings D. E. Zongker et al. Environment matting and compositing, SIGGRAPH'99. Y.-Y. Chuang et al. Environment matting extensions: Towards higher accuracy and real-time capture. SIGGRAPH'2000, pp.121-130, 2000. P. E. Debevec , C. J. Taylor and J. Malik, Modeling and rendering architecture from photographs:…, SIGGRAPH'96.
6	Readings Y.-Y. Chuang et al. Video matting of complex scenes. ACM Trans. on Graphics, 21(3):243-248, July 2002 Y.-Y. Chuang et al. Shadow matting. ACM Transactions on Graphics, 22(3):494-500, July 2003. A. Schödl et al., Video textures. SIGGRAPH'2000, pp. 489-498, 2000. M. Uyttendaele et al. Image-based interactive exploration of real-world environments. IEEE Comp. Graphics and Applications, 24(3), May/June 2004.
7	Lightfields and Lumigraphs (with lots of slides from Michael Cohen)
8	Modeling light How do we generate new scenes and animations from existing ones? Classic “3D Vision + Graphics”: take (lots of) pictures recover camera pose build 3D model extract texture maps / BRDFs synthesize new views
9	Computer Graphics
10	Computer Vision
11	Combined
12	But, vision technology falls short
13	… and so does graphics.
14	Image Based Rendering
15	Ray Constant radiance time is fixed 5D 3D position 2D direction
16	All Rays Plenoptic Function: all possible images too much stuff!
17	Line Infinite line 4D 2D direction 2D position non-dispersive medium
18	Ray Discretize, then interpolate Distance between 2 rays Which is closer together?
19	Image What is an image? All rays through a point Panorama
20	Panoramic Mosaics Convert panoramic image sequence into a cylindrical image + + … + =
21	Image Image plane 2D position in plane
22	Object Light leaving towards “eye” 2D just dual of image
23	Object All light leaving object
24
25	Object All images
26	Lumigraph / Lightfield Outside convex space 4D
27
28	Lumigraph How to ? organize capture render
29	Lumigraph - Organization 2D position 2D direction
30
31
32
33
34
35
36	Lumigraph - Rendering For each output pixel determine s,t,u,v either find closest discrete RGB interpolate near values
37
38	Lumigraph - Rendering Nearest closest s closest u draw it Blend 16 nearest quadrilinear interpolation
39	Lumigraph - Rendering Depth Correction closest s intersection with “object” best u closest u
40	Lumigraph - Rendering Depth Correction quadralinear interpolation new “closest” like focus [Dynamically Reparameterized Light Fields, Isaksen, SG’2000]
41	Lumigraph - Rendering Fast s,t,u,v finding scanline interpolate texture mapping shear warp
42	Lumigraph - Ray Space 3D space ray space surface depth Û slope in ray space
43	Lumigraph - Ray Space Image effects: parallax occlusion transparency highlights
44	Lumigraph - Demo Lumigraph Lion, Fruit Bowl
45	Complex Light Field acquisition Digital Michelangelo Project Marc Levoy, Stanford University Lightfield (“night”) assembled by Jon Shade
46	Unstructured Lumigraph What if the images aren’t sampled on a regular 2D grid? can still re-sample rays ray weighting becomes more complex [Buehler et al., SIGGRAPH’2000]
47	Surface Light Fields Turn 4D parameterization around: image @ every surface pt. Leverage coherence: compress radiance fn (BRDF * illumination) after rotation by n
48	Surface Light Fields [Wood et al, SIGGRAPH 2000]
49	3D Representations Image (and panoramas) are 2D Lumigraph is 4D What happened to 3D? 3D Lumigraph subset Concentric mosaics
50	3D Lumigraph One row of s,t plane i.e., hold t constant
51	3D Lumigraph One row of s,t plane i.e., hold t constant thus s,u,v a “row of images” [Sloan et al., Symp. I3DG 97]
52	Concentric Mosaics Replace “row” with “circle” of images [Shum & He, SIGGRAPH’97]
53	Concentric Mosaics
54	Concentric Mosaics Rendering ( as seen from above )
55
56
57	2.5D Representations Image is 2D Lumigraph is 4D 3D 3D Lumigraph subset Concentric mosaics 2.5D Layered Depth Images Sprites with Depth (impostors) View Dependent Surfaces (see Façade)
58	Layered Depth Image
59
60
61	Sprites with Depth Represent scene as collection of cutouts with depth (planes + parallax) Render back to front with fwd/inverse warping [Shade et al., SIGGRAPH’98]
62	Environment matting and compositing D. E. Zongker, D. M. Werner, B. Curless and D. H. Salesin. SIGGRAPH'99
63	Environment Matting Capture the reflections and refractions of a real-world object Composite object over a novel background
64	Environment Matting - examples
65	Environment Matting Capture the mapping from each image pixel to a real-world ray direction(s)
66	Acquisition setup Use several monitors with stripes
67	Environment matting equation Captures foreground color and background directions
68	Environment matting - result
69	Environment matting extensions [Chuang et al., SIGGRAPH’2001] accurate (multiple refractions): soft stripes fast (video rate): color ramp
70	Image-Based Modeling
71	Image Based Models
72	Image-Based Modeling Create 3D model (and texture maps) from images automated (structure from motion, stereo) interactive Façade system
73	Façade Select building blocks Align them in each image Solve for camera pose and block parameters (using constraints)
74	View-dependent texture mapping Determine visible cameras for each surface element Blend textures (images) depending on distance between original camera and novel viewpoint
75	Model-based stereo Compute offset from block model Some more results:
76	Image-Based Faces Estimate shape from images Match metrics to shape Project video onto shape Texture map Animate [Z. Liu et al., MSR-TR-2000-11]
77	Hierarchy of Light Fields [Levoy] 8D: Refractive/reflective environment 5D: Plenoptic Function (Ray) 4D: Lumigraph / Lightfield 4D*: Environment Matte (single view) 3D: Lumigraph Subset 3D: Concentric Mosaics 2.5D: Layered Depth Image 2.5D: Image Based Models 2D: Images and Panoramas
78	Graphics/Imaging Continuum
79	What lies beyond Image-Based Rendering?
80	Video-Based Rendering Image-Based Rendering: render from (real-world) images for efficiency, quality, and photo-realism Video-Based Rendering use video instead of still images for dynamic elements and source footage generate computer video instead of computer graphics
81	VBR Examples Facial animation Video Rewrite, … Layer/matte extraction Video Matting, … Dynamic (stochastic) elements Video Textures, … 3-D world navigation Image-Based Realities, …
82	Facial animation Modeling from still images [Pighin et al., SG’98] Lip-synching from video Video Rewrite [Bregler et al., SG’97] [Ezzat et al., SG’02]
83	Matting and Compositing
84
85	Video Matting Pull dynamic a-matte from video with complex backgrounds [Chuang et al. @ UW, SIGGRAPH’2002]
86	Video Matting
87	Shadow Matting Transfer a shadow from one background to another: Extract and model photometry (darkening) Extract and model geometry (deformation)
88	Shadow Matting
89	Shadow Matting
90	Shadow Matting Video
91	Video Textures
92	Video Textures How can we turn a short video clip into an ¥ amount of continuous video? dynamic elements in 3D games and presentations alternative to 3D graphics animation? [Schödl, Szeliski, Salesin, Essa, SG’2000]
93	Video Textures Find cyclic structure in the video (Optional) region-based analysis Play frames with random shuffle Smooth over discontinuities (morph)
94	Region-based analysis
95	Crossfading and morphing
96	Video portrait
97	Dynamic scene element Live waterfall in static panorama
98	Interactive fish
99	A complete animation
100	Video-Based Tours
101	Video-Based Walkthroughs Move camera along a rail (“dolly track”) and play back a 360° video Applications: Homes and architecture Outdoor locations (tourist destinations)
102	Surround video acquisition system
103	OmniCam Built by Point Grey Research (Ladybug) Six camera head Portable hard drives, fiber-optic link Resolution per image: 1024 x 768 FOV: ~100^o x ~80^o Acquisition speed: 15 fps uncompressed
104	Acquisition platforms Robotic cart Wearable
105	Demo A Virtual Home Tour
106	Open issues How to best sample and interpolate Light Field (sub-?) pixel accurate stereo reflections, refractions, … Compositing how to insert Light Field into new environment relighting …?
107	Summary Image–Based Rendering Light Fields and Lumigraphs Panoramas and Concentric Mosaics Matting: natural, environment, and shadows Image-Based models Video-Based Rendering Facial animation Video Textures and Animating Stills Video-based tours
108	Summary Image–Based Rendering Light Fields and Lumigraphs Panoramas and Concentric Mosaics Environment Matting Image-Based models Video-Based Rendering Facial animation Video matting Video Textures and Animating Stills Video-based tours