1	Image Alignment and Stitching Computer Vision CSE576, Spring 2005 Richard Szeliski
2	Today’s lecture Image alignment and stitching motion models cylindrical and spherical warping point-based alignment global alignment automated stitching (recognizing panoramas) ghost and parallax removal compositing and blending
3	Readings Szeliski & Shum, SIGGRAPH'97 (Sections 1-4). Szeliski, Image Alignment and Stitching, MSR-TR-2004-92 (Sections 2, 4, 5). Recognizing Panoramas, Brown & Lowe, ICCV’2003
4	Motion models
5	Motion models What happens when we take two images with a camera and try to align them? translation? rotation? scale? affine? perspective? … see interactive demo (VideoMosaic)
6	Motion models
7	Motion models
8	Homographies Perspective projection of a plane Lots of names for this: homography, texture-map, colineation, planar projective map Modeled as a 2D warp using homogeneous coordinates
9	Plane perspective mosaics 8-parameter generalization of affine motion works for pure rotation or planar surfaces Limitations: local minima slow convergence difficult to control interactively
10	Rotational mosaics Directly optimize rotation and focal length Advantages: ability to build full-view panoramas easier to control interactively more stable and accurate estimates
11	3D → 2D Perspective Projection
12	3D Rotation Model Projection equations Project from image to 3D ray (x₀,y₀,z₀) = (u₀-u_c,v₀-v_c,f) Rotate the ray by camera motion (x₁,y₁,z₁) = R₀₁ (x₀,y₀,z₀) Project back into new (source) image (u₁,v₁) = (fx₁/z₁+u_c,fy₁/z₁+v_c)
13	Image Mosaics (Stitching) [Szeliski & Shum, SIGGRAPH’97] [Szeliski, MSR-TR-2004-92]
14	Image Mosaics (Stitching)
15	Image Mosaics (stitching) Blend together several overlapping images into one seamless mosaic (composite) + + … + =
16	Mosaics for Video Coding Convert masked images into a background sprite for content-based coding + + + =
17	Establishing correspondences Direct method: Use generalization of affine motion model [Szeliski & Shum ’97] Feature-based method Compute feature-based correspondence [Lowe ICCV’99; Schmid ICCV’98, Brown&Lowe ICCV’2003] Compute R from correspondences (absolute orientation)
18	Stitching demo
19	Panoramas What if you want a 360° field of view?
20	Cylindrical panoramas Steps Reproject each image onto a cylinder Blend Output the resulting mosaic
21	Cylindrical Panoramas Map image to cylindrical or spherical coordinates need known focal length
22	Cylindrical projection Map 3D point (X,Y,Z) onto cylinder
23	Cylindrical warping Given focal length f and image center (x_c,y_c)
24	Spherical warping Given focal length f and image center (x_c,y_c)
25	3D rotation Rotate image before placing on unrolled sphere
26	Radial distortion Correct for “bending” in wide field of view lenses
27	Fisheye lens Extreme “bending” in ultra-wide fields of view
28	Inverse Warping Get each pixel I₀(u₀) from its corresponding location u₁ = h(u₀) in I₁(u₁)
29	Image Stitching Align the images over each other camera pan ↔ translation on cylinder! Blend the images together (demo)
30	Project 2 – image stitching Take pictures on a tripod (or handheld) Warp images to spherical coordinates Extract features Align neighboring pairs using RANSAC Write out list of neighboring translations Correct for drift Read in warped images and blend them Crop the result and import into a viewer
31	Matching features
32	RAndom SAmple Consensus
33	RAndom SAmple Consensus
34	Least squares fit
35	Assembling the panorama Stitch pairs together, blend, then crop
36	Problem: Drift Error accumulation small (vertical) errors accumulate over time apply correction so that sum = 0 (for 360° pan.)
37	Full-view (360° spherical) panoramas
38	Full-view Panoramas
39	Global alignment Register all pairwise overlapping images Use a 3D rotation model (one R per image) Use feature based registration of unwarped images Discover which images overlap other images using feature selection (RANSAC) Chain together inter-frame rotations Optimize all R estimates together (next time)
40	3D Rotation Model Projection equations Project from image to 3D ray (x₀,y₀,z₀) = (u₀-u_c,v₀-v_c,f) Rotate the ray by camera motion (x₁,y₁,z₁) = R₀₁ (x₀,y₀,z₀) Project back into new (source) image (u₁,v₁) = (fx₁/z₁+u_c,fy₁/z₁+v_c)
41	Absolute orientation [Arun et al., PAMI 1987] [Horn et al., JOSA A 1988] Procrustes Algorithm [Golub & VanLoan] Given two sets of matching points, compute R p_i’ = R p_iwith 3D rays p_i = (x_i,y_i,z_i) = (u_i-u_c,v_i-v_c,f) A = Σ_ip_i p_i’^T = Σ_ip_i p_i^T R^T = U S V^T= (U S U^T)R^T V^T= U^TR^T R = VU^T
42	Stitching demo
43	Texture Mapped Model (sphere)
44	Texture Mapped Model (cubical)
45	Recognizing Panoramas Matthew Brown & David Lowe ICCV’2003
46	Recognizing Panoramas
47	Finding the panoramas
48	Finding the panoramas
49	Finding the panoramas
50	Finding the panoramas
51	Fully automated 2D stitching
52	Get you own copy!
53	System components Feature detection and description more uniform point density Fast matching (hash table) RANSAC filtering of matches Intensity-based verification Incremental bundle adjustment [Brown, Szeliski, Winder, CVPR’05]
54	Probabilistic Feature Matching
55	RANSAC motion model
56	RANSAC motion model
57	RANSAC motion model
58	Probabilistic model for verification
59	How well does this work? Test on 100s of examples…
60	How well does this work? Test on 100s of examples… …still too many failures (5-10%) for consumer application
61	Matching Mistakes: False Positive
62	Matching Mistakes: False Positive
63	Matching Mistakes: False Negative Moving objects: large areas of disagreement
64	Matching Mistakes Accidental alignment repeated / similar regions Failed alignments moving objects / parallax low overlap “feature-less” regions (more variety?) No 100% reliable algorithm?
65	How can we fix these? Tune the feature detector Tune the feature matcher (cost metric) Tune the RANSAC stage (motion model) Tune the verification stage Use “higher-level” knowledge e.g., typical camera motions → Sounds like a big “learning” problem Need a large training/test data set (panoramas)
66	Deghosting and blending (optional material)
67	Local alignment (deghosting) Use local optic flow to compensate for small motions [Shum & Szeliski, ICCV’98]
68	Local alignment (deghosting) Use local optic flow to compensate for radial distortion [Shum & Szeliski, ICCV’98]
69	Image feathering Weight each image proportional to its distance from the edge (distance map [Danielsson, CVGIP 1980] Cut out the appropriate region from each image and then blend together
70	Region-based de-ghosting Select only one image in regions-of-difference using weighted vertex cover [Uyttendaele et al., CVPR’01]
71	Region-based de-ghosting Select only one image in regions-of-difference using weighted vertex cover [Uyttendaele et al., CVPR’01]
72	Cutout-based de-ghosting Select only one image per output pixel, using spatial continuity Blend across seams using gradient continuity (“Poisson blending”) [Agarwala et al., SG’2004]
73	Cutout-based compositing Photomontage [Agarwala et al., SG’2004] Interactively blend different images: group portraits
74	Cutout-based compositing Photomontage [Agarwala et al., SG’2004] Interactively blend different images: focus settings
75	Cutout-based compositing Photomontage [Agarwala et al., SG’2004] Interactively blend different images: people’s faces
76	Final thought: What is a “panorama”? Tracking a subject Repeated (best) shots Multiple exposures “Infer” what photographer wants?