1	Global Alignment and Structure from Motion CSE576, Spring 2009 Sameer Agarwal
2	Overview Global refinement for Image stitching Camera calibration Pose estimation and Triangulation Structure from Motion
3	Readings Chapter 3, Noah Snavely’s thesis Supplementary readings: Hartley & Zisserman, Multiview Geometry, Appendices 5 and 6. Brown & Lowe, Recognizing Panoramas, ICCV 2003
4	Problem: Drift
5	Global optimization Minimize a global energy function: What are the variables? The translation t_j = (x_j, y_j) for each image What is the objective function? We have a set of matched features p_i,j = (u_i,j, v_i,j) For each point match (p_i,j, p_i,j+1): p_i,j+1 – p_i,j = t_j+1 – t_j
6	Global optimization
7	Global optimization
8	Global optimization
9	Ambiguity in global location Each of these solutions has the same error Called the gauge ambiguity Solution: fix the position of one image (e.g., make the origin of the 1^st image (0,0))
10	Solving for rotations
11	Solving for rotations
12	Parameterizing rotations How do we parameterize R and ΔR? Euler angles: bad idea quaternions: 4-vectors on unit sphere Axis-angle representation (Rodriguez Formula)
13	Nonlinear Least Squares
14	Camera Calibration
15	Camera calibration Determine camera parameters from known 3D points or calibration object(s) internal or intrinsic parameters such as focal length, optical center, aspect ratio: what kind of camera? external or extrinsic (pose) parameters: where is the camera? How can we do this?
16	Camera calibration – approaches Possible approaches: linear regression (least squares) non-linear optimization vanishing points multiple planar patterns panoramas (rotational motion)
17	Image formation equations
18	Calibration matrix Is this form of K good enough? non-square pixels (digital video) skew radial distortion
19	Camera matrix Fold intrinsic calibration matrix K and extrinsic pose parameters (R,t) together into a camera matrix M = K [R \| t ] (put 1 in lower r.h. corner for 11 d.o.f.)
20	Camera matrix calibration Directly estimate 11 unknowns in the M matrix using known 3D points (X_i,Y_i,Z_i) and measured feature positions (u_i,v_i)
21	Camera matrix calibration Linear regression: Bring denominator over, solve set of (over-determined) linear equations. How? Least squares (pseudo-inverse) Is this good enough?
22	Camera matrix calibration Advantages: very simple to formulate and solve can recover K [R \| t] from M using QR decomposition [Golub & VanLoan 96] Disadvantages: doesn't compute internal parameters can give garbage results more unknowns than true degrees of freedom need a separate camera matrix for each new view
23	Multi-plane calibration Use several images of planar target held at unknown orientations [Zhang 99] Compute plane homographies Solve for K^-TK^-1 from H_k’s 1 plane if only f unknown 2 planes if (f,u_c,v_c) unknown 3+ planes for full K Code available from Zhang and OpenCV
24	Pose estimation and triangulation
25	Pose estimation Use inter-point distance constraints [Quan 99][Ameller 00] Solve set of polynomial equations in x_i^2p Recover R,t using procrustes analysis.
26	Triangulation Problem: Given some points in correspondence across two or more images (taken from calibrated cameras), {(u_j,v_j)}, compute the 3D location X
27	Triangulation Method I: intersect viewing rays in 3D, minimize: X is the unknown 3D point C_j is the optical center of camera j V_j is the viewing ray for pixel (u_j,v_j) s_j is unknown distance along V_j Advantage: geometrically intuitive
28	Triangulation Method II: solve linear equations in X advantage: very simple Method III: non-linear minimization advantage: most accurate (image plane error)
29	Structure from Motion
30	Structure from motion Given many points in correspondence across several images, {(u_ij,v_ij)}, simultaneously compute the 3D location x_i and camera (or motion) parameters (K, R_j, t_j) Two main variants: calibrated, and uncalibrated (sometimes associated with Euclidean and projective reconstructions)
31	Orthographic SFM [Tomasi & Kanade, IJCV 92]
32	Extensions Paraperspective [Poelman & Kanade, PAMI 97] Sequential Factorization [Morita & Kanade, PAMI 97] Factorization under perspective [Christy & Horaud, PAMI 96] [Sturm & Triggs, ECCV 96] Factorization with Uncertainty [Anandan & Irani, IJCV 2002]
33	SfM objective function Given point x and rotation and translation R, t Minimize sum of squared reprojection errors:
34	Scene reconstruction
35	Feature detection
36	Feature detection
37	Feature detection Detect features using SIFT [Lowe, IJCV 2004]
38	Feature matching Match features between each pair of images
39	Feature matching
40	Reconstruction Choose two/three views to seed the reconstruction. Add 3d points via triangulation. Add cameras using pose estimation. Bundle adjustment Goto step 2.
41	Two-view structure from motion Simpler case: can consider motion independent of structure Let’s first consider the case where K is known Each image point (u_i,j, v_i,j, 1) can be multiplied by K^-1 to form a 3D ray We call this the calibrated case
42	Notes on two-view geometry How can we express the epipolar constraint? Answer: there is a 3x3 matrix E such that p'^TEp = 0 E is called the essential matrix
43	Properties of the essential matrix
44	Properties of the essential matrix p'^TEp = 0 Ep is the epipolar line associated with p e and e' are called epipoles: Ee = 0 and E^Te' = 0 E can be solved for with 5 point matches see Nister, An efficient solution to the five-point relative pose problem. PAMI 2004.
45	The Fundamental matrix If K is not known, then we use a related matrix called the Fundamental matrix, F Called the uncalibrated case F can be solved for linearly with eight points, or non-linearly with six or seven points
46	Photo Tourism overview