Computer Vision (CSE 576), Spring 2003

Project 2:  Panoramic Mosaic Stitching

Assigned:  Monday April 21, 2003
Due: Sunday May 4th, 2003, artifacts due Tuesday May 6 (both by 11:59pm)

In this project, you will implement a system to combine a series of photographs into a 360° panorama.  Your software will automatically align the photographs (determine their overlap and relative positions) and then blend the resulting photos into a single seamless panorama.  You will then be able to view the resulting panorama inside an interactive Web viewer.  To start your project, you will be supplied with some test images and skeleton code you can use as the basis of your project, a sample solution executable you can use to compare with your program, and instructions on how to use the viewer.

Due to the large number of students in this class, we ask students to form groups of two to take pictures that will form panoramas.  This will involve signing out camera equipment as groups of two.  You MUST WORK INDIVIDUALLY when writing the code; but you will turn in one artifact per team of two.

To see examples of previous students panoramas, click here.

Running the sample solution

Running from the command line

Running from a shortcut

Running the skeleton program

1. Select the "ImageLib" project in the Solution Explorer (do NOT select the "v4gP1" project, for some reason this won't work).
2. From the "Project" menu choose "Properties" to bring up the "Property Pages" dialog.
3. Select the "Debugging" Property page.
4. Enter your arguments in the "Command Arguments" field.
5. Click "Ok".
6. Now when you execute your program from within Visual Studio the arguments you entered will be passed to it automatically.

Description

Here are the suggested steps you should follow:

Taking the Pictures

Skip this step for the test data. Its camera parameters can be found in the sample commands in stitch2.txt, which is provided along with the skeleton code.

1. Take a series of photos with a digital camera mounted on a tripod. Here is a web page explaining how to use the equipments. Please read it before you go out to shoot. Then you should sign up to borrow the Kaidan head that lets you make precise rotations and the Canon PowerShot A10 camera for this purpose. For best results, overlap each image by 50% with the previous one, and keep the camera level using the levelers on the Kaidan head.
   
   
2. Also take a series of images with a handheld camera.  You can use your own or use the Canon PowerShot A10 camera that you signed up for. If you are using the Canon camera, it has a “stitch assist” mode you can use to overlap your images correctly, which only works in regular landscape mode.  If you are using your own camera, you have to estimate the focal length (Brett Allen describes one creative way to measure rough focal length using just a book and a box, or alternatively use a camera calibration toolkit to get precise focal length and radial distortion coefficients).  The parameters for the class cameras are given below. The following focal length is valid only if the camera is zoomed out most.
   
   

   Camera	resolution	focal length	k1	k2
   Canon Powershot A10, tag CS30012716	480x640	678.21239 pixels	-0.21001	0.26169
   Canon Powershot A10, tag CS30012717	480x640	677.50487 pixels	-0.20406	0.23276
   Canon Powershot A10, tag CS30012718	480x640	676.48417 pixels	-0.20845	0.25624
   Canon Powershot A10, tag CS30012927	480x640	671.16649 pixels	-0.19270	0.14168
   Canon Powershot A10, tag CS30012928	480x640	674.82258 pixels	-0.21528	0.30098
   Canon Powershot A10, tag CS30012929	480x640	674.79106 pixels	-0.21483	0.32286
   test images	384x512	595 pixels	-0.15	0.0

    

3. Make sure the images are right side up (rotate the images by 90° if you took them in landscape mode), and reduce them to a more workable size (480x640 recommended). You can use external software such as PhotoShop or the Microsoft Photo Editor to do this. Or you may want to set the camera to 640x480 resolution from the start, by following the steps below:
   
   1. Turn the mode dial on the back of the camera to one of the 3 shooting modes--auto (camera icon), manual (camera icon + M) or stitch assist (overlaid rectangles).
   2. Press MENU button.
   3. Press the left/right arrow to choose Resolution, then press SET.
   4. Press the left/right arrow and choose S (640x480).
   5. Press MENU again.

   (Note: If you are using the skeleton software, save your images in (TrueVision) Targa format (.tga), since this is the only format the skeleton software can currently read. Also make sure the aspect ratio of the image (width vs. height) is either 4:3 or 3:4 (480x640 will do) which is the only aspect ratio supported by the skeleton software. Finally, ensure that the image is saved in 24bit tga, not 32bit. This is very important for Lucas-Kanade to work.)

Writing the Code

Note: The skeleton code includes an image library, ImageLib, that is fairly general and complex.  It is NOT necessary for you to peek extensively into this library!  We have created some notes for you here.

1. Warp each image into cylindrical coordinates. (file: WarpCylindrical.cpp, routine: warpCylindricalField)

   [TODO] Compute the inverse map to warp the image by filling in the skeleton code in the warpCylindricalField routine to:
   

   1. convert the given cylindrical image coordinate into the corresponding planar image coordinate using the coordinate transformation equation from the lecture notes (mosaics slide 12, 13)
   2. apply radial distortion using the equation from the lecture notes (projection slide 25)

   (Note: You will have to use the focal length f estimates for the half-resolution images provided above (you can either take pictures and save them in small files or save them in large files and reduce them afterwards) . If you use a different image size, do remember to scale f according to the image size.)
   
   

2. Compute the alignment of the images in pairs. (file: LucasKanade.cpp, routine: PyramidalLucasKanade, LucasKanadeStep)

    To do this, you will have to implement a hierarchical (coarse-to-fine) Lucas-Kanade style translational motion estimation.  The skeleton for this code is provided in LucasKanade.cpp.

   PyramidalLucasKanade constructs an image pyramid of specified number of levels for two images img1 and img2, and walks down the hierarchy (from coarse to fine), at each level updating the displacement of img2 from img1 by iteratively calling LucasKanadeStep.

   LucasKanadeStep takes two images img1, img2, and the initial translation vector (u,v) as input, and computes an updated translation (u',v') = (u+du,v+dv) which minimizes |img2(x+u',y+v')-img1(x,y)| over all x, y. Note that, instead of evaluating the image derivatives (Ix, Iy and It) between img2(x+u,y+v) and img1(x,y), it first creates img2t, a warp of img2 using the translation (u,v), then computes the image derivatives between img2t(x,y) and img1(x,y).

   [TODO] First, write a loop in PyramidalLucasKanade which walks through the image pyramid from coarse to fine and calls the LucasKanadeStep routine to update the displacement at each level of the pyramid (motion slide 26). More specifically:
   

   1. construct an image pyramid of specified number of levels n (already written in the skeleton code)
   2. for each level l of the pyramid, proceeding from the coarsest level (n-1) to the finest (0):
      1. update the translation vector between the two images at level l of pyramid by calling LucasKanadeStep specified number of times
      2. initialize the translation vector at level l-1 by scaling translation vector at level l

   [TODO] Then, you will have to fill in the missing code in LucasKanadeStep to:
   

   1. compute the per-pixel error and intensity gradients
   2. accumulate the 2x2 matrix and 2x1 vector
   3. solve the 2x2 system and update the translation estimate (motion slide 13)

    

3. Stitch and crop the resulting aligned images. (file: BlendImages.cpp, routines: BlendImages, AccumulateBlend, NormalizeBlend)
   
   [TODO] Given the warped images and their relative displacements, figure out how large the final stitched image will be and their absolute displacements in the panorama (BlendImages).

   [TODO] Then, resample each image to its final location and blend it with its neighbors (AccumulateBlend, NormalizeBlend). Try a simple horizontal “hat” function as your weighting function, similar to the one described in lecture (sampling slide 12) (this is a simple 1-D version of the distance map described in [Szeliski & Shum]).  For extra credit, you can try other blending functions or figure out some way to compensate for exposure differences. In NormalizeBlend, remember to set the alpha channel of the resultant panorama to opaque!

   [TODO] Crop the resulting image to make the left and right edges seam perfectly (BlendImages). The horizontal extent can be computed in the previous blending routine since the first image occurs at both the left and right end of the stitched sequence (draw the “cut” line halfway through this image).  Use a linear warp to the mosaic to remove any vertical “drift” between the first and last image.  This warp, of the form y' = y + ax, should transform the y coordinates of the mosaic such that the first image has the same y-coordinate on both the left and right end.  Calculate the value of 'a' needed to perform this transformation.

Creating the Panorama

1. Use the above program you wrote to warp/align/stitch images into the resulting panorama.
   
   • To remove the radial distortion and warp the image input1.tga into cylindrical coordinate with focal length = 600, radial distortion coefficients k1=-0.21 and k2=0.25 (v4gP1 is the name of the program):
     
         v4gP1 cylWarp input1.tga warp1.tga 600 -0.21 0.25
     
     
   • To align two images warp1.tga and warp2.tga which have a displacement of ~200 pixels horizontal and ~0 vertical, with 4 levels of image pyramid and 3 steps of Lucas-Kanade search at each level:
     
         v4gP1 LucasKanade warp1.tga warp2.tga 200 0 4 3
     
     
   • Run the previous step for all adjacent pairs of images and save the output into a separate file pairlist.txt which may look like this:
     
         warp1.tga warp2.tga 213.49 -5.12
         warp2.tga warp3.tga 208.19 2.82
         ......
         warp9.tga warp1.tga 194.76 -3.88
     
     
   • Then stitch the images into the final panorama pano.tga:
     
         v4gP1 blendPairs pairlist.txt pano.tga
     

   You may also refer to the file stitch2.txt provided along with the skeleton code for the appropriate command line syntax. This command-line interface allows you to debug each stage of the program independently.
   
   

2. Convert your resulting image to a JPEG and paste it on a Web page along with code to run the interactive viewer. Click here for instructions on how to do this.

Debugging Guidelines

You can use the test results included in the images/ folder to make sure whether your program is running correctly. Comparing your output to that of the sample solution is also a good way of debugging your program.

1. Testing the warping routines:
   
   
   • In the images/ folder in the skeleton code, a few example warped images are provided for test purposes. The camera parameters used for these examples can be found in the sample command file stitch2.txt. See if your program produces the same output.
   • You may also test with different input images and/or camera parameter values by comparing the results with those of the sample solution.
     
     
2. Testing the alignment routines:
   
   
   • A few example alignment results are provided in the file pairlist2/4.txt. The corresponding shell commands can be found in stitch2/4.txt.
   • To test LucasKanadeStep only, try passing 1 as the height of pyramid (i.e., no image pyramid) in the command line and compare the output of your program with that of the sample solution.
   • You may also want to print out the translation vector at each iteration of LucasKanadeStep (as the sample solution does).
     
     
3. Testing the blending routines:
   
   
   • An example panorama is included in the images/ folder. Compare the resulting panorama with this image.
   • You may also test with other panoramas by running the sample solution on different inputs.

What to Turn in

1. Turn in the executable (v4gP1.exe).

2. Turn in the code that you wrote (just the .cpp files you modified and any new files you needed).

3. In the artifact directory, turn in a web page (here are some tips) containing the following:

• At least three panoramas:  (1) the test sequence, (2), one from the Kaidan head, and (3) one from a hand-held sequence.  Each panorama should be shown as (1) a low-res inlined image on the web page, (2) a link that you can click on to show the full-resolution .jpg file, AND (3) embedded in a viewer as described above.

• a short description of what worked well and what didn’t.  If you tried several variants or did something non-standard, please describe this as well.

Bells and Whistles

Here is a list of suggestions for extending the program for extra credit. You are encouraged to come up with your own extensions. We're always interested in seeing new, unanticipated ways to use this program!

Although Lukas-Kanade gives sub-pixel motion estimation, the motion vectors are rounded to integers when blending the images into the mosaic in BlendImages.cpp. Try to blend images with sub-pixel localization. 

Sometimes, there exists exposure difference between images, which results in brightness fluctuation in the final mosaic. Try to get rid of this artifact. 

Make edits to one of your panoramas using your scissoring tool and photoshop.  Import it to a viewer and include as a fourth panorama on the artifact page.

Try shooting a sequence with some objects moving.  What did you do to remove “ghosted” versions of the objects?

Try a sequence in which the same person appears multiple times, as in this example.

Implement pyramid blending, as shown in class and described in Burt & Adelson's paper.

Implement and create full-view panoramas, such as spherical or cubic. This is discussed more in the assigned reading.

Panorama Links

• Panoramas.dk: weekly archive of full-screen, high-quality panoramas worldwide
• VR Seattle: Seattle & Washington panorama
• Peru Panoramas

Last modified on April 17, 2003