Computer Vision (CSE 490CV/EE 400B), Winter 2002

Project 2:  Panoramic Mosaic Stitching

Assigned:  Friday, Feb 8, 2002
Due:  Thursday, Feb 26, 2002 (by 11:59pm)

In this project, you will implement a system to combine a series of photographs into a 360° panorama.  Your software will automatically align the photographs (determine their overlap and relative positions) and then blend the resulting photos into a single seamless panorama.  You will then be able to view the resulting panorama inside an interactive Web viewer.  To start your project, you will be supplied with some test images, skeleton code you can use as the basis of your project, and instructions on how to use the viewer.

To see examples of previous students panoramas, click here.

Here are the suggested steps you should follow (skip steps 1 - 3 for test data):

1. Take a series of photos with a digital camera mounted on a tripod.  You should sign up to borrow the Kaidan head that lets you make precise rotations and the Canon PowerShot A10 camera for this purpose. For best results, overlap each image by 50% with the previous one, and keep the camera level using the levelers on the Kaidan head.
2. Also take a series of images with a handheld camera.  You can use your own or use the Canon PowerShot A10 camera that you signed up for. If you are using the Canon camera, it has a “stitch assist” mode you can use to overlap your images correctly, which only works in regular landscape mode.  If you are using your own camera, you have to estimate the focal length (Brett Allen describes one creative way to measure rough focal length using just a book and a box, or alternatively use a camera calibration toolkit to get precise focal length and radial distortion coefficients).  The parameters for the class cameras are given below.
3. Make sure the images are right side up (rotate the images by 90° if you took them in landscape mode), and reduce them to a more workable size (around 400x500--50% size for the course cameras). You can use external software such as PhotoShop or the Microsoft Photo Editor to do this.  If you are using the skeleton software, save your images in (TrueVision) Targa format (.tga), since this is the only format the skeleton software can currently read.
4. Warp each image into cylindrical coordinates.  The easiest way to do this is to flesh out the skeleton code in WarpCylindrical.cpp by adding the necessary coordinate transformation equations (from the lecture notes).  Warp each image using the warpCylindrical routine, using the following focal length f estimates for the half-resolution images (you can either take pictures and same them in small files or save them in large files and reduce them afterwards) . If you use a different image size, do remember to scale f according to the image size). The following focal length is valid only if the camera is zoomed out most.

   Camera	resolution	focal length	k1	k2
   Canon Powershot A10, tag CS30012716	480x640	679.09090 pixels	-0.21321	0.39741
   Canon Powershot A10, tag CS30012717	480x640	676.66016 pixels	-0.19098	0.21707
   Canon Powershot A10, tag CS30012718	480x640	675.69544 pixels	-0.20465	0.33038

5. Compute the alignment of the images in pairs.  To do this, you will have to implement a hierarchical (coarse-to-fine) Lucas-Kanade style translational motion estimation.  The skeleton for this code is provided in LucasKanade.cpp.  You will have to fill in the missing code in LucasKanade to:
   1. compute the per-pixel error and intensity gradients
   2. accumulate the 2x2 matrix and 2x1 vector
   3. solve the 2x2 system and update the translation estimate
   The control routine that uses the image pyramid to search for the best displacement at a coarse level, and then refines these estimates using Lucas-Kanade has already been written for you. Once you have computed each pair-wise translation (including the translation between the last image and the first one), write these numbers out to a file.
6. Stitch the resulting aligned images.  Read in a descriptor file containing the warped image names and their relative displacements.  Figure out how large the final stitched image will be.  Then, resample each image to its final location and blend it with its neighbors.  Try a simple horizontal “hat” function as your weighting function, similar to the one described in lecture (filtering slide 32) (this is a simple 1-D version of the distance map described in [Szeliski & Shum]).  For extra credit, you can try other blending functions or figure out some way to compensate for exposure differences.
7. Crop the resulting image to make the left and right edges seam perfectly.  The horizontal extent can be computed in the previous blending routine since the first image occurs at both the left and right end of the stitched sequence (draw the “cut” line halfway through this image).  Use a linear warp to the mosaic to remove any vertical “drift” between the first and last image.  This warp, of the form y' = y + ax, should transform the y coordinates of the mosaic such that the first image has the same y-coordinate on both the left and right end.  Calculate the value of 'a' needed to perform this transformation.
8. Convert your resulting image to a JPEG and paste it on a Web page along with code to run the interactive viewer. Click here for instructions on how to do this.

   Turn in the code that you wrote (just the .cpp files you modified and any new files you needed).  In the artifact directory, turn in a web page containing the following:

   • At least three panoramas:  (1) the test sequence, (2), one from the Kaidan head, and (3) one from a hand-held sequence.  Each panorama should be shown as (1) a low-res inlined image on the web page, (2) a link that you can click on to show the full-resolution .jpg file, AND (3) embedded in a viewer as described above.

   • a short description of what worked well and what didn’t.  If you tried several variants or did something non-standard, please describe this as well.

Bells and Whistles

Here is a list of suggestions for extending the program for extra credit. You are encouraged to come up with your own extensions. We're always interested in seeing new, unanticipated ways to use this program!

Sometimes, there exists exposure difference between images, which results in brightness fluctuation in the final mosaic. Try to get rid of this artifact. 

Make edits to one of your panoramas using your scissoring tool and photoshop.  Import it to a viewer and include as a fourth panorama on the artifact page.

Try shooting a sequence with some objects moving.  What did you do to remove “ghosted” versions of the objects?

Try a sequence in which the same person appears multiple times, as in this example.

Implement pyramid blending, as shown in class and described in Burt & Adelson's paper.