Computer Vision (CSE 490CV/EE 400B), Winter 2002

Project 2:  Panoramic Mosaic Stitching

Assigned:  Friday, Feb 8, 2002
Due:  Thursday, Feb 26, 2002 (by 11:59pm)

In this project, you will implement a system to combine a series of photographs into a 360° panorama.  Your software will automatically align the photographs (determine their overlap and relative positions) and then blend the resulting photos into a single seamless panorama.  You will then be able to view the resulting panorama inside an interactive Web viewer.  To start your project, you will be supplied with some test images, skeleton code you can use as the basis of your project, and instructions on how to use the viewer.

Here are the suggested steps you should follow (skip steps 1 - 3 for test data):

  1. Take a series of photos with a digital camera mounted on a tripod.  You should sign up to borrow the Kaidan head that lets you make precise rotations and the Canon PowerShot A10 Camera for this purpose. For best results, overlap each image by 50% with the previous one, and keep the camera level using the levelers on the Kaidan head.
  2. Also take a series of images with a handheld camera.  You can use your own or sign up to borrow the Canon PowerShot A10 camera from Cris Mesling (mesling@cs.washington.edu). If you are using the Canon camera, it has a “stitch assist” mode you can use to overlap your images correctly, which only works in regular landscape mode.
  3. Make sure the images are right side up (rotate the images by 90° if you took them in landscape mode), and downsample them to a more workable size (around 400x500--50% size for the course cameras). You can use external software such as PhotoShop or the Microsoft Photo Editor to do this.  If you are using the skeleton software, save your images in (TrueVision) Targa format (.tga), since this is the only format the skeleton software can currently read.
  4. Warp each image into cylindrical coordinates.  The easiest way to do this is to flesh out the skeleton code in WarpCylindrical.cpp by adding the necessary coordinate transformation equations (from the lecture notes).  Warp each image using the warpCylindrical routine, using the following focal length f estimates for the half-resolution images (you can either take pictures and same them in small files or save them in large files and downsample them afterwards) . If you use other image size, do remember to scale f with image size):
    Camera resolution focal length k1 k2
    Canon Powershot A1, tag CS30012716 480x640 ??? pixels ??? ???
             
             
  5. Compute the alignment of the images in pairs.  To do this, you will have to implement a hierarchical (coarse-to-fine) Lucas-Kanade style translational motion estimation.  The skeleton for this code is provided in LucasKanade.cpp.  You will have to fill in the missing code in LucasKanade to:
      1. compute the per-pixel error and intensity gradients
      2. accumulate the 2x2 matrix and 2x1 vector
      3. solve the 2x2 system and update the translation estimate
    The control routine that uses the image pyramid to search for the best displacement at a coarse level, and then refines these estimates using Lucas-Kanade has already been written for you. Once you have computed each pair-wise translation (including the translation between the last image and the first one), write these numbers out to a file.
  6. Stitch the resulting aligned images.  Read in a descriptor file containing the warped image names and their relative displacements.  Figure out how large the final stitched image will be.  Then, resample each image to its final location and blend it with its neighbors.  Try a simple horizontal “hat” function as your weighting function (this is a simple 1-D version of the distance map described in [Szeliski & Shum]).  For extra credit, you can try other blending functions or figure out some way to compensate for exposure differences.
  7. Crop the resulting image to make the left and right edges seam perfectly.  The horizontal extent can be computed in the previous blending routine since the first image occurs at both the left and right end of the stitched sequence (draw the “cut” line halfway through this image).  Use an affine warp to remove any vertical “drift” between the first and last image. Optionally, remove any black (a = 0) pixels at the top and bottom edges.
  8. Convert your resulting image to a JPEG and paste it on a Web page along with code to run the interactive viewer. For the instruction on how to do this, click here.

    Turn in the code that you wrote (just the .cpp files you modified and any new files you needed), and a web page containing the following:

Extra credit:  You can try some of the following for extra credit:

  1. Try shooting a sequence with some people moving in it.  What did you do to remove “ghosted” versions of the people?
  2. Try stitching something other than a cylindrical panorama, e.g., seam together two scans from a flatbed scanner.  Warning:  this is a fair amount of work, since your alignment code now needs to estimate rotations or even more.
  3. Detect and remove moving objects from your panorama, filling in the background behind them.

To download the skeleton code and start working on it, click here.