The following files are available for download. Although it may be more convenient to download one with everything.

• Feature detection executable from project 1
• Panorama stitching skeleton and its solution executable
• An example of panorama test images and algorithm outputs
• The 360° panorama viewer applet
• A template for your web deliverable

We have also added a .bat script which demonstrates running all commands for the test image set, and a pairlist.txt file for the same set which includes the expected alignPair results. Note that the .bat script assumes that it, features.exe, panorama.exe, and the target images are all present in the same folder. Note that when running this script on your own images, you will have to edit pairlist file yourself to put it your alignPair outputs.

If you're unfamiliar with .bat scripts, it's a text file containing line-delimited console commands. You can open it in Notepad or another text editor to see what's going on. It can be helpful to execute them from within a terminal session so that the execution window stays open after the script runs (Start+R, enter cmd).

You may also want to install IrfanView. This program is very helpful for opening and transcoding image formats (among other things). This article on its batch conversion feature may be helpful when converting your photographs to .tga format. (Photoshop users: be aware that it does some sanity checking when opening files which may fool you while debugging, specifically about whether your alpha channel is correct.)

Getting Things to Run

The project will consist of a pipeline of command line EXE programs (Feature.exe from project 1 and Panorama.exe) that will operate on images or intermediate results to produce the final panorama output. The steps required to complete the panorama are listed below:

	Step	EXE
1.	Calibrate your camera and take pictures
2.	Warp to spherical coordinates	(Panorama.exe)
3.	Extract features	(Features.exe)
4.	Match features	(Features.exe)
5.	Align neighboring pairs using RANSAC	(Panorama.exe)
6.	Write out list of neighboring translations	(Panorama.exe)
7.	Correct for drift	(Panorama.exe)
8.	Read in warped images and blend them	(Panorama.exe)
9.	Crop the result and import into a viewer

Running the sample solution

Features.exe and Panorama.exe are command line programs that requires arguments to work properly. Thus you need to run it from the command line, or from a shortcut to the executable that has the arguments specified in the "Target" field of the shortcut properties.  The one exception to this is that running Features.exe without command line arguments opens up the UI.

Running from the command line

To run from the command line, click the Windows Start button and select "Run". Then enter "cmd" in the "Run" dialog and click "OK". A command window will pop up where you can type DOS commands. Use the DOS "cd" (change directory) command to navigate to the directory where Features.exe or Panorama.exe is located. Then type "Features" or "Panorama" followed by your arguments. If you do not supply any arguments, the program will print out information on what arguments it expects or open the UI in the case of Features.exe.

Running from a shortcut

Another way to pass arguments to a program is to create a shortcut to it. To create a shortcut, right-click on the executable and drag to the location where you wish to place the shortcut. A menu will pop up when you let go of the mouse button. From the menu, select "Create Shortcut Here". Now right-click on the short-cut you've created and select "Properties". In the properties dialog select the "Shortcut" tab and add your arguments after the text in the "Target" field. Your arguments must be outside of the quotation marks and separated with spaces.

Running the skeleton program

You can run the skeleton program from inside Visual Studio. However, you will need to tell Visual Studio what arguments to pass. Here's how:

1. Select the "ImageLib" project in the Solution Explorer (do NOT select the "Panorama" project, for some reason this won't work).
2. From the "Project" menu choose "Properties" to bring up the "Property Pages" dialog.
3. Select the "Debugging" Property page.
4. Enter your arguments in the "Command Arguments" field.
5. Click "Ok".
6. Now when you execute your program from within Visual Studio the arguments you entered will be passed to it automatically.
7. Repeat the above steps for the solution for the Features component.

 

Calibrating the camera and taking photos

Calibrating the camera

When using your own camera, you have to estimate the focal length and radial distortion coefficients of your camera (see lecture week 3 lecure notes for details). One way to do this is to print one or several checkerboard patterns, take photos of them from different angles and use calibration software to calculate the desired coefficients. You will use GML Camera Calibration Toolbox, a windows GUI program for the last task. After installing and running it, follow the on-screen steps. The resulting parameters we are interested in are the focal length and the first two distortion parameters (k1 and k2 in the notes). You can print 5x8 checkerboard or 7x10 checkerboard patterns (which typically print to square sizes of 28mm and 19mm, respectively). Calibration quality improves if you print and include both of these patterns in your calibration photos. See this page for more tips on taking good calibration photos.

If you are having trouble getting consistent calibration numbers with GML: In GML toolkit GUI go to Calibration -> Calibration Type and set it to Native OpenCV and set Object Detection -> Detection Method OpenCV Native. This should give you more accurate and consistent calibration numbers. Alternatively, you can set k1 and k2 to 0 and use this guide on how to calculate focal length from your camera's exif tags.

If you are not using Windows, you may find this camera calibration page helpful in pointing you to alternatives. In particular, if you have free OpenCV library installed, you may follow this guide to calibration.

Taking photos

Take a series of photos with your digital camera. For best results, overlap each image by 50% with the previous one, and keep the camera level and rotating about a single axis. Make sure the images are right side up (rotate the images by 90° if you took them in landscape mode), and reduce them to a more workable size (480x640 recommended). You can use external software such as PhotoShop or Gimp to do this or you may want to set the camera to 640x480 resolution from the start.

Image formatting

• Your images need to be in .TGA format and have a 4:3 (or 3:4) aspect ratio in order to be compatible with the project skeleton.
• Your output panoramas need to be in .JPG format in order to be compatible with the Java viewer.
• Your input images should be kept reasonably small, e.g. 480x640. The computation time for larger images may be significant.

Skeleton Code ToDo

Note: The skeleton code includes an image library, ImageLib, that is fairly general and complex.  It is NOT necessary for you to peek extensively into this library!  We have created some notes for you here.

1. Warp each image into spherical coordinates. (file: WarpSpherical.cpp, routine: warpSphericalField)

[TODO] Compute the inverse map to warp the image by filling in the skeleton code in the warpSphericalField routine to:

1. convert the given spherical image coordinate into the corresponding planar image coordinate using the coordinate transformation equation from the lecture notes
2. apply radial distortion using the equation from the lecture notes

(Note: If you use an image size for your panorama photos that's different from image size for calibration photos, remember to scale f according to the panorama image size.)

2. Compute the alignment of the images in pairs. (file: FeatureAlign.cpp, routines: alignPair, countInliers, and leastSquaresFit)

To do this, you will have to implement a feature-based translational motion estimation.  The skeleton for this code is provided in FeatureAlign.cpp.  The main routines that you will be implementing are:

int alignPair(const FeatureSet &f1, const FeatureSet &f2, const vector<FeatureMatch> &matches, MotionModel m, float f, int nRANSAC, double RANSACthresh, CTransform3x3& M);

int countInliers(const FeatureSet &f1, const FeatureSet &f2, const vector<FeatureMatch> &matches, MotionModel m, float f, CTransform3x3 M, double RANSACthresh, vector<int> &inliers);

int leastSquaresFit(const FeatureSet &f1, const FeatureSet &f2, const vector<FeatureMatch> &matches, MotionModel m, float f, const vector<int> &inliers, CTransform3x3& M);

AlignPair takes two feature sets, f1 and f2, the list of feature matches obtained from the feature detecting and matching component (described in the first part of the project),  a motion model (described below), and estimates and inter-image transform matrix M.  For this project, the enum MotionModel only takes on the value eTranslate.

AlignPair uses RANSAC (RAndom SAmpling Consensus) to pull out a minimal set of feature matches (one match for this project), estimates the corresponding motion (alignment) and then invokes countInliers to count how many of the feature matches agree with the current motion estimate.  After repeated trials, the motion estimate with the largest number of inliers is used to compute a least squares estimate for the motion, which is then returned in the motion estimate M.

CountInliers computes the number of matches that have a distance below RANSACthresh is computed.  It also returns a list of inlier match ids.

LeastSquaresFit computes a least squares estimate for the translation using all of the matches previously estimated as inliers.  It returns the resulting translation estimate in the last column of M.

[TODO] You will have to fill in the missing code in alignPair to:

1. Randomly select a valid matching pair and compute the translation between the two feature locations.
2. Call countInliers to count how many matches agree with this estimate.
3. Repeat the above random selection nRANSAC times and keep the estimate with the largest number of inliers.
4. Write the body of countInliers to count the number of feature matches where the SSD distance after applying the estimated transform (i.e. the distance from the match to its correct position in the image) is below the threshold. (and don't forget to create the list of inlier ids.)
5. Write the body of leastSquaresFit, which for the simple translational case is just the average displacement between the matching feature positions.

3. Stitch and crop the resulting aligned images. (file: BlendImages.cpp, routines: BlendImages, AccumulateBlend, NormalizeBlend)
   
   [TODO] Given the warped images and their relative displacements, figure out how large the final stitched image will be and their absolute displacements in the panorama (BlendImages).

   [TODO] Then, resample each image to its final location and blend it with its neighbors (AccumulateBlend, NormalizeBlend). Try a simple feathering function as your weighting function (see mosaics lecture slide on "feathering") (this is a simple 1-D version of the distance map described in [Szeliski & Shum]).  For extra credit, you can try other blending functions or figure out some way to compensate for exposure differences. In NormalizeBlend, remember to set the alpha channel of the resultant panorama to opaque!

   [TODO] Crop the resulting image to make the left and right edges seam perfectly (BlendImages). The horizontal extent can be computed in the previous blending routine since the first image occurs at both the left and right end of the stitched sequence (draw the "cut" line halfway through this image).  Use a linear warp to the mosaic to remove any vertical "drift" between the first and last image.  This warp, of the form y' = y + ax, should transform the y coordinates of the mosaic such that the first image has the same y-coordinate on both the left and right end.  Calculate the value of 'a' needed to perform this transformation.

Creating the Panorama

Use the above program you wrote to warp/align/stitch images into the resulting panorama.

• To remove the radial distortion and warp the image input1.tga into spherical coordinate with focal length = 600, radial distortion coefficients k1=-0.21 and k2=0.25 (Panorama is the name of the program):
  
     Panorama sphrWarp input1.tga warp1.tga 600 -0.21 0.25
• Then, use the feature detecting and matching component to compute the features in the warped images. To align two feature sets warp1.f and warp2.f, with a set of matches in match1to2.txt, using 200 iterations of RANSAC with an outlier threshold distance of 1 pixel:
  
      Panorama alignPair warp1.f warp2.f match1to2.txt 200 1
• Run the previous step for all adjacent pairs of images and save the output into a separate file pairlist.txt which may look like this:
  
      warp1.tga warp2.tga 213.49 -5.12
      warp2.tga warp3.tga 208.19 2.82
      ......
      warp9.tga warp1.tga 194.76 -3.88
• Then stitch the images into the final panorama pano.tga:
  
      Panorama blendPairs pairlist.txt pano.tga 200

You may also refer to the file stitch.bat provided along with the skeleton code for the appropriate command line syntax. This command-line interface allows you to debug each stage of the program independently.

Debugging Guidelines