Assigned: Wednesday, April 16, 2008
Due: Tuesday, April 29, 2008 (by 11:59pm)
Artifact Due: Thursday, May 1, 2008 (by 11:59pm)
***Sign-up for Panorama kit here.
In this project, you will implement a system to combine a series of photographs into a 360° panorama (see panorama below). Your software will detect discriminating features in the images, find the best matching features in the other images, automatically align the photographs (determine their overlap and relative positions) and then blend the resulting photos into a single seamless panorama. You will use your project 1 code (i.e., Features.exe) to detect discriminating features in the images, find the best matching features in the other images and so you main focus here is to automatically align the photographs (determine their overlap and relative positions) and then blend the resulting photos into a single seamless panorama. You will then be able to view the resulting panorama inside an interactive Web viewer. To start your project, you will be supplied with some test images and skeleton code you can use as the basis of your project and instructions on how to use the viewer.
The project will consist of a pipeline of command line EXE programs
(Feature.exe and Panorama.exe) that will operate on images or intermediate
results to produce the final panorama output. A complete description of
each component can be found on the dedicated component web page.
The steps required to complete the panorama are listed below:
|
Step |
EXE |
1. |
Take pictures on a tripod (or handheld) |
|
2. |
Warp to spherical coordinates |
(Panorama.exe) |
3. |
Extract features |
(Features.exe) |
4. |
Match features |
(Features.exe) |
5. |
Align neighboring pairs using RANSAC |
(Panorama.exe) |
6. |
Write out list of neighboring translations |
(Panorama.exe) |
7. |
Correct for drift |
(Panorama.exe) |
8. |
Read in warped images and blend them |
(Panorama.exe) |
9. |
Crop the result and import into a viewer |
|
Features.exe and Panorama.exe are command line programs that requires arguments to work properly. Thus you need to run it from the command line, or from a shortcut to the executable that has the arguments specified in the "Target" field of the shortcut properties. The one exception to this is that running Features.exe without command line arguments opens up the UI.
To run from the command line, click the Windows Start button and select "Run". Then enter "cmd" in the "Run" dialog and click "OK". A command window will pop up where you can type DOS commands. Use the DOS "cd" (change directory) command to navigate to the directory where Features.exe or Panorama.exe is located. Then type "Features" or "Panorama" followed by your arguments. If you do not supply any arguments, the program will print out information on what arguments it expects or open the UI in the case of Features.exe.
Another way to pass arguments to a program is to create a shortcut to it. To create a shortcut, right-click on the executable and drag to the location where you wish to place the shortcut. A menu will pop up when you let go of the mouse button. From the menu, select "Create Shortcut Here". Now right-click on the short-cut you've created and select "Properties". In the properties dialog select the "Shortcut" tab and add your arguments after the text in the "Target" field. Your arguments must be outside of the quotation marks and separated with spaces.
You can run the skeleton program from inside Visual Studio. However, you will need to tell Visual Studio what arguments to pass. Here's how:
You will use the feature detection and matching component to combine a series of photographs into a 360° panorama. Your software will automatically
align the photographs (determine their overlap and relative positions) and then
blend the resulting photos into a single seamless panorama. You will then be
able to view the resulting panorama inside an interactive Web viewer. To start
this component, you will be supplied with some test
images and skeleton code that will guide you (the main solution file in
Visual Studio is Features.sln). In addition, a
solution executable can be found for
Windows or for
Linux.
You can download the test images separately here.
Taking the Pictures
ToDo
Creating the Panorama
Debugging
Extra Credit
Panorama Links
You will be checking out a panorama kit (camera, tripod, and Kaidan head). Remember to bring extra batteries with you, these cameras drain batteries.
Skip this step for the test data. Its camera parameters can be found in the sample commands in stitch4.txt, which is provided along with the skeleton code.
Camera |
resolution |
focal length |
k1 |
k2 |
Canon Powershot A10, tag CS30012716 |
480x640 |
678.21239 pixels |
-0.21001 |
0.26169 |
Canon Powershot A10, tag CS30012717 |
480x640 |
677.50487 pixels |
-0.20406 |
0.23276 |
Canon Powershot A10, tag CS30012718 |
480x640 |
676.48417 pixels |
-0.20845 |
0.25624 |
Canon Powershot A10, tag CS30012927 |
480x640 |
671.16649 pixels |
-0.19270 |
0.14168 |
Canon Powershot A10, tag CS30012928 |
480x640 |
674.82258 pixels |
-0.21528 |
0.30098 |
Canon Powershot A10, tag CS30012929 |
480x640 |
674.79106 pixels |
-0.21483 |
0.32286 |
test images |
384x512 |
595 pixels |
-0.15 |
0.0 |
(Note: If you are using the skeleton software, save your images in (TrueVision) Targa format (.tga), since this is the only format the skeleton software can currently read. Also make sure the aspect ratio of the image (width vs. height) is either 4:3 or 3:4 (480x640 will do) which is the only aspect ratio supported by the skeleton software.)
Note: The skeleton code includes an image library, ImageLib, that is fairly general and complex. It is NOT necessary for you to peek extensively into this library! We have created some notes for you here.
[TODO] Compute the inverse map to warp the image by filling in the skeleton code in the warpSphericalField routine to:
(Note: You will have to use the focal length f estimates for the half-resolution images provided above (you can either take pictures and save them in small files or save them in large files and reduce them afterwards) . If you use a different image size, do remember to scale f according to the image size.)
To do this, you will have to implement a feature-based translational motion estimation. The skeleton for this code is provided in FeatureAlign.cpp. The main routines that you will be implementing are:
int alignPair(const FeatureSet &f1, const FeatureSet &f2, const vector<FeatureMatch>
&matches, MotionModel m, float
f, int nRANSAC, double RANSACthresh, CTransform3x3& M);
int countInliers(const FeatureSet &f1, const FeatureSet &f2, const vector<FeatureMatch>
&matches, MotionModel m, float
f, CTransform3x3 M, double RANSACthresh, vector<int> &inliers);
int leastSquaresFit(const FeatureSet &f1, const FeatureSet &f2, const vector<FeatureMatch>
&matches, MotionModel m, float
f, const vector<int>
&inliers, CTransform3x3& M);
AlignPair takes two feature sets, f1 and f2, the list of feature matches obtained from the feature detecting and matching component (described in the first part of the project), a motion model (described below), and estimates and inter-image transform matrix M. For this project, the enum MotionModel only takes on the value eTranslate.
AlignPair uses RANSAC (RAndom SAmpling Consensus) to pull out a minimal set of feature matches (one match for this project), estimates the corresponding motion (alignment) and then invokes countInliers to count how many of the feature matches agree with the current motion estimate. After repeated trials, the motion estimate with the largest number of inliers is used to compute a least squares estimate for the motion, which is then returned in the motion estimate M.
CountInliers
computes the number of matches that have a distance below RANSACthresh
is computed. It also returns a list of inlier
match ids.
LeastSquaresFit computes a least squares estimate for the translation using all of the matches previously estimated as inliers. It returns the resulting translation estimate in the last column of M.
[TODO] You will have to fill in the missing code in alignPair to:
[TODO] Then, resample each image to its final location and blend it with its neighbors (AccumulateBlend, NormalizeBlend). Try a simple feathering function as your weighting function (see mosaics lecture slide on "feathering") (this is a simple 1-D version of the distance map described in [Szeliski & Shum]). For extra credit, you can try other blending functions or figure out some way to compensate for exposure differences. In NormalizeBlend, remember to set the alpha channel of the resultant panorama to opaque!
[TODO] Crop the resulting image to make the left and right edges seam perfectly (BlendImages). The horizontal extent can be computed in the previous blending routine since the first image occurs at both the left and right end of the stitched sequence (draw the “cut” line halfway through this image). Use a linear warp to the mosaic to remove any vertical “drift” between the first and last image. This warp, of the form y' = y + ax, should transform the y coordinates of the mosaic such that the first image has the same y-coordinate on both the left and right end. Calculate the value of 'a' needed to perform this transformation.
Note that you can also use SIFT features to do the
alignment, which can be useful for testing this component before the feature
detection and matching component is finished. To do so, add the work sift to
the end of the command, as in:
Panorama alignPair warp1.key
warp2.key match1to2.txt 200 1 sift
Sample SIFT features and matches have been provided to you.
You may also refer to the file
stitch2.txt provided along with the skeleton code for the appropriate command line syntax. This command-line interface allows you to debug each stage of the program independently.
You can use the test results included in the images/ folder to check whether your program is running correctly. Comparing your output to that of the sample solution is also a good way of debugging your program.
Here is a list of suggestions for extending the program for extra credit. You are encouraged to come up with your own extensions. We're always interested in seeing new, unanticipated ways to use this program!
In addition to your source code and executables, turn in a web page describing your approach and results. In particular:
Panorama Mosaic Stitching
This portion of the web page should contain the following:
The web-page should be placed in the project2/artifact directory along with all the images in JPEG format. If you are unfamiliar with HTML you can use any web-page editor such as FrontPage, Word, or Visual Studio 7.0 to make your web-page. The KompoZer HTML editor is easy to use and highly recommended. Here are some webpage design tips.
Last modified on Wednesday, April 16, 2008