Panorama
Mosaic Stitching Synopsis
(click here to return to the main
page)
In this component, you will use the feature
detection and matching component to combine a series
of photographs into a 360°
panorama. Your software will automatically align the photographs
(determine
their overlap and relative positions) and then blend the resulting
photos into
a single seamless panorama. You will then be able to view the resulting
panorama inside an interactive Web viewer. To start this component, you
will be
supplied with some test
images and skeleton code that will guide you (the main solution
file in Visual Studio is Features.sln). In addition, a solution
executable can be
found here.
You can download the test images separately here.
Getting
Things to Run
Taking
the
Pictures
ToDo
Creating
the Panorama
Debugging
Extra
Credit
Panorama
Links
Each group will be checking out a panorama kit
(camera, tripod, and Kaidan head). Remember to bring extra batteries with you, these
cameras
drain batteries.
Skip this step for the test data.
Its
camera parameters can be found in the sample commands in stitch4.txt,
which is provided along with the skeleton code.
- Take
a series of photos with a digital camera mounted on a
tripod. Here is a
web page explaining how to use the equipment. Please read it
before you go out to shoot. Then you should borrow the Kaidan head
that lets you make precise rotations and the Canon
PowerShot A10
camera for this purpose. For best results, overlap each image by 50%
with the previous one, and keep the camera level using the levelers on
the Kaidan head.
- Also
take a series of images with a handheld camera. You can use
your own or use the Canon PowerShot
A10 camera that you signed up for. If you are using the Canon camera,
it has a “stitch assist” mode you can use to overlap your images
correctly, which only works in regular landscape mode. If you
are using your own camera, you have to estimate the focal length. The
simplest way to do this is through the EXIF tags of the images, as
described by Noah Snavely (one of your
TAs) here. Alternatively, you can use a camera
calibration toolkit to get more precise focal length and radial
distortion coefficients. Finally, Brett
Allen describes one creative way to measure rough focal length using
just a book
and a box. The
parameters for the class cameras are given below. The
following focal length is valid only if the camera is zoomed out all
the way.
Camera
|
resolution
|
focal length
|
k1
|
k2
|
Canon Powershot
A10, tag CS30012716
|
480x640
|
678.21239 pixels
|
-0.21001
|
0.26169
|
Canon Powershot
A10, tag CS30012717
|
480x640
|
677.50487 pixels
|
-0.20406
|
0.23276
|
Canon Powershot
A10, tag CS30012718
|
480x640
|
676.48417 pixels
|
-0.20845
|
0.25624
|
Canon Powershot
A10, tag CS30012927
|
480x640
|
671.16649 pixels
|
-0.19270
|
0.14168
|
Canon Powershot
A10, tag CS30012928
|
480x640
|
674.82258 pixels
|
-0.21528
|
0.30098
|
Canon Powershot
A10, tag CS30012929
|
480x640
|
674.79106 pixels
|
-0.21483
|
0.32286
|
test images
|
384x512
|
595 pixels
|
-0.15
|
0.0
|
- Make sure the
images are right side up (rotate the images by 90° if you took them
in landscape mode), and reduce them to a more workable size (480x640
recommended). You can use external software such as PhotoShop or the
Microsoft Photo Editor to do this. Or you may want to set the camera to
640x480 resolution from the start, by following the steps below:
- Turn the mode
dial on the back of the camera to one of the 3 shooting modes--auto
(camera icon), manual (camera icon + M) or stitch assist (overlaid
rectangles).
- Press MENU
button.
- Press the
left/right arrow to choose Resolution, then
press SET.
- Press the
left/right arrow and choose S (640x480).
- Press MENU
again.
(Note: If you are using the
skeleton software, save
your images in (TrueVision)
Targa
format (.tga),
since this is the only format the
skeleton software can currently read. Also make sure the
aspect ratio of the
image (width vs. height) is either 4:3 or 3:4 (480x640 will
do) which is
the only aspect ratio supported by the skeleton software.)
ToDo
Note: The skeleton code includes
an image
library, ImageLib, that is fairly general and
complex. It is NOT
necessary for you to peek extensively into this library! We
have created
some notes for you here.
- Warp each image
into spherical coordinates. (file: WarpSpherical.cpp,
routine: warpSphericalField)
[TODO]
Compute the inverse map to warp the
image by filling in the skeleton code in the warpSphericalField
routine to:
- convert the
given spherical image coordinate into the corresponding planar image
coordinate using the coordinate transformation equation from the
lecture notes
- apply radial
distortion using the equation from the lecture notes
(Note: You will
have to use the focal length f estimates for the
half-resolution images provided above (you can either take pictures and
save
them in small files or save them in large files and reduce them
afterwards) .
If you use a different image size, do remember to scale f
according to
the image size.)
- Compute the
alignment of the images in pairs. (file: FeatureAlign.cpp,
routines: alignPair,
countInliers, and leastSquaresFit)
To do this, you will have
to implement a feature-based
translational motion estimation. The
skeleton for this code is provided in FeatureAlign.cpp. The main routines that you
will be
implementing are:
int
alignPair(const FeatureSet
&f1, const
FeatureSet
&f2, const
vector<FeatureMatch>
&matches, MotionModel
m, float
f, int nRANSAC, double
RANSACthresh,
CTransform3x3& M);
int
countInliers(const
FeatureSet
&f1, const
FeatureSet
&f2, const
vector<FeatureMatch>
&matches, MotionModel
m, float
f, CTransform3x3 M, double RANSACthresh, vector<int>
&inliers);
int
leastSquaresFit(const
FeatureSet &f1, const
FeatureSet &f2, const
vector<FeatureMatch>
&matches, MotionModel
m, float f, const vector<int>
&inliers, CTransform3x3& M);
AlignPair
takes two feature sets, f1 and f2,
the list of feature matches obtained from the feature detecting and
matching component (described in the first part of the project), a motion
model (described below), and estimates and inter-image
transform matrix
M. For this project, the enum
MotionModel
only
takes on the value eTranslate.
AlignPair
uses RANSAC (RAndom
SAmpling Consensus) to
pull out a minimal set of
feature matches (one match for this project), estimates the
corresponding
motion (alignment) and then invokes countInliers to
count how many of
the feature matches agree with the current motion estimate. After repeated trials, the
motion estimate
with the largest number of inliers is used to compute a least squares
estimate
for the motion, which is then returned in the motion estimate M.
CountInliers computes the number of matches that have a
distance
below RANSACthresh is computed.
It also returns a
list of inlier match ids.
LeastSquaresFit computes a least squares estimate for the
translation using all of the matches previously estimated as inliers. It returns the resulting
translation estimate
in the last column of M.
[TODO]
You will have to fill in the missing
code in alignPair to:
- Randomly select
a valid matching pair and compute the translation between the two
feature locations.
- Call countInliers to count how many matches
agree with this estimate.
- Repeat the above
random selection nRANSAC times and keep the estimate with the largest
number of inliers.
- Write the body
of countInliers to count the number
of feature matches where the SSD distance after applying the
estimated transform (i.e. the distance from the match to its correct position in the image) is below the threshold. (and don’t
forget to create the list of inlier ids.)
- Write the body
of leastSquaresFit,
which for the simple translational case is just the average
displacement between the matching feature positions.
- Stitch and crop
the resulting aligned images. (file: BlendImages.cpp,
routines: BlendImages,
AccumulateBlend, NormalizeBlend)
[TODO] Given the warped images and their relative
displacements, figure out how large the final stitched image will be
and their absolute displacements in the panorama (BlendImages).
[TODO]
Then, resample each image to its
final location and blend it with its neighbors (AccumulateBlend,
NormalizeBlend). Try
a simple feathering
function as your weighting function (see mosaics lecture slide on
"feathering") (this is a simple 1-D version of the distance map
described
in [Szeliski
& Shum]). For extra credit, you can try
other blending functions
or figure out some way to compensate for exposure differences. In NormalizeBlend,
remember to set the alpha channel of
the resultant panorama to opaque!
[TODO]
Crop the resulting image to make the
left and right edges seam perfectly (BlendImages).
The horizontal extent can be computed in the previous blending routine
since the
first image occurs at both the left and right end of the stitched
sequence
(draw the “cut” line halfway through this image). Use a
linear warp to
the mosaic to remove any vertical “drift” between the first and last
image. This warp, of the form y' = y + ax, should transform
the y
coordinates of the mosaic such that the first image has the same
y-coordinate
on both the left and right end. Calculate the value of 'a'
needed to
perform this transformation.
- Use
the above program you wrote to warp/align/stitch images into the
resulting panorama.
- To remove the radial distortion and warp the image input1.tga
into spherical coordinate with focal length = 600,
radial distortion coefficients k1=-0.21 and k2=0.25 (Panorama is the
name of the program):
Panorama
sphrWarp
input1.tga warp1.tga
600 -0.21 0.25
- Then, use the feature detecting and matching
component to compute the features in the warped images. To
align two feature sets warp1.f and warp2.f, with a set of matches in match1to2.txt,
using 200 iterations of RANSAC with an outlier threshold distance of 1
pixel:
Panorama
alignPair warp1.f
warp2.f match1to2.txt 200 1
Note that you can also use SIFT features to do the alignment, which
can be useful for testing this component before the feature detection
and matching component is finished. To do so, add the work sift to the end of the command, as in:
Panorama
alignPair warp1.key
warp2.key match1to2.txt 200 1 sift
Sample SIFT features and matches have been provided to you.
- Run the previous step for all adjacent pairs of
images and save the output into a separate file pairlist.txt which
may look like this:
warp1.tga warp2.tga 213.49 -5.12
warp2.tga warp3.tga 208.19 2.82
......
warp9.tga warp1.tga 194.76 -3.88
- Then
stitch the images into the final panorama pano.tga:
Panorama
blendPairs
pairlist.txt pano.tga blendWidth
You
may also
refer to the file
stitch2.txt provided
along with the skeleton code
for the appropriate command line syntax. This command-line interface
allows you
to debug each stage of the program independently.
- Convert your
resulting image to a JPEG (Photoshop and other tools in the Sieg lab
can help you with this) and paste it on a Web page along with code to
run the interactive viewer. Click here
for instructions on how to do this.
You can use the test results
included
in the images/ folder to check whether your program is running
correctly.
Comparing your output to that of the sample solution is also a good way
of
debugging your program.
- Testing the warping routines:
- In
the images/ folder in the skeleton code, a few example warped images
are provided for test purposes. The camera parameters used for these
examples can be found in the sample command file stitch2.txt. See if
your program produces the same output.
- You may also test with different input images
and/or camera parameter values by comparing the results with those of
the sample solution.
- Testing the alignment routines:
- A
few example alignment results are provided in the file pairlist2/4.txt.
The corresponding shell commands can be found in stitch2/4.txt.
- To test alignPair only, try
passing in an image that has been cropped with two different rectangles
(and maybe rotated by a tiny amount, say 2 degrees).
- We have also provided sample SIFT key files and matches for the test images, for testing the alignment routines.
- Testing the blending routines:
- An
example panorama is included in the images/ folder. Compare the
resulting panorama with this image.
- You
may also test with other panoramas by running the sample solution on
different inputs.
Here is a list of suggestions for
extending
the program for extra credit. You are encouraged to come up with your
own
extensions. We're always interested in seeing new, unanticipated ways
to use
this program!
- Although the feature-based
aligner gives sub-pixel
motion estimation (because of least squares), the motion vectors are
rounded to
integers when blending the images into the mosaic in BlendImages.cpp.
Try to blend images with sub-pixel localization.
- Sometimes, there exists
exposure difference between
images, which results in brightness fluctuation in the final mosaic.
Try to get
rid of this artifact.
- Try shooting a sequence with
some objects moving.
What did you do to remove “ghosted” versions of the objects?
- Try a sequence in which the
same person appears
multiple times, as in this
example.
- Implement a better blending technique, e.g., pyramid
blending,
poisson
imaging blending,or
graph cuts.