Finding a good image match indicator
After I got the main program working, I wanted to try and create an autostiching program.
First, I wanted to see if there was some simple calculation that I could do during the LucasKanade
step that would be a good indication of whether two images were a good match or not. The first thing
I tried was computing an average of It (the difference between the intensities of the two images) over
all the pixels. I called this value ItAvg. My reasoning was that images that match should have pixels with
about the same intensity at the same position. The following are the computed ItAvg values for all possible
pairs of the four test images:
As you can see, it turns out ItAvg is not a good indication of whether two images are a good match or not.
The three image pairs that should match (08-09, 09-10, 10-11) do not have ItAvg values that really stand out
from the others. I believe this is because even if two images are a good match, exposure differences can
cause one image to be generally lighter or darker than the other image. So ItAvg is more a measure
of the exposure differences between two images. But even if two images have different exposure levels, if
the images match, then the intensity difference should pretty much be constant from pixel to pixel.
This led me to try computing the square of the difference between It and the current ItAvg for each pixel, and
taking the average of this value over all pixels. I called this value ItDev. The following are the computed ItDev
values for all possible pairs of the test images:
As you can see, ItDev is a much better indication of whether two images match or not. The three image pairs
that should match all have much lower ItDev values than the other pairs. This is because in image pairs that
match, cooresponding pixels have intensity values that differ by an almost constant amount, while in image
pairs that do not match, the pixel intensities will differ by an almost random amount.
Implementaion
Now that I have a good match indicator, creating an autostitch program is simply a matter of implementation
details. I decided to write it in Java, since I have more experience with it. The program will look in a given
folder for all the tga images, apply cylWarp with given focal length and k1 and k2 radial distortion constants,
order the images into one or more panoramas and generate the cooresponding pairlist files, and then stitch the
panoramas together. As you can see, most of the work is delegated to the v4gp1 program. To keep it simple,
all the images are assumed to have the same focal length and k1 and k2 constants and also about the same
horizontal displacement.
To order the images into panoramas, the program starts with a sequence of images that form part of the
panorama and a pool of unordered images, and tries to "grow" the panorama by placing one of the unordered images
on either the left or the right side on the partial panorama. It chooses the image to add by calculating the
ItDev values for all the possible left and right image pairs and then choosing the image with the lowest ItDev.
However, if all the image pairs have ItDev values that are above some given cutoff value, then it stops growing
the current panorama and starts growing a new panorama with one of the remaining images in the unordered image
pool.
Usage
The program runs from the command line and takes 6 arguments:
java AutoStitch folder f k1 k2 h_disp cutoff
folder is the absolute path to a folder that contains the tga images.
Note that due to the way v4gp1 parses the pairlist file, the path to the folder and the image files cannot
have any spaces in their names.
f k1 k2 are the camera focal length and k1 and k2 radial distortion constants.
h_disp is the initial guess of the horizontal displacement. It is also
used as the blend width.
cutoff is a value that determines how many different panoramas are
created. A low cutoff value will cause each image to be its own panorama. A high cutoff value will
place all images in a single panorama, even if they should actually belong to different panoramas.
A good value for this is between 500 and 1000.
Results
To test the program I took 4 images from each of 3 different panoramas along with another image and put them all
into a folder. The program correctly managed to generate the 3 panoramas when I used a cutoff value of 600.
Input images:
Output images:
Although it looks like only 2 images got stitched into each output panorama, in fact all 4 of the images were
correctly stitched together for each panorama. The problem is that v4gp1 crops away the left and right sides of the
generated panorama since it is meant to be used to stitch complete 360 degree panoramas. When stitching 360 degree
panoramas, the first and last images will be the same, so the panorama is cropped so the left and right edges fit
together. When building panoramas that are not the full 360 degrees, it would be better not to do any cropping so
more of the panorama can be seen.
As a second test, I tried running autostitch on all of the images from the 3 panorama sequences all together (62 images total).
Together, they should form 3 complete panoramas. I tried running autostitch with different cutoff values to see how this
would affect the results. Here are the 62 input images:
Here are the results of running autostitch with a cutoff value of 600: (Autostitch also generated several smaller panoramas,
which are not shown)
Here are the results of running autostitch with a cutoff value of 1000:
Here are the results of running autostitch with a cutoff value of 1600:
When the cutoff value was 600, autostitch generated several panoramas that only contained 1 or 2 images (not shown),
causing the longer panoramas to contain less images. Most noticably, the dorm room sequence got split up into 2 smaller panoramas.
When the cutoff value was 1000, the 3 panoramas were correctly generated, with no stitching together of images from
different panoramas. However, some images were still not included in the panoramas, as you can see since the result
panoramas are not the full 360 degree panoramas.
Increasing the cutoff value to 1600 did not help to include more of the images in each panorama. The same dorm room and
red square sequences were generated when the cutoff was 1600 as when the cutoff was 1000. In fact, setting the cutoff
to 1600 actually caused one of the dorm room images to be incorrectly blended into the lounge sequence, as you can see if you look
carefully at the right side of the lounge sequence.
As you can see from the results, when the cutoff value is too low, autostitch is more likely to give up on growing the current
panorama. So there will be more panoramas generated than there should be. On the other hand, when the cutoff value is too high,
autostitch is less likely to give up on growing the current panorama. So images from different panoramas may actually be stitched
together. The trick is to find the cutoff value that returns the best result. Of course, a cutoff value may not always exist that gives the correct result. This just
goes to show that ItDev information alone is not enough to do perfect automatic panorama stitching.
