Panoramic Mosaic Stitching
Report by Ankit Gupta
Panorama with Moving Objects
When
the sequence of images contains moving objects, naive blending approach
introduces ghosts for moving objects as seen in the panorama below.
The "Haunted" Quad (using kaidan head)
(The rough transition towards right
is seen because I misplaced one image from the sequence so there is no
overlap there and hence no feathering)
To remove the ghosting artifacts, I do the following. When
we accumlate colors for a particular pixel from different
images, I only aggregate colors from an image if they are
similar to the colors already accumlated. I define this notion of
similarity by using a 5X5 RGB descriptor window centred at that pixel.
The ratio for the SSD is taken to be 7500. I also tried using RGB
difference over a single pixel instead of a patch but that did not give
good results due
to the potential local noise in the images. The following figure shows
the result.
The Quad without Ghosts (using kaidan head) [
Live Picture Viewer]
(This strategy is not perfect and some artifacts can be seen in regions where background and foregorund are of similar color)
This argument is based on the assumption that the foreground and
background are of sufficiently different colors. The following image
demonstrates the working of this strategy and is seen to fail (like
around the lady's arm) where the foreground and background are of
similar color. Choosing a particular threshold for a similarity measure
is a tough
problem and again boild down to the constant debate in computer vision
over invariance and discriminative power of a similarity measure. The
threshold of 7500 here was set by empirical testing. A limitation of
this method is that the result will vary with the order you accumlate
your images in. But in simple cases this strategy should be sufficient
for removing the ghosts.
Panoramas with Illumination Variance
Illumination variation in the adjacent images cause a brightness
transition in the panorama. Though this transition is smoothened by feathering
in the blend function but the panorama still does not look good. An
example is shown below.
Test sequence without illumination stabilization
I convert the images into YIQ space and match the image statistics
over Y channel i.e. I make the means and standard deviations of
all
images equal. Then I combine this Y channel with the original I and Q
channels. This strategy is based on the assumption that the
illlumination changes while capturing photographs do not affect the
chrominance values.
The result is better but still not perfect. This is because the above
strategy is a global illumination remapping. Ideally we will want
to segment out the image, match the segments across images and then
match the illumination for those segments. The remapped segments in one
image can then be smoothly blended. Rather this smooth blending and
remapping of various segments can be posed as a graph cut minimization
problem and solved. The hard part is to segment out the image properly
and match the segments.
All the panoramas shown on this page have been generated by stabilized illumination.
What worked well and what did not
The basic process of panoramic stitching worked well. Illumination
variation across the sequence does hurt and the global image statistics
matching approach improves the visual quality but does not give perfect
results. Ideally we will like to match illumination between matched
image segments as described earlier. For removing ghosts, simple color
guided accumlation does help improve results but it is a local
technique and better results can be obtained by using a graph cut based
labeling formulation over the set of overlapping images. The hand
shot sequence also came out quite well considering I took the same
number of images as the Kaidan head will take.
Extra Credit
1. Removing ghosts of
moving objects
- explained above
2. Illumination invariance - explained above
3. Using subpixel blending - After computing the floating point pixel values through least
squares motion vectors, bilinear interpolation is used over the source
image to find the subpixel color value rather than rounding of floats
and sampling the integer pixel value.
Ankit Gupta
Deptt. of Computer Science and Engg.
University of Washington
