panorama of greenlake warped by a polar
Unfortunately, it looks like there were too many people enjoying the
good weather on Saturday. Simple blending using feathering leads to
ghosting artifacts due to moving people. To fix this, I use a simple
method that is inspired by Graphcut Textures
However, instead of considering all the images simultaneously and
solving a multi label graph cut problem or following the approach given
in the paper where they update old seams on finding a new seam, I
simply find the optimal seam for each image independently when it is
being added to the panorama.
More precisely, the problem reduces to finding a binary labelling over
the pixels in the area of overlap between the new image and the
accumulated image with the label denoting whether the pixel
should be copied from the accumulated image or from the new
image. This can be formulated as min-cut max-flow problem over the grid
graph of the overlapped pixels. The weight of the edge connecting
the pixels i
is defined as:
where img(i) and acc(i) denote the
pixel values at location i in the new image and the accumulated image
respectively. The intuition behind choosing this weight function is
that the numerator encourages the seam to pass between pixels that
have similar values in both the new image and the accumulated image while
the denominator encourages seams through high gradient regions since we
are less likely to see artifacts in those area. I used Vladimir
Kolmogorov's maxflow code available here
. Result (with seams) using this method (Click on the image to clearly see the seams in full resolution image):
Note that it successfully removes the ghosting artifacts. Its
interesting to see the seam passing between the two images of the same
person and how it preserves both the person and the shadow. However,
without feathering exposure variations start to appear. These
should also have an adverse effect on the seam finding. I use a very
simple method to correct for exposures -- adding the median of
the differences between the accumulated image and the new image over
the overlapping region. Using the median should also help to take care
of the "outliers" like moving objects. Here' a result using
exposure correction (Click to see full resolution seam image and the
Some of the seams are still visible.
One can perhaps try applying a 1D Gaussian smoothing across the seams.
Notice how the water was calm when I started taking the panorama
(towards the right) and how the waves appeared towards the left -- the
difference clearly visible in the extreme left where the first image in
the sequence meets the last image.
The cool image at the top of the page is produced by warping the
panorama using GIMP (plus some inpainting and touchup to remove
inpainting artifacts). Here's a similar image for red square:
turns out that I was extra careful while taking images to ensure
overlap between adjacent images and I ended up taking 32 images. There
is lot of ghosting due to misregisteration and moving objects.
The max-flow method can also help in hiding the misregisteration
artifacts. Here's the output (Click to see full resolution seam image
and the panorama, sometimes the seams dont look good because my code for drawing seams was slightly buggy and I was too lazy to fix it):
What worked well and what didn't
- Feature matching worked pretty well for the green lake dataset
and the test sequence but somehow there is some
misregisteration in the Red Square dataset. However, I was still
getting reasonable number of inliers during RANSAC matching, so it
could have been due to bad dataset (non zero tilt angle, etc.)
- RANSAC was pretty robust and I could lower the thresholds of feature matching without affecting the results very much.
- Simple feathering produces surprisingly pleasant looking results!
- My primitive method of exposure correction worked pretty well for
the Green Lake dataset but it had a tendency to 'drift' (You can see
that the green lake panorama produced using max-flow is brighter
towards the right and grows darker towards the left). The 'drift' was
much more pronounced in the test sequence where there are much larger
exposure variations. I also tried using a 'multiplicative' model instead of an 'additive' one, i.e assuming that variation is due to scaling in intensity and rectifying in a similar fashion (considering the median multiplicative factor) but it seemed to give worse results even though it models the changes due to exposure differences more accurately.
- The max-flow algorithm to find the optimal seam before placing
each image worked fairly well. It would be interesting to compare how
much advantage do we get by doing simultaneous optimization of all
- Primitive exposure correction.
- Removal of ghosting artifacts using max-flow
The course webpage also talks about using subpixel motion vectors while blending, but I think that it's already being done in the AccumulateBlend() function via the call to PixelLerp().