360o panorama of greenlake warped by a polar transformation
Unfortunately, it looks like there were too many people enjoying the good weather on Saturday. Simple blending using feathering leads to ghosting artifacts due to moving people. To fix this, I use a simple method that is inspired by Graphcut Textures. However, instead of considering all the images simultaneously and solving a multi label graph cut problem or following the approach given in the paper where they update old seams on finding a new seam, I simply find the optimal seam for each image independently when it is being added to the panorama.
More precisely, the problem reduces to finding a binary labelling over the pixels in the area of overlap between the new image and the accumulated image with the label denoting whether the pixel should be copied from the accumulated image or from the new image. This can be formulated as min-cut max-flow problem over the grid graph of the overlapped pixels. The weight of the edge connecting the pixels i and j is defined as:
where img(i) and acc(i) denote the
pixel values at location i in the new image and the accumulated image
respectively. The intuition behind choosing this weight function is
that the numerator encourages the seam to pass between pixels that
have similar values in both the new image and the accumulated image while
the denominator encourages seams through high gradient regions since we
are less likely to see artifacts in those area. I used Vladimir
Kolmogorov's maxflow code available here. Result (with seams) using this method (Click on the image to clearly see the seams in full resolution image):
Note that it successfully removes the ghosting artifacts. Its interesting to see the seam passing between the two images of the same person and how it preserves both the person and the shadow. However, without feathering exposure variations start to appear. These should also have an adverse effect on the seam finding. I use a very simple method to correct for exposures -- adding the median of the differences between the accumulated image and the new image over the overlapping region. Using the median should also help to take care of the "outliers" like moving objects. Here' a result using exposure correction (Click to see full resolution seam image and the panorama):
Note that it successfully removes the ghosting artifacts. Its interesting to see the seam passing between the two images of the same person and how it preserves both the person and the shadow. However, without feathering exposure variations start to appear. These should also have an adverse effect on the seam finding. I use a very simple method to correct for exposures -- adding the median of the differences between the accumulated image and the new image over the overlapping region. Using the median should also help to take care of the "outliers" like moving objects. Here' a result using exposure correction (Click to see full resolution seam image and the panorama):
Some of the seams are still visible. One can perhaps try applying a 1D Gaussian smoothing across the seams. Notice how the water was calm when I started taking the panorama (towards the right) and how the waves appeared towards the left -- the difference clearly visible in the extreme left where the first image in the sequence meets the last image.
The cool image at the top of the page is produced by warping the
panorama using GIMP (plus some inpainting and touchup to remove
inpainting artifacts). Here's a similar image for red square:
Hand Held Sequence
Green Lake (Viewer)
It
turns out that I was extra careful while taking images to ensure
overlap between adjacent images and I ended up taking 32 images. There
is lot of ghosting due to misregisteration and moving objects.
The max-flow method can also help in hiding the misregisteration artifacts. Here's the output (Click to see full resolution seam image and the panorama, sometimes the seams dont look good because my code for drawing seams was slightly buggy and I was too lazy to fix it):
The max-flow method can also help in hiding the misregisteration artifacts. Here's the output (Click to see full resolution seam image and the panorama, sometimes the seams dont look good because my code for drawing seams was slightly buggy and I was too lazy to fix it):
Test Sequence
(Viewer)
What worked well and what didn't
- Feature matching worked pretty well for the green lake dataset and the test sequence but somehow there is some misregisteration in the Red Square dataset. However, I was still getting reasonable number of inliers during RANSAC matching, so it could have been due to bad dataset (non zero tilt angle, etc.)
- RANSAC was pretty robust and I could lower the thresholds of feature matching without affecting the results very much.
- Simple feathering produces surprisingly pleasant looking results!
- My primitive method of exposure correction worked pretty well for the Green Lake dataset but it had a tendency to 'drift' (You can see that the green lake panorama produced using max-flow is brighter towards the right and grows darker towards the left). The 'drift' was much more pronounced in the test sequence where there are much larger exposure variations. I also tried using a 'multiplicative' model instead of an 'additive' one, i.e assuming that variation is due to scaling in intensity and rectifying in a similar fashion (considering the median multiplicative factor) but it seemed to give worse results even though it models the changes due to exposure differences more accurately.
- The max-flow algorithm to find the optimal seam before placing each image worked fairly well. It would be interesting to compare how much advantage do we get by doing simultaneous optimization of all seams.
Extra Credit
- Primitive exposure correction.
- Removal of ghosting artifacts using max-flow