## Project 2:  Panoramic Mosaic Stitching

Rahul Garg

360o panorama of greenlake warped by a polar transformation

Red Square (Viewer)

Green Lake (Viewer)

Unfortunately, it looks like there were too many people enjoying the good weather on Saturday. Simple blending using feathering leads to ghosting artifacts due to moving people. To fix this, I use a simple method that is inspired by Graphcut Textures.  However, instead of considering all the images simultaneously and solving a multi label graph cut problem or following the approach given in the paper where they update old seams on finding a new seam, I simply find the optimal seam for each image independently when it is being added to the panorama.

More precisely, the problem reduces to finding a binary labelling over the pixels in the area of overlap between the new image and the accumulated image with the label denoting whether  the pixel  should be copied from the accumulated image or from the new image. This can be formulated as min-cut max-flow problem over the grid graph of  the overlapped pixels. The weight of the edge connecting the pixels i and j is defined as:

where img(i) and acc(i) denote the pixel values at location i in the new image and the accumulated image respectively. The intuition behind choosing this weight function is that the numerator encourages the seam to pass between pixels that have similar values in both the new image and the accumulated image while the denominator encourages seams through high gradient regions since we are less likely to see artifacts in those area. I used Vladimir Kolmogorov's maxflow code available here.  Result (with seams) using this method (Click on the image to clearly see the seams in full resolution image):

Note that it successfully removes the ghosting artifacts. Its interesting to see the seam passing between the two images of the same person and how it preserves both the person and the shadow. However, without feathering  exposure variations start to appear. These should also have an adverse effect on the seam finding. I use a very simple method to correct for exposures -- adding  the median of the differences between the accumulated image and the new image over the overlapping region. Using the median should also help to take care of the "outliers" like moving objects. Here' a result using exposure correction (Click to see full resolution seam image and the panorama):

Some of the seams are still visible. One can perhaps try applying a 1D Gaussian smoothing across the seams. Notice how the water was calm when I started taking the panorama (towards the right) and how the waves appeared towards the left -- the difference clearly visible in the extreme left where the first image in the sequence meets the last image.

The  cool image at the top of the page is produced by warping the panorama using GIMP (plus some inpainting and touchup to remove inpainting artifacts). Here's a similar image for red square:

Hand Held Sequence
Green Lake (Viewer)

It turns out that I was extra careful while taking images to ensure overlap between adjacent images and I ended up taking 32 images. There is lot of ghosting due to misregisteration and moving objects.

The max-flow method can also help in hiding the misregisteration artifacts. Here's the output (Click to see full resolution seam image and the panorama, sometimes the seams dont look good because my code for drawing seams was slightly buggy and I was too lazy to fix it):

Test Sequence
(Viewer)

What worked well and what didn't

• Feature matching worked pretty well for the green lake dataset and the test sequence but somehow there is some misregisteration in the Red Square dataset. However, I was still getting reasonable number of inliers during RANSAC matching, so it could have been due to bad dataset (non zero tilt angle, etc.)
• RANSAC was pretty robust and  I could lower the thresholds of feature matching without affecting the results very much.
• Simple feathering produces surprisingly pleasant looking results!
• My primitive method of exposure correction worked pretty well for the Green Lake dataset but it had a tendency to 'drift' (You can see that the green lake panorama produced using max-flow is brighter towards the right and grows darker towards the left). The 'drift' was much more pronounced in the test sequence where there are much larger exposure variations. I also tried using a 'multiplicative' model instead of an 'additive' one, i.e assuming that variation is due to scaling in intensity and rectifying in a similar fashion (considering the median multiplicative factor) but it seemed to give worse results even though it models the changes due to exposure differences more accurately.
• The max-flow algorithm to find the optimal seam before placing each image worked fairly well. It would be interesting to compare how much advantage do we get by doing simultaneous optimization of all seams.

Extra Credit
• Primitive exposure correction.
• Removal of ghosting artifacts using max-flow
The course webpage also talks about using subpixel motion vectors while blending, but I think that it's already being done in the AccumulateBlend() function via the call to PixelLerp().