CSEP 576 Project2: Panoramic Mosaic Stitching

Jacki Roberts

Image1: Carkeek Park (tripod aligned)

A sunny day found someone with an orange backpack trespassing on the property of the Burlington Northern Santa Fe railroad near Carkeek park.

Panorama of Carkeek Park

Viewer:

Image2: Carkeek Park (handheld)

Panorama from Handheld Camera

Viewer:

 

Image3: Sample for debugging

Panorama file

Viewer:

 

What Went Well and not so well

Lucas Kanade (Extra)

The work on the whole was easier than I expected. My largest effort went into making the Lucas Kanade defined alignment on the highly-textured, rocky surface in the foreground as reasonable as possible, since small errors were highly noticable. Both the same executable and my own used a similar derivative function: dx=f(x±1,y)-f(x,y) and dy=f(x,y±1)-f(x,y).

In some cases, such as as the pictures perpendicular to the start point, the Newton-Raphson method swung hugely out of control using this derivative set, settling on an optimal solution quite far from the empirically best one. This occurred with both the sample and my own executables, making me conclude that a single derivative function might not be best for all regions of the panorama. In an area that grew out of control, often a better solution was available by changing the derivative from x+1 to x-1, or y-1 to y+1. This did not always fix the problem, but usually produced better results.

I believed that a more sensitive derivative might help matters, so I used a Sobel derivative in the code found here. This turned out not to make much of a difference. The places where the simpler derivatives were off occasionally saw closer to expected results with the Sobel derivative, but just as frequently were still far off. Consider the following comparison:

Images	Visual Expectation	Lucas Kanade result from Sample exe	Lucas Kanade result from my exe	Lucas Kanade result from my exe, using Sobel
warp8 to warp9	between (239,4) and (237,3)	(218.00, 28.28)	(231.44, 13.26)	(217.14, 29.26)

One thing that helped the process but takes away independence from user input was to come up with a close offset with which to start the Newton-Raphson method. This entailed opening the two images in another application, overlaying them to a visually close match, Switching between which image is on top, so that the two edges are matched independently, and then averaging the x and y offsets. Manually determining this visually cut down on the number of times N-R failed.

Lucas Kanade with the Handheld Images

The handheld images were much more sensitive to vertical rotation, which meant that a single warp for the entire image was not enough to allow them to align nicely even after several iterations of Lucas Kanade. There are places, especially those close to the camera in the handheld panorama, that show significant ghosting. Controlling vertical movement using the tripod was a clear benefit.

Blending Images

The following two pictures represent a blend region of 10 on the sample panorama. The first uses the sample executable, and the second is done with my executable (the angle difference on the vertical lines visible in the output is due to my shift using a rotation rather than a y'=y+ax function). They both use the edge of the image rather than the center of the overlap as the blend start. This makes it much easier to incorporate three or more images at a single point, but does not feather in quite the way discussed in class. Another way would be to use the center of the blend as the center of the offset, but this becomes difficult when three images overlap, especially when they include vertical displacements.

 

Shifting

I found it cleaner to rotate the image rather than shift the y-values only. I rotated about the left bottom edge of the first subimage, by the amount of drift to the same pixel in the last, matching subimage.

Cropping

The shifting in both the sample and my solution allows alls holes to be cropped out only if the drift was monotonically increasing or decreasing. Otherwise, there were still holes along the top and bottom edge of the finished panorama. Another solution would be to start at the bottom, and check each row for "black pixels", and make the minimum y the first row without the faulty pixels, then do the same at the top. The image could be shifted and the final result cropped according to this  adjusted height.