3D Reconstruction from Photometric Stereo

The 3D reconstructions turned out pretty well in general, although several of them yielded undesirable artifacts. This was due to the fact that our model assumed a diffuse surface and uniform albedo (among other assumptions), and some of the images violated these assumptions more so than others.

Things that worked well:

Calculating the light direction, surface normals, and depths was straightforward. Finding the brightest pixel in the chrome images and computing surface normals both required using pixel intensity values, and while the provided skeleton code used just the green value, I tried using other values. These included using just red and blue values, the mean of the 3 values, and 2 different linear combinations (the Rec. 709 and Rec. 601 luma coefficients). These different methods yielded slightly different brightest pixel positions and surface normals, but were all very close the each other. The differences among these methods are best seen in the rgb map of the surface normals and are otherwise hard to discern from the needle, albedo, and depth map.

Computing depth from diffuse surfaces worked really well. The reconstructions using the Buddha, horse, and rock images have surfaces that show very few artifacts (excluding those around the images' silhouettes and a little point below Buddha's left foot). These objects' surfaces weren't as reflective as the other objects and demonstrate how well this program calculates depth from diffuse surfaces.

Things that didn't work too well:

3D reconstruction using images with more reflective surfaces fared poorly. Images that violated the diffuse surface assumption resulted in unexpected artifacts. For example, the gray sphere, while not as specular as the chrome sphere, isn't completely diffuse. If you look at the albedo map, you can see that near the center is a bright patch that reveals the sphere's reflectiveness. Correspondingly, the 3D construction isn't a smooth sphere and there's a slight bump at the location of the bright spot in the albedo map. This effect is shown more prominently in the owl's stomach and cat's nose/left paws. You can see in the 3D reconstruction that they are slightly more protruded than expected. The program tried to handle shadows by weighing pixels by their intensity, but a side effect is that specular highlights are over-weighted, resulting in skewed depths.

The most conspicuous flaw in this program is revealed by the owl's right eye. Looking at the 3D reconstruction from the same perspective as the source images doesn't reveal any issues. However, once you change the perspective, you can see that the owl's right eye is spiky.
Solving the surface normal for pixels that were non-black in 0-2 images of the image set resulted in a normal with NaN values, in which case the program guesses a normal of <0, 0, 1>. These pixels are then ignored entirely when computing the albedo and depth. In all the images in the owl image set, the owl's right eye is black, and so there isn't enough information to properly compute the right surface normals.

Another interesting limitation of this program is that all the source images are taken from the same perspective, and so some features of an image will occlude other parts of image. The owl's beak hides the left side of face, and so in real life you would expect the face to be symmetrical, but the program computes the depth so that the owl's left side of the face is a lot more shallow and distorted than what's expected. This problem arises because photometric stereo assumes a fixed perspective under various lighting conditions.

Other issues and possible improvements:

• Specular highlights have exaggerated depths:
  One way to fix this problem is to identify specular regions and interpolate the depths of pixels in those regions based on the depths of pixels outside the region.
• A normal of <0, 0, 1> is guessed for pixels whose surface normal contains NaN:
  Instead of guessing <0, 0, 1> we can assume that the surface is smooth and interpolate a normal from neighboring surface normals. Or we could hand-feed pre-computed surface normals for these pixels.
• Shadows/too many black pixels may not provide enough information for accurate 3D reconstruction:
  We could always take more pictures with different lighting angles so as to eliminate shadows and get rid of image sets where pixels are non-black in 0-2 of the images.
• The reconstructions along the silhouettes have this wall-like artifact:
  Some fixes would be to prevent the pixels along the mask silhouette from being rendered, or to shrink the image masks so that the edges are not included in any computation.

Buddha Images

Horse Images

Rock Images

Gray Images

Owl Images

Cat Images