Assigned: Friday, February 26

Due: Friday, March 12

Submission: Individual

In this project, you will be implementing a Photometric Stereo algorithm. Photometric stereo is a method for reconstruction of the depth map (the surface) and albedo of an object from several photographs of the object in a fixed position but under different lighting conditions. Your job will be to write MATLAB functions that will be able to calibrate the lighting directions, find the best fit normal and albedo at each pixel, and finally find a surface which best matches the solved normals. All these parts were explained in class, please refer to your notes and slides for detailed explanation. Here is an example result of the algorithm:

We have provided six different datasets of diffuse (or nearly
diffuse) objects for you to try out:

gray |
buddha |
horse |
cat |
owl |
rock |

They can be downloaded here (in tga format) or here (in png format).

After you unzip it, you will see seven directories. They correspond to the above six objects and a chrome ball. Each of the directories contains 12 images the object photographed under a different lighting direction. The chrome ball images should be used to compute lighting directions for computing normal maps for the other six objects.

Under each of the directory, there is also a mask image. The masks indicate what pixels are on the objects. You only need to estimate normals and compute height values for these.

From the chrome ball images, you should estimate the lighting directions. In particular you should:

- Compute the centroid of the ball mask as the center of the ball
- Compute the average distances between the boundary pixels to the center as the radius
- Compute the centroid of the highlight pixels in each lighting image
- Read the lecture slides on how to compute the actual lighting direction

1. **(15%)**
**Calibration**: Before we can extract normals
from images, we have to calibrate our capture setup. This includes determining
the lighting intensity and direction, as well as the camera response function.
For this project, we have already taken care of two of these tasks: First, we
have linearized the camera response function, so you
can treat pixel values as intensities. Second, we have balanced all the light
sources to each other, so that they all appear to have the same brightness.
You'll be solving for albedos relative to this
brightness, which you can just assume is 1 in some arbitrary units. In other
words, you don't need to think too much about the intensity of the light
sources.

The one remaining calibration step we have left for you is calibrating lighting
directions. One method of determining the direction of point light sources is
to photograph a shiny chrome sphere in the same location as all the other
objects. Since we know the shape of this object, we can determine the normal at
any given point on its surface, and therefore we can also compute the
reflection direction for the brightest spot on the surface.

Write a function that gets as input the file chrome.txt (see Data section),
reads the sphere images and the mask and returns the matrix L of size 3x12 that
specifies the lighting direction per image (there are 12 images per object, see
Data section).

`L = recoverLighting('chrome.txt');`

Write a function that gets as input the txt file that specifies the names of the images (see Data section) and outputs the images and the mask:

`[imgs,mask]=readImages('objectName.txt');`

2.
**(25%) Recovering surface normals**:
The appearance of diffuse objects can be modeled as where
I is the pixel intensity, kd is the albedo, L is the lighting
direction (a unit vector), and n is the unit surface normal. (Since our images
are already balanced as described above, we can assume the incoming radiance
from each light is 1.) Assuming a single color channel, we can rewrite this as so
the unknowns are together. With three or more different image samples under
different lighting, we can solve for the product by
solving a linear least squares problem (for this you need to write the above
equation in a matrix form as we did in class). The objective function is:

To help deal with shadows and noise in dark pixels, its helpful to weight the solution by the pixel intensity: in other words, multiply by Ii:

The objective Q is then minimized with respect to g. Once we have the vector g = kd * n, the length of the vector is kd and the normalized direction gives n.

Note that weighting each term by the image intensity reduces
the influence of shadowed regions; however, it has the drawback of **over**weighting saturated pixels, due
to specular highlights, for example.

Write a function that gets as input: the images, mask and the lighting matrix
from part 1. The function estimates the surface normals
and outputs them.

`N = recoverNormals(imgs,mask,L);`

**(25%) Recovering the albedo**: This gives a way to get the normal and albedo for one color channel. Once we have a normal n for each pixel, we can solve for the albedos by another least squares solution. But this one ends up being a simple projection. The objective function is

To minimize it, differentiate with respect to kd, and set to zero:

This can be done for each channel independently to obtain a per-channel albedo.

Write a function that gets as input the images, mask, recovered surface normals and lighting and outputs the albedos.

albedos = recoverAlbedos(imgs,mask,L,N);

**(25%) Recovering the surface from normals:**As described in class, we have 2 constraints for each pixel inside the object

-nx = nz z(x+1,y) - nz z(x,y)

-ny = nz z(x,y+1) - nz z(x,y)

and 1 constraint for the pixels on the boundary:

nx z(x,y+1) - nx z(x,y) = ny z(x+1,y) - ny z(x,y)

all these can be written as matrix equation Mz=b, and solved using least squares. Note that the size of matrix M can be very large however most of the entries in M are zero -- this is a very**sparse**matrix. In MATLAB you can define the matrix to be sparse.

Write a function that gets as input the surface normals and the mask of the object, and outputs the surface depth map.

surface = normals2surface(N,mask);

For this project we are asking you to submit a document (HTML/WORD/PDF) that shows the results and dicusses the implementation and the algorithm. In particular, your document should include:

- For each object:
- RGB encoded surface normals
- Albedo map
- Two or more views of the reconstructed surface without albedos (you can use MATLAB's 'surfl' function for this)
- Discussion of the results: what worked and what didn't work? which objects were reconstructed better/worse? why? discuss the issues you had with the algorithm, how would you improve it?
- What will happen if you will run the algorithm with less images and with different choice of images? How are the reconstruction results affected by this?
- What will happen if you will use all the pixels instead only the pixels in the mask? (discuss each part of the algorithm).

Extra credit problems do not count towards your score for the project, but are accumulated separately. One star (*) is awarded for easier extra credit problems, with more stars for more difficult problems. At the end of the course, the stars you've earned will be used to adjust your final grade upward by at most 5%.

· (*) Modify the objective function for surface reconstruction and rewrite the matrix accordingly. One idea is to include a "regularization" (smoothing) term. This can sometimes help reduce noise in your solutions. Another idea is to weight each edge constraint by some measure of confidence. For instance, you might want the constraints for dark pixels where the normal estimate is bad to be weighted less heavily than bright pixels where the estimate is good.

· (*) Capture your own source images and reconstruct them

· (**) Try the example based photometric stereo method, using the data on their website.

- (**) You are welcome to do any other extensions. The bonus depends on how useful and difficult these extensions are.

To turn in your project, submit a single zip file to the dropbox at:

https://catalysttools.washington.edu/collectit/dropbox/iansimon/8903

The zip file should be named `project4.zip` and contain all of your
M-files and your assignment write up in HTML, WORD or PDF format. The zip file
should include any images needed for your write up.

- Woodham, Optical Engineering, 1980, Photometric Stereo.
- Hayakawa, Journal of the Optical Society of America, 1994, Photometric stereo under a light source with arbitrary motion.
- Example-Based Photometric Stereo: Shape Reconstruction with General, Varying BRDFs A. Hertzmann, S. M. Seitz. IEEE Trans. PAMI. August 2005. Vol. 27, No. 8, pages 1254-1264.