EigenFaces

The first trend in these results is that as you add more eigenfaces, the recognition becomes better. This makes sense because adding more eigenfaces essentially adds more detail to the abstract face description (coefficients) and allows the algorithm to consider more "attributes" of a face when finding a match.

The next trend is that beyond a certain number of eigenfaces, the matching ability of the algorithm does not increase much and can even decrease. Smiling faces really are different from non smiling faces. When using lots of eigenfaces, you are projecting these smiling faces into a space that doesn't characterize the variance of smiling faces well and the results become less meaningful.

I don't think this algorithm could achieve perfect recognition and wouldn't be surprised if the recognition rate became less accurate when even more faces were added to the experiment. The optimum number of eigenfaces to use would be the number that can amplify the variance between faces yet not be sensitive the variance between smiling and non smiling. It seems reasonable to use 10 for the rest of this project. Another approach, if we could use the smiling faces for training, would be to increase the number of eigenfaces and characterize how a face changes when smiling. Then a more sophisticated discrimination algorithm could be used that explicitly groups smiling and non smiling images of the same person together. The Moghaddam article that Steve pointed out does this to a certain extent [Moghaddam, B., "Principal Manifolds and Probabilistic Subspaces for Visual Recognition", IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), June 2002 (Vol 24, Issue 6, pps 780-788)]

The eigenface technique obviously could not correctly identify every smiling face. The following images display more detail about some of the queries and were done using 11 eigenfaces. I have shown some of the results for a successful query - the correct face was identified, for an almost correct query - the correct face was ranked second, and for a failed query - the correct face was ranked 11th. It is interesting how the reconstruction of the smiling face doesn't have a smile. This is because the eigenface training was only done on non smiling faces. Looking at these samples, it would seem that the primary determinant for a successful match is how much a particular individuals face changes from non smiling to smiling. Smiling Ko looks very different from non smiling Ko, the cheeks are raised and the eyes are smaller. The reconstructed versions of these faces are relatively far apart.

Recognition query that worked
	Smiling Query	Closest matches from non smiling database
Name	Mullen	Mullen	Hazen	Echart	Lester	Gauthier
Original
Reconstructed
MSE		8.0 x 103	5.2 x 104	5.2 x 104	6.1 x 104	6.3 x 104

Recognition query that almost worked
	Smiling Query	Closest matches from non smiling database
Name	Hoyt	Gauthier	Hoyt	McNamara	Lane	Simon
Original
Reconstructed
MSE		2.0 x 104	2.3 x 104	2.7 x 104	2.9 x 104	3.1 x 104

Recognition query that didn't work
	Smiling Query	Closest matches from non smiling database
Name	Ko	Zhang	Mccnew	Gauthier	Li	Ko
Original
Reconstructed
MSE		4.3 x 104	4.9 x 104	5.2 x 104	5.5 x 104	8.6 x 104

---

Cropping and finding faces

I used the technique mentioned in the assignment for the error function. The MSE is multiplied by the distance the face is from the average and divided it by the variance of the face. This worked OK, but still could get fooled by low texture areas. Using this technique is quite slow because every pixel is scanned at a number of scales.

Below are some examples of found faces. They were found using 10 Eigenfaces trained with the non smiling images. The algorithm didn't work for every picture I tried. Some of the reasons are:

• The faces in the training set all had uniform illumination. When I tried finding faces in outdoor images, the shadows on the face fooled the algorithm. One could get around this by adding some outdoor images to the training set.
• It appeared that the algorithm had trouble with different lighting/exposure, it couldn't find bright faces. The algorithm subtracts out the average face, but does not do any offset adjustment to compensate for bright faces. Simply offsetting each training and test image so that the average of the pixel values was zero might have helped.
• This algorithm was only looking at intensity variations that formed a facial pattern. Adding a color test would probably help through out non face regions.
• The algorithm was still fooled by low texture areas. Projecting a low texture area onto the eigenfaces resulted in low coefficients. When a face was reconstructed from these coefficients, it ended up being very close to the original and therefor would edge out valid face regions. A better error term could probably be found.

	Cropped Aseem Optimum scale: 0.47 Error: 231
	Cropped Peter Optimum Scale: 0.72 Error: 226

min_scale: 1.0, max_scale: 1.1, scale_step: 0.02

min_scale: 0.56, max_scale: 0.62, scale_step: 0.01

---

Whistle 1 - Image Morphing

I morphed from Aseem to myself using the cropped images from above and 29 eigenfaces trained on the non smiling images:

The I morphed from myself away from Aseem:

---