How many eigenfaces to use?

From the graph we see that as we increase the number of eigenfaces considered, the recognition improves but later on it is seen to saturate. Using more number of eigenfaces requires more time while recognition. So there is a trade off between the speed and accuracy. I also think that using more and more eigenfaces will tend to overfit the system which might not be good when we want to detect faces which are not in the database.

The number of eigenfaces to be used can be decided by looking at the eigenvalues. The eigenvectors corresponding to small eigenvalues only contain information about detailed differences in comparison to a higher level discrimination by the eigenvectors with larger eigenvalues. We can choose to remove the vectors with smaller eigenvalues without affecting the results by a great amount. Hence one way to do this will be to sort the eigenvalues in descending order and plot them as shown below.

What about recognition errors?

As observed from the recognition accuracy graph, we do get a lot of wrong matches. Most of the times, the correct answer did appear among the top few results for the match.

Examples:

query	rank 1	rank 2 (correct)

query	rank 1	rank 4 (correct)

In the second example, the matched faces do look quite similar. (Maybe that's an understatement!)

The mistakes are sometimes good as in the second example and bad as in the first example. In general, it is observed that the system does a reasonable job.

Face detection using eigenfaces

Generating Skin image

I implemented a naive skin detection system first based on the color range values. The color constraints on a pixel to be a skin pixel are -

(R>=1.2*G) AND (G<=8*B) && (G>=0.9*B) && (R<=2.8*G)

To take care of the color noise, I check this constraint over a 11X11 patch centered around that pixel and accordingly make it a potential candidate for a face. This detection works quite well as seen below:




This criterion for skin was empirically set and it is not validated over a big database. Ideally, we need to do some kind of learning on the data we are processing since a single skin color model will not fit people from any geographical location. This is noted in one of the examples later.

Generating Facial Skin image

For face detection, I go one step further and filter potential face candidates based on the number of skin pixels in the patch. This stage is useful to remove the effect of color aggregation in the patch-based method in the previous stage. This threshold has been empirically set to 0.1. So the assumption is that any rectangular patch can be a face only if it has 90% skin pixels inside it. The threshold of 0.1 is quite strict in the sense that it does not allow people to have facial hair. If we want to take care of that, we have to loosen this criterion by increasing the threshold. Thus this threshold can be seen as the maximum portion of face covered with hair.

The skin image after this filtering looks like the following:


Detecting faces from the "Facial Skin" image

Now that we have the potential candidates, we compute the errors on forward and back projections of the candidate on the eigenspace. We choose the top results from this. Also, we take care that any two resulting faces do not overlap with each other much. This overlap is checked by computing distance between the centres of the patches. If this distance is less than 50 pixels, then we choose the better one and discard the other. This threshold has been hardcoded as of now and will have to depend on the expected density of faces in the image.

The following image shows the detected faces in the image (scale: 0.3, top 15 faces).



To see the effect of the facial hair threshold, consider the following image:

Has no "facial skin" with threshold 0.1	Faces detected after threshold increased to 0.4

Examples of cropped faces:

	min_scale=0.45 max_scale=0.55 step=0.01
	min_scale=0.3 max_scale=0.5 step=0.1
	min_scale=0.3 max_scale=0.5 step=0.1

More examples of face detection in group photos



min_scale=0.45, max_scale=0.55, step=0.01, facial_threshold=0.1




min_scale=0.65, max_scale=0.85, step=0.1, facial_threshold=0.4. In this example, I had to augment the skin criterion with a minimum threshold on RGB each. Also, the left most face does not appear in the top 9 faces because of a false positive else where. But it does come up when we choose top 10 faces instead. Also, it seems having illumination changes on the face makes a difference. Here the faces are illuminated from their left side and where this effect is more, the system makes some mistakes.


Choice of scale parameters: In most of my experiments, the following parameters worked well: min_scale=0.3 max_scale=0.7 step=0.05. This is an exhaustive interval over which my images worked well. The interval and step size can be varied by having some estimates about the size of the faces in the image.

Mistakes in face detection - Face detection is not fool proof. During the experimentation I did observe false positives which in turn cause some proper faces to not appear in the final rankings. This is mainly because of the loose training of the system. I also tried using a normalization phase before PCA where I multiplied the MSE by the ratio of distance from mean to standard deviation. But somehow I did not observe an improvement. I think that was the case because the standard deviation is computed only on a small set of observations and is not accurate. There are many parameters in the system and I think that using priors from the data will improve the system.

Face morphing

Click here to see movie

Click here to see movie	Click here to see movie

Exaggerating the differences between the two faces by making them move away from each other in the eigenspace just seems to increase contrast. It does not make much sense to me.

Extra Credit

1. Skin detection using color cues
2. Facial skin detection using voting based mechanism on skin patches
3. Implemented the speedup mentioned on the course web page for computing eigenfaces
4. Face morphing