The eigenfaces

The graph

The number of eigenfaces to be used seems to be around 20. Increasing the dimension can cause one to overfit the data, i.e instead of finding a general face structure the algorithm also encodes the peculiarities of that image into one of the eigenVectors which can result in overfitting and at the time of recognition can result in errors. There doesn't seem to be a clear answer for the number of eigenfaces but some kind of "mid-value" should work.

On left is Aseem's photo. Groupphoto on the right with a scale around 0.95
On left is the groupphoto. There is an additional mark (false +ve) in there. I think that part has some similarity to faces :). I tried scales of around 1.00 - 1.10. On right is my photo (scales = 0.60 - 0.65)
Extra Credit : Implemented the speedup suggested by Amir