Testing recognition with cropped class images

1.Describe the trends you see in your plots. Discuss the tradeoffs; how many eigenfaces should one use?  Is there a clear answer?

            First, my results were not like results that was described in assignment webpage. It said "You should expect only about 56%(14/25) accuracy with 10 eigenfaces". However, my accuracy was more like 32%(8/25) when I was using 10 eigenfaces. At using 21 eigenfaces my accuracy was closed to 56%, which was 60%(15/15).
            Anyways here is some description for my plot - There were two peaks in my plot. First one was around 10 eigenfaces used and second one was around 21 eigen faces used. Looking at my plot again, it seems like using more eigen faces is generally better.
            However, theoretically, that is not true! - There is trade off using more eigenfaces vs. less eigenfaces. Using more eigenfaces, say 100 eigenfaces to be extreme, will have a lot of false positives. Because using a lot of eigenfaces will have larger face hyperspace and it will include even non-face(by human perspective) like space as face space. On the otherhand, if you use too small number of eigenfaces, it will start to have a lot of false negatives. Since, there is only few eigenfaces, the face hyperspace is too small and the program start to reject anything that is not in that small hyperspace - which might be actual human faces!
            Therefore, what I think about my plot is, I should have plotted for more number of eigenfaces. My 21 eigenfaces has similar accuracy with that I should have with 10 eigenfaces(as described in assignment page), and right after 21 eigenfaces I started to see decreasing trend of accuracy. So 25 is too early for my program to see the trade off I explained above. So if I have plotted longer, I might have seen that trade off.


2.You likely saw some recognition errors in step 3; show images of a couple.  How reasonable were the mistakes?  Did the correct answer at least appear highly in the sorted results?

            Mistakes were reasonable. The correct faces were in upper part of the sorted list. For example, using 10 eigenfaces to recognize faces here are few examples for not correctly classified cases -

Face '../data/smiling_cropped/01.tga' recognized as being closest too:
0: ../data/nonsmiling_cropped/14; MSE: 887.011
1: ../data/nonsmiling_cropped/01; MSE: 944.166

Face '../data/smiling_cropped/02.tga' recognized as being closest too:
0: ../data/nonsmiling_cropped/20; MSE: 1256.75
1: ../data/nonsmiling_cropped/22; MSE: 1425.32
2: ../data/nonsmiling_cropped/13; MSE: 1856.15
3: ../data/nonsmiling_cropped/03; MSE: 1906.29
4: ../data/nonsmiling_cropped/09; MSE: 2074.17
5: ../data/nonsmiling_cropped/02; MSE: 2093.62

Face '../data/smiling_cropped/03.tga' recognized as being closest too:
0: ../data/nonsmiling_cropped/01; MSE: 827.866
1: ../data/nonsmiling_cropped/03; MSE: 1179.48

etc..

Cropping and finding faces
1. What min_scale, max_scale, and scale step did you use for each image?

imagefilename: elf.tga minscale: 0.45 maxscale: 0.55 step: 0.01
Comment: (Used main.exe) I could not cropped the faces correctly. Because image with least mse was background low texture area. However, 2nd and 3rd results actually got the faces right - 2nd as adult face, 3rd as baby face.

imagefilename: IMG_0003.tga(nonsmiling group picture) minscale: 0.45 maxscale: 0.55 step: 0.01
Comment:

imagefilename: IMG_0004.tga(smiling group picture) minscale: 0.45 maxscale: 0.55 step: 0.01
Comment: -

imagefilename: IMG_0002.tga(class group picture) minscale: 1.0 maxscale: 1.5 step: 0.01
Comment: To do this findface, I used main_heu.exe, which is using findface with little bit of processing descripbed in assignment page. It worked better than one that not uses it. However, still gets confused by low texture area.

imagefilename: webport.tga(portrait found on web) minscale: 0.25 maxscale: 0.30 step: 0.01
Comment: -

imagefilename: StatGroup200601.tga(group picture found on web) minscale: 0.8 maxscale: 0.9 step: 0.01
Comment: Upper one uses main.exe and lower one uses main_heu.exe. I guess heuristic helps little bit better on getting fooled by low texture area - notice that lower one draw the box on book instead of black shirts area.

2. Did your attempt to find faces result in any false positives and/or false negatives?  Discuss each mistake, and why you think they might have occurred.

            There were several false positives and false negatives. In case of IMG_0002.tga, the class group image, StatGroup200610.tga the web group image, and the elf.tga image. False positives mostly occurred around low texture region where the region is closed to single plain color. I think the reason is if the current testing region is plain, there is no strong features to tell current testing region is not close face hyperspace, then the program think current testing region is close to face hyperspace. It seems like using process that described in assignments webpage helps little bit but not too much.
            There were many false negative case occurred on my experiments on same images (except elf.tga). It seems like false positive mostly occurred when size of faces varies a lot, or different illumination condition, or the rotation of faces. Additionally, any testing faces that are far from face hyperplane that was developed during training time, gets rejected easily. Looking at my successful face finding cases, the faces they found are very close to average face. Especially 2 images I mentioned above.

Verify Face
1. What MSE thresholds did you try? Which one worked best? What search method did you use to find it?

            I tried MSE between 1000 and 4000 with various interval. It seemed like between MSE of 1500~1700 worked the best. The way I found these number goes here:
            To count the number of false positive, I tried to verify smile/01.tga to all other 24 faces. It can have total 23 false positives. To count the number of false negatives I tried to verify all smiling faces to self-nosmile faces. It can have total 24 false negtives. To get best MSE for verifying faces, I added up the rate of false positive and false negaive, the MSE that minimized that number would be the best one. (There is small problem with this approach - I only considered false positive for smile/01.tga. Using false positive rate for average of all 24 faces might give you better result.) So using that added up false positive and false negative rate, I did gradient descent to find some MSE that minimizing added rate x. (I stopped early beacuse I was doing gradient descent by hand.. not by computer).

2. Using the best MSE threshold, what was the false negative rate? What was the false positive rate?
            My best MSE threshold was 1500. The fp rate was 0.12 and false negative rate was 0.41.

MSE&False Negative, Positive Table

MSE1000
#falsePos: 0
#falseNeg : 20/24 = 0.83
total: 0.83

MSE1300
#falsePos: 4/24 = 0.16
#falseNeg : 17/24 = 0.71
total: 0.87

MSE1500
#falsePos: 3/24 = 0.12
#falseNeg : 10/24 = 0.41
total: 0.56

MSE1700
#falsePos: 8/24 = 0.33
#falseNeg : 8/24 = 0.33
total: 0.66

MSE2000
#falsePos: 12/24 = 0.5
#falseNeg : 5/24 = 0.21
total = 0.71

MSE3000
#falsePos: 21/24 = 0.875
#falseNeg : 1/24 = 0.04
total = 0.915

MSE4000
#falsePos: 23/24 = 0.95
#falseNeg : 1/24 = 0.04
total = 0.99