Assignment 4: Face Detection
CSE 455, Winter 2017

Due: Mar 9 at 11:59pm (Thursday); Late Date Feb 12 at 11:59pm (Sunday)
with penalty 10% per day

Before you start:
Download the assignment's files here.

Assignment:
For this assignment you'll be implementing a face detector. There are three main steps: (1) computing features, (2) learning a classifier and (3) detecting faces in a query image. Feel free to reuse any code from previous assignments. For background, I would recommend reading the paper "Robust Real-time Object Detection", Viola and Jones, Int. Journal of Computer Vision, 2001 (it's included in the homework download called "ViolaJonesIJCV2001.pdf". )

In the project, you should only have to update one file "Project4.cpp. " The buttons in the UI will call the corresponding functions in "Project4.cpp. "

What to tun in:
To receive credit for the project, you need to turn in the completed file "Project4.cpp", any additional files needed for compiling, the requested images and files for each task.

Tasks:

Step 1: (10 pts) For the first step, you'll compute features for the training dataset. Follow these steps:

void DisplayAverageFace (QImage *displayImage, double *trainingData, int *trainingLabel, int numTrainingExamples, int patchSize)

This function should display the average over all positive and a separate average over all negative training instances. That is, compute the average image I_pos over all face images (positive instances) and I_neg over all background images (negative instances. ) The results should look like this:

I_pos       I_neg

The average face is on the left and the average background is on the right. For this assignment you don't need to worry about color, you can just use the graytone images for all questions.

void IntegralImage (double *image, double *integralImage, int w, int h)

This computes the integral image as described in class. That is, every pixel should be the sum of the pixels to its upper left. To test your results, open image "IntegralTest.png" and compare it to "IntegralAnswer.bmp".

Pixels in the output image array are store in a column-first order -- image[x+1] is the pixel right of x, and image[x+w] is the pixel one row down from x. This is similar to HW2.

void ComputeFeatures (double *integralImage, int c0, int r0, int size, double *features, CWeakClassifiers *weakClassifiers, int numWeakClassifiers, int w)

Implement ComputeFeatures, and its two helper functions SumBox and BilinearInterpolation. w is the width of the entire image, and size is the patch size. ComputeFeatures takes as input the randomly initialized weak classifiers. To see how the weak classifiers are initialized, please see InitializeFeatures. Each weak classifier contains 2 or 3 boxes (specified by m_NumBoxes ) defined by the member variable m_Box. Multiply the sum of pixels within each box by m_BoxSign and add them together to find the corresponding feature response. Store the resulting values in the array "features. " Compute the sum of the pixels within a box using the helper function SumBox. SumBox in turn uses the helper function BilinearInterpolation to handle fractional pixel values.

Hint: The values of m_Box store the upper left corner and lower right corner of the box in NORMALIZED coordinates. (c0,r0)=(0,0) is the upper left corner and (c1,r1)=(1,1) is the lower right corner. To compute the corners of the box in the image you'll need to multiply by the box size and add the offset. For example, the "x" coordinate of the upper left corner is computed using m_Box[i][0][0]*size + c0, and the "y" coordinate is computed using m_Box[i][0][1]*size + r0.

Required: Open the dataset "TrainingDataSmall.txt" and press "Average Face". Save the resulting image to "1a.png". Open the image "IntegralTest2.png" and press "Integral Image. " Save the resulting image to "1b.png". Open the dataset "TrainingDataSmall.txt" and press "Compute Features. " Save the resulting image to "1c.png".

Step 2: (20 pts) Next, we are going to compute your classifier using AdaBoost. The AdaBoost classifier is implemented in the function AdaBoost. You need to implement two helper functions used by AdaBoost. Both of these functions are described in detail in the course notes and in the paper "Robust Real-time Object Detection" described above.

double FindBestThreshold (int *featureSortIdx, double *features, int *trainingLabel, double *dataWeights, int numTrainingExamples, CWeakClassifiers candidateWeakClassifier, CWeakClassifiers *bestClassifier)

Given a specific feature, this helper function finds the best threshold, polarity and weight for a weak classifier, as well as its error. The feature's values for each training example are stored in "features" and the weights for every training example are stored in "dataWeights". You'll need to update the member variables m_Polarity, m_Threshold and m_Weight of "bestClassifier". You should return the error associated with the weak classifier, and the weight is just alpha=ln((1.0-error)/error) or ln(1.0) if error=0. To help in computing these features, the array "featureSortIdx" stores the indices of the sorted features. (See HW4 slides!) When assigning the polarity, use either 1 or 0, with 1 = face above threshold, and 0 = face below threshold.

void UpdateDataWeights (double *features, int *trainingLabel, CWeakClassifiers weakClassifier, double *dataWeights, int numTrainingExamples)

Given the addition of a new weak classifier, UpdateDataWeights updates the weighting of each training example. The new weak classifier and the feature value and label of each training example are passed into the function. After updating the weights, make sure they sum to 1 (ie. normalize). See HW4 slides. It is for the version of AdaBoost that Viola-Jones used.

Hint: The function AdaBoost outputs a text file called "AdaBoost.txt" (For Mac users it will be inside the executable package. You can use show content to find it.). Compare your numbers to those found in "AdaBoostTA.txt. " The numbers won't be exactly the same since the features are chosen randomly, but you should notice a similar trend, i.e. you should see the error slowly decrease.

Required: Open "TrainingDataMedium.txt". Press "Compute Features". Press "AdaBoost". Save the resulting image as "2.png" (this took 30 seconds to compute on my machine. ) The first two features should look fairly similar to the features shown in "Robust Real-time Object Detection" in Figure 5. Save the resulting classifier as "classifier.txt". Also turn in your file "AdaBoost.txt".

Step 3: (10 pts) In this step you will implement the functions necessary to detect faces in a query image. The function FindFaces has already been implemented for you. It searches over all scales (between minScale and maxScale) and positions in the image for a face. At each location it calls the helper function ClassifyBox to compute the classification score for a face existing in the box specified by (c0, r0) and size. (c0, r0) is the upper left-hand corner of the box, and size is the width and height of the box.

double ClassifyBox(double *integralImage, int c0, int r0, int size, CWeakClassifiers *weakClassifiers, int numWeakClassifiers, int w)

w is the width of the entire image, and size is the patch size. This should return the classification score for the box computed using your AdaBoost classifier. Hint1: You should be able to use ComputeFeatures as a helper function. Hint2: Class Lecture slides give you a hard classification version of the score. It says that a box is classified as a face if: sum_t alpha_t*h_t(x) > 0.5*sum_t alpha_t. In order to make a score instead of a yes/no come out of ClassifyBox, you just subtract the right side from the left side, ie. score = sum_t alpha_t*h_t(x) - 0.5*sum_t alpha_t. (The alpha_t's are the classifier weights that are stored in the m_Weight fields.) ClassifyBox is called by FindFaces, which is given to you. It checks whether the score is above a threshold and then stores it. This is mentioned in HW4 slides.

void NMS(QMap *faceDetections, double xyThreshold, double scaleThreshold, QImage *displayImage).

This step removes overlapping face detections. If two neighboring face detections exist within xyThreshold of each other in position AND scaleThreshold in scale, remove the less confident face detection.

Required: Open the classifier "classifier.txt" from the previous step. Open the image "barca2.jpg". Press "Find Faces" with default settings. Save the resulting image as "3a.png". Press "NMS" with default settings. Save the resulting image as "3b.png". Adjust the parameters to "Find Faces" and "NMS" to find the best results over the four images in the "query" directory, and save the resulting images as "3c.png, 3d.png, 3e.png and 3f.png".

Bells and Whistles (extra credit)
(Whistle = 1 point, Bell = 2 points)

The function InitializeFeatures uses 3 of the 4 features described in "Robust Real-time Object Detection". Add the fourth feature (looks like 2x2 boxes) to the randomly generated features by updating InitializeFeatures. Show before and after face detections. Does this increase the accuracy of the face detections?

Try training using different datasets sizes and numbers of candidate weak classifiers. To do this make a new training data set file "TrainingData*.txt". The format is:

# of training examples (20000 max)
64 (patch size, should always be 64)
# of candidate weak classifiers (pick as many as you have memory for)
# of weak classifiers selected by AdaBoost (should be less than previous line)
faces (don't change, this is the directory containing faces)
background (don't change, this is the directory containing background patches)

Show face detection results before and after changing the parameters. Do any changes boost the performance?

Implement the cascaded classifier described in "Robust Real-time Object Detection". You'll have to implement a cascade of AdaBoost classifiers. This should significantly reduce your time needed to detect faces.

Gather another dataset of 64x64 patches of another type of object and train your classifier. Can you recognize anything else other than faces?