EigenFaces Based Normalization

The purpose of this project is to enable face recognition from one 2D image when there already exists a database of canonical images – one per subject. We chose to start from the EigenFaces algorithm.

Abstract
The purpose of this project is to enable face recognition from one 2D image when there already exists a database of canonical images – one per subject. We chose to start from the EigenFaces algorithm. We proposed that in order that the EigenFaces algorithm could recognize the test images correctly, they have to be canonical too, just as the train images are. Thus the purpose of the project is to make test images canonical, when the test images can be taken from any 3D angle, with any facial expression, and may be wearing sunglasses etc.

The problem
Without any 3D input or models we have to transform any 2D image of a person, under any condition to his basic frontal image, when his identity is obviously unknown.

(1)
The problem seems hard enough but there are still a few complications:
– If the azimuth angle is large enough, a whole side of the face does not appear in the test image and cannot be reconstructed.
– Some parts of the face may be hidden by the nose or sunglasses and cannot be reconstructed.
– In using traditional methods (EigenFaces, Correlation) a face of a person taken from a large angle can often look more like the standard image of another person.

The solution
We assume Brightness Constancy between the train and test images of a person. This means that all pixels remain at the same brightness level, and only their location may change, which allows us to assume the train image is a coordinate transformation of the test image:
(2) . We also assume (in order to make the problem computable) that this transformation is polynomial, with rank up to 3.
We then use optimization methods where we simultaneously calculate the optimal parameters of the transformation between the test and train images, and also the EigenFaces vector of coefficients which defines the identity of the test image.

Results
Before we made any changes to the test images the recognition rate was 19%
After we made the images closer to canonical by finding the eyes the rate was 45%
After we used the first automatic algorithm described the rate was 54%
After using the robust optimization method in the second part the rate was 75%

Another qualitative comparison was made with the FACEVACS 4.0 face recognition module by Cognitec, which was the big winner in FRVT 2002 in the scenario relevant for this project.

 

Artificial image

Regular EignFaces projection

After using the first part

After using the second part

Outliers

The canonical image of the person

 16  15  14  13  12  11
 26  25 24  23  22  21
 36  35 34  33  32  31
 46  45 44  43  42  41
 56  55 54  53  52 51
 66  65  64  63  62  61
 Genuine test image

Regular EignFaces projection

After using the first part

After using the second part

Outliers

The canonical image of the person

 76  75 74  73  72  71
 86  85  84  83  82  81
 96  95 94  93  92  91
 106  105  104  103  102  101
 116  115 114  113  112 111
 126  125 124  123  122  121
 136  135 134  133  132  131

 

The pictures show precisely what is the added value of the optimization process, and exactly why the recognition rates are that much higher after using it.

The outliers images show that the algorithm found automatically which parts in the face are very different from the train image.

Tools
We used a standard PC running Matlab in order to implement our algorithmic solution. Even our most complicated solutions only took a few seconds per image, so there was no need to use any other tools. We also used an OPENCV module for the initial face detection, which was run from Visual C.

Conclusions
In this project we empirically proved our argument that EigenFaces fails to deal with images with severely varying pose and facial expression changes.
We also proved empirically our proposition that some sort of canonization is needed and may improve the recognition rates.
We developed a novel algorithm for automatic face canonization, implemented it in Matlab and tested it on a database of face images.
We showed that our algorithm increased the recognition rates considerably, and also that it produces automatically a transformation which makes the image look closer to canonical.

Acknowledgment
I would to thank my supervisor, Tomer Michaeli for his devoted assistance during the whole project, from the idea stage, through the algorithmic and implementation states. We are also grateful to the Ollendorf Minerva Center Fund for supporting this project.