Simplified Gesture Recognition

Abstract
The goal of this project was to develop a program for real-time gesture recognition: A continuous stream of hand signs images is recorded and has to be processed immediately. The camera that records these signs provides pictures in real-time conditions to a PC that has to classify the signs. Such systems should offer the possibility to command computers just using hand signs.
In order to simplify the task, the signs range has been restricted, and finally, the system to be design will classify the different sign thanks to an estimation of the number of fingers in the incoming picture.

The problem
The first issue to be discussed is linked to the real-time acquisition and processing of pictures in a Matlab environment. Then, some more technical issues linked to the algorithm of recognition have to be solved:
In a large incoming picture, how to find the relevant part, say the hand without its background? Indeed, the user will never exhibit his hand exactly at the same place regarding to the background. And once this relevant part is available, how to read the sign and classify it correctly, say count the number of fingers?

The solution
The real-time constants can be managed using a Graphical User Interface that displays at every time the input picture and the result of its analysis. C++ environments are efficient enough for such issues.

Concerning the necessity to find the relevant part of the hand in the whole input picture, the following few steps are required:

• Creating a backup of the background
• Differentiating the input picture and the background
• Creating a binary picture thanks to a threshold
• Collecting the non-zero area and rejecting the palm area

By this way, it is possible to collect only the relevant part of the picture. Then, several methods have been tested for the classification:

• A non-efficient method consist in counting to the number of pixels set to ‘1’, which increases almost linearly with the number of fingers
• A non-efficient method consist in classifying according to the number of edges in the picture, which depends on the number of fingers
• A very efficient method is based on a simple Neural Network with only eight neurons in its single hidden layer
• Another efficient method, which associates weights to pixels and calculates the weighted average of the relevant pixels

Tools
Image acquisition has been done using a Panasonic BL-200 color video-camera (in mode 320×240 RGB) linked to a PC that should includes the following softwares:

• Asus Live: acquisition and recording of single images or of movies as avi-files, usable in the Matlab environment
• VideoOCX : real-time acquisition of images realized in a C++ environment using a Wide set of C++ ready-for-use functions
• Microsoft Visual C++ 6.0: developing the GUI and running the C++ real-time stand alone application using the Matlab Add-In
• Matlab 6.1 (or more recent): developing the algorithm and running the Matlab files

Conclusions
According to the requirements and constraints detailed in the related documentation, two efficient methods have been developed:
Neural networks, which have already been proven efficient for classification issues, have been used and lead to very satisfying results: indeed the rate of successful recognition can reach 95%. The difficulty when using this method is to perform the training and all the preprocessing it requires.
Another method that doesn’t use neurons, but based on an observation and an analysis of the internal work of the neural network, has been developed: it combined pixels counting and weighting with number of edges. In the documentation, it has been explained why does such geometrical processing and the proposed neural network have lookalike properties and performances.

Acknowledgment
I am grateful to my project supervisor Johanan Erez for his help and guidance throughout this work, and also for taking care of me as Exchange Student.
I’m also grateful to the Ollendorff Minerva Center Fund for supporting this project.