This project is one of the first projects in THE VISION AND IMAGE SCIENCE LABORATORY which implements computer vision techniques on the LEGO-MINDSTORMS platform.
Abstract
This project is one of the first projects in THE VISION AND IMAGE SCIENCE LABORATORY which implements computer vision techniques on the LEGO-MINDSTORMS platform, the goal of this project was to built moving lego platforms (i.e. cars) and to establish a pursuit of one car after another car, this is done by transmitting frames from the wireless camera onboard the pursuing car to a PC, analyzing each frame using color detection and other computer vision techniques thus locating the chased car, and transmitting movement commands via an infrared tower, to the pursuing car.
The problem
Detecting the chased car in the frame, using well known, low complexity, computer vision techniques such as: color detection, shape feature extraction and region growing. thus, upon finishing the frame analyzing, we can transmit to the persuiting car the corresponding commands based on the current position of the chased car, in a rate which is good enough for real time tracking.
Integrating the system which is made of different hardware and software components, some already implemented, therefore dictating some conditions and interfaces.
The solution
Chart 1 clearly shows the flow of the project, the flow starts with an initialization process in order to extract the color features of the sticker by which the chased car will later on be detected.
In the light blue area each frame is being scanned for the sticker colors, and using the region growing technique, we extract the shape features of the sticker, and compare it with the sticker’s given color and shape features. failure in this area will not lead to frame request, but instead the program will continue to scan until finding another area suspected to be the sticker or the frame’s last pixel.
The sticker is then authenticated by the discovering the presence of the 2nd color, and upon success we can update the history of chased car positions with the last coordinate. According to the chased car last position, commands are being transmitted by infrared to the persuiting car, and another frame is requested.
Flow chart
Tools and Environment
Figure 1 Computer Figure 2 X10 wireless camera Figure 3 IR tower and an RCX unit
The Cars
Figure 4 Chasing car Figure 5 Chased Car (target)
Project environment
1. RCX unit – small programmable unit by LEGO based on the H8 by Hitachi, provides serial I/O, A/D, 16k ROM, 32K RAM and timers. can be loaded up to 6k of programs , it produce voltage to two side engines which control our vehicle according to the value of an inner variable which can be changed while running by the program via the infra red tower.
2. The chase scene – corner of a room at the vision and image science laboratory its important to emphasis that it wasn’t changed in any way to ease the chance by removing colors which are the same or similar to the searched colors only the combination of the two colors attached mustn’t exist in the room.
3. Infra red tower – transmits the commands of movements to the chasing car in 2400 baud up to the distance of 3.5 – 4 meters
4. PC (computer) – ordinary Pentium 4 1.7 GHz 0.5GRAM, ASUS Geforce V7700/64 VIVO graphics card is installed and connected to the receiver side of the cordless camera, infra red tower is connected to the com1 port. Most of the project was implemented in C++ on Visual Studio 6 using MFC capabilities for GUI, Cdib (pictures objects), and ActiveX (for controlling the infra red tower) features.
5. Lego vehicles – moving, versatile, simple and fun to build platform, which is flexible enough to fit the vehicles purpose.
-The chased car – uploaded with an escape program, and has a Lego structure which includes the two colors which are to be searched in 3 directions
-The chasing car – uploaded with a program which changes the movement nature according to an inner variable, has the cordless camera onboard, and a shield from light above it to prevent reflections as much as possible.
6. Cordless camera – transmits the image captured by the lens to the receiver which is connected to the computer via the video card so it can be analyzed by it later. The cordlessness provides independent and free movement.
In the end, the goal of the project was achieved entirely, and the pursuit is performed in an exquisite way, despite that, we have reached some conclusion about computer vision and integration of complex systems.
The target
the criteria in which we decided that the chased car will be chased upon is the combination of two colors attached together (true, one color could have been enough, but we realized that if one color is used, the environment of the chase must not include that color, and therefore creates an artificial feel, so we decided to use a combination of two colors).
This was chosen after considering using differential picture between frames which was abandoned due to the fact that the camera itself is moving and therefore it requires estimation of the movement which results in high complexity. We also considered shape recognition but since the chased car can be seen in all directions and in each direction it has a different shape this idea was abandoned as well.
The colors which were chosen weren’t a coincidence, we decided to chose from the variety of Lego bricks colors, but in order to improve the reliability of the system we chose the ones which have the largest distance in HS coordinates, this was done by doing a histogram of the environment (in HS color space) and than choosing the colors which are the most far from the dense areas on the histogram, finally, blue and red were where chosen.
Color Coordinates
The image is received in RGB coordinates, which represents all colors by combining different levels of red green and blue, the problem with this color space is that under different lighting conditions, there will be a shift in all three coordinates, therefore in order to locate a color we should locate that shift in all three coordinates, this is not an easy matter, and requires some help in the form of different color space named HSV, in this color space H holds the Hue, meaning the essence of the color, the other two coordinates stands for saturation and value, this is helpful because the H coordinate is hardly affected by different lighting conditions, and therefore in our project every pixel dealt with was first Converted from RGB to HSV.
Initialization
The initialization stage is performed once at the beginning of the execution in that stage, the chased car is placed in a special place, and the HSV values of the sticker are extracted to be used for the rest of the chase. This allows configuring the HSV values more specifically and accurately without any affects of difference in lighting conditions between different runs.
The Algorithm
As the initialization part is finished, an endless loop starts, and in the loop an image is requested (using a thread to prevent the I/O operation to block the execution) The image is then being scanned starting with the last coordinate in which the chased car was found (to save frame search time a very efficient scan is performed), upon finding an area of the desired color (Each point in the graph represents 3×3 square compared to the desired color; we decided to avoid searching for a single pixel in order to avoid initiating the region growing process in vain, i.e. only if 7 out of 9 pixels are in the desired color will the region growing process begin, Thus allowing ourselves to be more permissive in the region growing parameters.) an authentication process will begin, which means Seed planted region growing to determine if this is indeed a sticker or just part of the scenery, this is done by examining the shape feature, (size and approximately square shape), so far it has been proven in high probability that the sticker has been found, so now the Locating of the second color of the sticker for high probability authentication can begin, it is scanned in the expected position using our knowledge of the first side of the sticker, and again a region growing is performed in order to extract the second side of the sticker’s features, if again, it is proven to be a side of a sticker we can be certain that the sticker was found (and therefore the chased car) and we can direct the chasing car toward the center of the found sticker, we paint the found sticker in lively colors and also draw the last five positions in which the chased car was found (to improve visualization), and can move the scan another frame.
Seed planted region growing – this is a process which is done in order to spread over an area of a frame which has some criteria which separates him from the rest of the frame starting from a specific point, the spreading is done in an iterative way by examining each neighbor of a pixel which in the group (meets with that criteria), In our case that criteria is to have the same H coordinate value, but in order to improve the spreading capabilities the desired H coordinate value is updated in each iteration, thus allowing to reach places which didnÂ’t meet with the initial criteria.
Direction Decision and sending
After the car was found we direct the chasing car towards the center of the chased car but it is also done according to the size of the sticker that was found, if the size of the sticker is big, then the chasing car is too close and it backs off, because it is possible the chased car is moving towards it, if the size of the sticker is small, the chasing car moves forward or forward with a little turn, in order to catch up with the chased car.
If the sticker is in a normal size the chasing car keeps that distance and makes its turns in place. All these movement decisions are then transmitted to the chasing car by the infra red tower (also done by a thread in order to prevent blocking I/O).
Conclusions
In the end, the goal of the project was achieved entirely, and the pursuit is performed in a very satisfying way, despite that, we have reached some conclusion about computer vision and integration of complex systems.The subject of color and shape detection is troublesome when it is implemented in such a pursuit where there is a change of target angle and the position of the camera, it creates different lighting conditions and reflections, which makes the color detection much harder. When integrating complex systems, one must not only worry about all the elements communicating together using their interfaces (quite a problem by itself), but also make sure that each element is performing well enough on its own, thus preventing ‘bottle necks’ of performance (hard to achieve when some elements are already given)
Acknowledgment
We are grateful to our project supervisor Guy Gilboa for his help and guidance throughout this work.
We are also grateful to the Ollendorf Minerva Center Fund for supporting this project.