the multiplication in which can be easily implemented using panning.
The α we used in this project was 0.25, which was an experimental value that balanced the tracking sensitivity and displaying stabilization. Since for smaller α, which indicated the old_input dominate the output, the cursor would be more stabilized but “harder” to move.A mean filter was also be tied in the project , but the result was not as good as the low pass filter.
Mandelbrot Set - Parallelization Scheme :
In this project, we used a completely different method to implement the Mandelbrot Set comparing to Lab4. Instead of using SRAM to store the iteration value n, which was related to color, no writing or reading from SRAM or M9K was applied in this project.
Referring Nerdy Stuff’s multi-pipelined Mandelbrot Set,we use a real-time calculation and displaying of each pixel on VGA screen. For every complex number on the screen (every pixel) 23 pipelines needed to be go through to calculate the iterations before convergence. Which means the result displayed on the VGA screen were actually the pixel calculated 23 VGA clocks before, which was the pixel 23 ahead in x direction. This won’t affect the display result so significantly as long as the pipelines we used were limited. And also, for each pipeline doing one - time iteration, using three multipliers, the number of pipeline also limited by the multipliers that available on DE2_115 board.
The advantage for this method is that we could make sure the time for one whole screen calculation was one VGA frame time exactly, no matter what the zoom level. To make sure of this, the time for one pixel calculation could never longer than 40 ns, which is the VGA scanning time for one pixel (under 640 by 480 resolution). This also set the limit for the number of pipelines.
The diagram below explaining the Mandelbrot set parallelization scheme.
Figure.5 Dataflow for Multi-pipelined Mandelbrot Set
Interface Controlling Mandelbrot Set :
Figure.6 Block Diagram for Interface
For the panning control, first thing need to figure out was the direction of the moving hand. For instance, when hand moves toward right, mean more patterns would be seen on the left side, then the new image should be updated with initial position shift a little bit to left. In our project, four directions, right, left, upper and down are implemented, and the speed for panning was 8 pixel per frame.
The zooming effect is realized by adjusting the resolution of real and imaginary step for each pixel on the Mandelbrot Set. In order to present a zoom-in effect, the difference between each pixel should become smaller as one’s hand approaches closer to the video camera; and vise versa. We introduced a zooming factor to quantitatively describe the resolution which determines both real and imaginary step for pixels in the Mandelbrot Set in roughly a linear relation. This conversion is done by the Converter 2 in the zooming module (see Figure ).
According to the relative distance between and and the camera, which is roughly estimated by calculating the area of the hand. This can be easily acquired by multiplying the width and height of the rectangle that fits right out of the hand. The zooming factor will have a linear relation to the percentage of pixels in the camera field covered by the hand. This conversion is done by the Converter 1 in the zooming module (see Figure ).
Controller : In order to have a smooth effect, there should not be any abrupt change in the increment or decrement on the zooming fraction. We adopted the most simple way by setting a constant value for both the increment and decrement, which proved to have a very responsive and smooth effect for zooming. Whenever the error between the expected zooming fraction is larger than a tolerance range, the actual zooming fraction will be adjusted until it reaches the expected value. This controller forms a negative feedback loop in the zooming module, which is illustrated in the control block diagram (see Figure ).
Figure.7 Zoom Diagram Block
Displaying the Final Product :
On the top module three switches control what is displayed on the VGA screen. The first mode is to display the down sampled skin color image, with detected skin color displayed in white blocks while black for the rest of the area. This mode could help us first check the lighting conditions for the hand tracking. We want to make sure the skin color could be correctly detected before moving to the next steps.
The second mode is the camera image with a cursor following the moving hand. Cause the cursor needed to be trained at reset, this image could help to make sure the hand position was correctly calculated.
The third mode the Mandelbrot Set image with hand position controlled. Cause we doing the zoom in and out with panning at the same time. So sometimes the image would suddenly zoom out because the user’s hand might move out of the camera. To make the display more explicit, we added an additional switch to display the panning only.
Code Reference :
For Hand tracking part, We referred “ Hand Tracking Pong” former ECE 5760 final project by Hanting Lu and Kedari Elety and Realtime Face Tracking - Perspective Projection on FPGA by Chuck Yang and Jasper Schneider. The link to their project can be found here. Also, “Fruit Ninja with video tracking” by Yuan Cui, Jin Sha and Wei Wang was referred for the centroid calculation method.Code could be found here.
For Mandelbrot Set, We referred Nerdy Stuff ’s blog for the multi-pipeline scheme.
Other Attempts :
There is an alternative way which promises to ensure real-time update while keeping enough fixed-point precision for differentiating the details in Mandelbrot Set. As is demonstrated in Figure , this approach fully makes use of the memory available on the board. While the clock rate for VGA displaying module is limited to 25 MHz, we could run the SRAM module and the computation module at fourth this speed. We have rewritten the module for each pipeline so that it does the calculation for each iteration within only one cycle. Each pipeline only accepts an zr and zi coordinate of a point on the Mandelbrot Set and outputs the result back to the upper module.
We have successfully drawn a full Mandelbrot Set using producer-consumer scheme with a single pipeline. The speed is within 0.1 seconds which as fast as what we achieved using 8 pipelines in the previous approach. After we implemented the multi-pipeline scheduler, we could expect to achieve 10 frames per second when the Mandelbrot Set has been zoomed in by a factor of 2, which promises to give a quite smooth visual effect in real-time.
• Speed of Execution:
Our system operates in real time to display the Mandelbrot Set controlled by a moving hand. Every time the whole image was recalculated and the whole screen could finish calculation in one frame time so the speed was pretty fast and no black screen would be seen when new images updated.
• Accuracy: :
For hand tracking, in most cases the cursor could follow the hand closely, but under relatively high speed of a moving hand, the cursor might lost tracking. This situation could be modified by increasing the α sightly, but it might also affect the stability of the cursor.
Since the way we did the centroid calculation was to find the four edges of the skin-colored chunk, users are recommended to wear long shirts or wear a watch to control the Mandelbrot Set.
The details of the Mandelbrot Set were not perfect, since we only implement one iteration per pipeline, which means considering the limited multiplier resource on DE2- 115 board and the timing issues, the iterations for each pixel was limited. This caused the details of the Mandelbrot Set was hard to be seen. For the same reason, the Mandelbrot Set could not be deeply zoomed in either. The limited zoom in factor in our project was 2.
The images below showcase some of the features of this project.
• Safety in the Design, Interference, and Usability:
There are no major safety hazards in this design. The system also does not interfere with other projects. Cause the hand tracking cursor was trained first at reset, so unless a big chunk of skin color moving in the background, the hand tracking would not be interfered.
This project is usable by anyone with bare hand. One thing to note is that users are recommended to wear long shirts or wear a watch or anything that can block the skin color on wrist.
In this project, we successfully prototyped a platform which allows players to interact in real-time with Mandelbrot Set. The system can correctly interpret the user’s intention based on the geometric features of a hand and respond to user's gestures at video rate. In addition, the change rate of the image features is greatly stabilized, providing a smooth visual effect.
Still, this prototype can be further improved in terms of the details in the Mandelbrot Set. Currently, the Mandelbrot Set has a limited zooming depth because the bit width of the complex numbers in Mandelbrot Set is set to be only 27 bit. More color details of the Mandelbrot Set can be shown without artifacts if we can better utilize the hardware resources (memory) in the FPGA by doing calculation at higher clock rate.
Human-machine interaction would be an even more interesting focus to work on. In order to make the panning more stable, we need to tweak for the best overall performance of downsampling ratio and the cut-off frequency of the spatial low-pass filter. Meanwhile, we find out in experiments that the light condition in the room would affect the correctness of skin-color detection. Other image processing methods such as mean filter can be tested to see their effectiveness. For image zooming, we can try to get a more accurate measure of the hand area by extracting more information out of the irregular shape of the hand. This would certainly be challenging as it increases the computation workload for the FPGA. We can also limit the distance range between the camera and one’s hand so that we can eliminate the confusion between a small hand projection and half of a hand within camera range when the hand is very close to the camera. After these improvements, it would be able to present better effect if zooming center is at the hand’s centroid instead of at a corner of the VGA screen.
The link to the video demonstration of the project is here.