"A two-player Rock Paper Scissors game playable on two serially-connected FPGAs"
For our final project in ECE 5760, we designed a two-player rock-paper-scissors game playable on two separate Altera DE2 boards. The system takes rock-paper-scissors input through an NTSC camera, communicates with another DE2 via RS-232 serial communication, and uses a Nios II processor to print all relevant outputs to a VGA screen.
The team with their finished product.
High Level Design top
We wanted to work on a project that deals with many different aspects of FPGA we learned throughout the semester. We also would like to work on the things we haven’t worked on yet such as serial communication and TV decoder. We wanted to have a project that is interactive and requires more than one user. We also wanted to have fun while working; therefore, we chose to implement a game. We wanted to implement the Rock, Paper, Scissors, and Shoot (RPSS) game. The camera captures the movement and communicates with the other user’s FPGA through serial communication. We first get the image from the video camera through the TV decoder then turn the data into the YCbCr environment. Then we pass the data through our skin recognition algorithm, and then turn the values into RGB values to be displayed on the monitor. Using the image on the monitor, the user calibrates his moves. The FPGA saves these moves, by calculating the white pixels on the black and white display. The white pixels correspond to the skin, therefore the hand. Then when the user plays the game, it captures its current move and compares it to the calibrated moves and understands the current move according to that. It communicates with the other FPGA through RS232 protocol.
We get aided by the previous final projects on how to use the video camera, and get the data from the camera. We used the Probabilistic Skin Recognition on YCbCr Environment algorithm for skin recognition. Again, the previous final projects were helpful to figure out how the RS232 works on hardware. We also used NiosII processor to display the game. By using a switch the user can see the videos, or the status of the game.
Movement Recognition (Rock Paper Scissors)
We used an uncommon approach for movement recognition. Instead of the edge recognition, we worked on the area. Since there are only three approaches and the area on the monitor of three movements are so different, we took advantage of this aspect of the game. We calculated of the area of the three different movement during the calibration and we compared the area of the current movement to these values, to see which one has the closest area, which will tell us the current movement. Since our skin recognition worked really well, this approach worked without a problem.
Logical flow of system
The diagram above explains the logical flow pretty clearly. The data from the video camera is captured by the TV Decoder, which saves the data in YCbCr in the SDRAM. Then the YUV modules read this data and turn into a 16 bit data for the color of each pixel. The skin filter checks this data and turn it into a black and white image, the YCbCr values are translated to RGB values. Then it goes through the mirror module to mirror the image so it is not reversed. This data is kept in the VGA buffer. The VGA controller chooses whether this data or the data from the NiosII processor, which displays the status of the game. The data is transmitted and received by RS232 module in the hardware.
We are not using the edge recognition system and we are using the area to figure out the movement. Therefore, the user cannot move his hand in the x- coordinate, which will change the area drastically. This is a software tradeoff due to the hardware we are using. We could have done edge recognition but for our project it was not necessary. We ended up using the NiosII and changing the display takes less than second but there is delay due to the hardware. However, we wanted to take this tradeoff as displaying the status of the game will be really hard by just using the hardware. The NiosII display is also limited due to the hardware. We could have spent more time to have a better display, more user-friendly and aesthetic display; however, it was not worth it as the algorithm behind the game is more satisfying than the aesthetics. So we concentrated more on the game not having any glitches or errors whatsoever.
Existing Copyrights and Standards
For serial communication we used RS232 protocol and for TV decoder we used YCbCr environment and for the display we used ISE, which is the VGA protocol. We used serial communication units from Altera IDE, VGA Controller written by Skyler Schneider.
Program Design top
The Verilog code is broken into various modules for the different interfaces native to the DE2 board. Much of the code for TV decoding and VGA output come from Bruce Land’s ECE 5760 webpage. The RS-232 serial transmission code is largely borrowed from Dr. John S. Loomis, Associate Professor of Electrical and Computer Engineering at University of Dayton. The code is based heavily on his RS-232 example code for performing both transmitting and receiving of data between two DE2 boards. In order to display text for the actual game, we use an instantiated Nios II processor system (designed using Altera’s SOPC Builder).
The TV decoding stage is where we perform skin-tone analysis. While the video signal is in YCbCr space, we gate skin tones to white using the following math:
Math for detecting skin tones in YCbCr space
If a pixel is found to be in the above range, it is displayed as white. To create higher contrast on the output, we additionally turn any non skin-tone pixels to black. This output is fed into an SDRAM module, which acts as a buffer for the TV signal before it is sent to the VGA controller.
Faces after skin detection
This black-and-white video signal, however, is only one of the signals we need to send to the VGA controller; the other comes from the instantiated Nios II system on the board. This system is used mainly for printing messages to the VGA screen relevant to the game’s execution. It has parallel IO’s mapped for a ready signal (switch 16), the data transmitted to the other board over RS-232, the data received from the other board over RS-232, and a valid bit used for timing wired to the board’s VGA_VS pulse (operating at 60 Hz). The Nios II system then determines the winner or loser of the rock-paper-scissors game for each round, maintains the timing of the game, and performs printing of all relevant messages to the VGA screen. The two VGA outputs (SDRAM and Nios II) are multiplexed by switch 17 on the DE2, which allows users to easily switch between camera mode and game execution mode.
The actual move detection is done through a custom module, which takes calibrated data for each of the three moves and the current move on screen as inputs. These calibrated points are the sum of all the skin-tone pixels of a screen buffer, saved using the KEY pushbutton and a combination of switches. When switch 0 is high and KEY is pressed, the system saves calibration data for rock. When switches 0 and 1 are high and KEY is pressed, the system saves calibration data for paper. When switches 0, 1, and 2 are high and KEY is pressed, the system saves calibration data for scissors. Once the user has finished calibrating their moves, they can see their move at any time on the seven-segment displays, and their current move is constantly transmitted to the other board.
Rock Paper Scissors with skin detection
The software part of the game is written in C and is run on the Nios II processor. It is operated through a state machine, which determins what to print to the screen based on whether the user is ready, the opponent is ready, and what move has been played.
Screen showing waiting for opponent
There is a gap of 1 second between each move cue (“Rock!”, “Paper!”, “Scissors!”, “Shoot!”) to simulate the actual game; this one second gap is generated using the parallel IO VGA_VS pulse, operating at 60 Hz. On the “Shoot!” stage, the program analyzes the transmitted data to see what move the user played as well as the received data to see what move the opponent played. These two moves are then compared to determine the winner of a point; rock beats scissors, paper beats rock, and scissors beats paper, with two of the same moves resulting in a tie. After the winner of the point is displayed on the screen, the score of the winner is incremented by 1 and the game begins again from “Rock!”. Once a player hits 3 points, the game ends and displays the winner.
Screens showing winner and loser as well as scores
At the end we could say that our project is working without any major or minor errors. The skin recognition is very well; it even recognizes the people’s skin from pictures. Detecting the movement from motion capturing is also flawless. Once you calibrate it, it detects the movements perfectly. The speed of execution is really good as the communication is done every micro second and the maximum delay between the two users is only 2 microseconds, which is undetectable by humans. Since the users are the humans in our project, this delay is also negligible.
The video display of the hand has slight noises; however, it does not affect the algorithm. The skin detection is not perfect but it is good enough as far as this project goes. The synchronization of the VGA between the NiosII processor and the hardware is really good as they are working on the same clock. It is accurate enough to play the game without ever finding any bugs. Algorithm finds the winner perfectly as there is no corner case where the project will choose the wrong player to win.
There are absolutely no safety issues as the project only has the FPGA, a video camera, a monitor, and a serial communication cable. The only problem could be the electricity; however, that has nothing to do with our project specifically and there is nothing we can do to avoid it.
The game is very user-friendly with its feature that you can constantly change the display between the video camera display and the status of the game display. You see what your move is on the screen; constantly aware of the score and the delay of the “rock paper scissors shoot” makes the game look real as it is how you would play in real life. Calibration of the movements may be the only disturbing feature to the usability of the game, but we tested the system with people of varying skin tones and the system still works well. Any person can play and enjoy the game.
The project met all of our expectations. Whatever, we promised in our initial proposal was implemented and works flawlessly. One thing we could have done is to get rid of the calibration stage and use the edge detection to figure out the movements to make the game user-friendlier. However, in this style it is still easy to play and the detection of the movements has no error. The skin recognition can be perfected so there would be no noise whatsoever; however, again, this was not necessary as far as this project goes. The designed conformed the applicable standards the way we have been using through the semester. The examples on the web page were really helpful to understand how to stick to the standards. For RS232, Rx and Tx cables were reversed, which was the necessity for this standard. The other important thing we kept in mind that we ran the VGA on 25.2 MHz, which also follows the VGA standard.
Intellectual Property Considerations
We used Skyler Schneider’s VGA Controller, an NTSC video example from the class webpage, and an example of Serial Communication written by John S. Loomis (referenced in appendices). We also used the Probabilistic Skin Recognition Algorithm in YCbCr environment that we found in the public domain. The game implemented is universal and anonym so we did not need to worry about any patent issues. We implemented the design from scratch so we did not reverse engineer an existing design. We used the video camera from the classroom, which was our only additional hardware. We cannot patent this game, as it is universal and anonym, yet we can still make it an actual business by selling the algorithm. However, in that case we have to get permissions and sign contact with Skyler, Bruce and Altera. We are not seeking for using this project in any professional way.
A. Distribution of Work
The two of us worked together for most of the lab, switching off at the computer in order to gain full understanding of the system. Roshun worked on getting serial communication working, Baturay worked on getting skin recognition working, and the two together worked on putting everything together and designing the actual game.
B. Code Listing
Full Project Zip
This section provides links to external reference documents, code, and websites used throughout the project.
We would like to sincerely thank Professor Bruce Land for teaching this class and his help in implementing this project.
We would also like to thank the lab TAs for their incredibly useful debugging help.