Two-Player FPGA Rock-Paper-Scissors

High Level Design    top

We wanted to work on a project that deals with many different aspects of FPGA we learned throughout the semester. We also would like to work on the things we haven’t worked on yet such as serial communication and TV decoder. We wanted to have a project that is interactive and requires more than one user. We also wanted to have fun while working; therefore, we chose to implement a game. We wanted to implement the Rock, Paper, Scissors, and Shoot (RPSS) game. The camera captures the movement and communicates with the other user’s FPGA through serial communication. We first get the image from the video camera through the TV decoder then turn the data into the YCbCr environment. Then we pass the data through our skin recognition algorithm, and then turn the values into RGB values to be displayed on the monitor. Using the image on the monitor, the user calibrates his moves. The FPGA saves these moves, by calculating the white pixels on the black and white display. The white pixels correspond to the skin, therefore the hand. Then when the user plays the game, it captures its current move and compares it to the calibrated moves and understands the current move according to that. It communicates with the other FPGA through RS232 protocol.

We get aided by the previous final projects on how to use the video camera, and get the data from the camera. We used the Probabilistic Skin Recognition on YCbCr Environment algorithm for skin recognition. Again, the previous final projects were helpful to figure out how the RS232 works on hardware. We also used NiosII processor to display the game. By using a switch the user can see the videos, or the status of the game.

Movement Recognition (Rock Paper Scissors)

We used an uncommon approach for movement recognition. Instead of the edge recognition, we worked on the area. Since there are only three approaches and the area on the monitor of three movements are so different, we took advantage of this aspect of the game. We calculated of the area of the three different movement during the calibration and we compared the area of the current movement to these values, to see which one has the closest area, which will tell us the current movement. Since our skin recognition worked really well, this approach worked without a problem.

Logical Structure

Logical flow of system

The diagram above explains the logical flow pretty clearly. The data from the video camera is captured by the TV Decoder, which saves the data in YCbCr in the SDRAM. Then the YUV modules read this data and turn into a 16 bit data for the color of each pixel. The skin filter checks this data and turn it into a black and white image, the YCbCr values are translated to RGB values. Then it goes through the mirror module to mirror the image so it is not reversed. This data is kept in the VGA buffer. The VGA controller chooses whether this data or the data from the NiosII processor, which displays the status of the game. The data is transmitted and received by RS232 module in the hardware.

Hardware/Software Tradeoffs

We are not using the edge recognition system and we are using the area to figure out the movement. Therefore, the user cannot move his hand in the x- coordinate, which will change the area drastically. This is a software tradeoff due to the hardware we are using. We could have done edge recognition but for our project it was not necessary. We ended up using the NiosII and changing the display takes less than second but there is delay due to the hardware. However, we wanted to take this tradeoff as displaying the status of the game will be really hard by just using the hardware. The NiosII display is also limited due to the hardware. We could have spent more time to have a better display, more user-friendly and aesthetic display; however, it was not worth it as the algorithm behind the game is more satisfying than the aesthetics. So we concentrated more on the game not having any glitches or errors whatsoever.

Existing Copyrights and Standards

For serial communication we used RS232 protocol and for TV decoder we used YCbCr environment and for the display we used ISE, which is the VGA protocol. We used serial communication units from Altera IDE, VGA Controller written by Skyler Schneider.

Program Design   top

Verilog

The Verilog code is broken into various modules for the different interfaces native to the DE2 board. Much of the code for TV decoding and VGA output come from Bruce Land’s ECE 5760 webpage. The RS-232 serial transmission code is largely borrowed from Dr. John S. Loomis, Associate Professor of Electrical and Computer Engineering at University of Dayton. The code is based heavily on his RS-232 example code for performing both transmitting and receiving of data between two DE2 boards. In order to display text for the actual game, we use an instantiated Nios II processor system (designed using Altera’s SOPC Builder).

The TV decoding stage is where we perform skin-tone analysis. While the video signal is in YCbCr space, we gate skin tones to white using the following math:

Math for detecting skin tones in YCbCr space

If a pixel is found to be in the above range, it is displayed as white. To create higher contrast on the output, we additionally turn any non skin-tone pixels to black. This output is fed into an SDRAM module, which acts as a buffer for the TV signal before it is sent to the VGA controller.

Faces after skin detection

This black-and-white video signal, however, is only one of the signals we need to send to the VGA controller; the other comes from the instantiated Nios II system on the board. This system is used mainly for printing messages to the VGA screen relevant to the game’s execution. It has parallel IO’s mapped for a ready signal (switch 16), the data transmitted to the other board over RS-232, the data received from the other board over RS-232, and a valid bit used for timing wired to the board’s VGA_VS pulse (operating at 60 Hz). The Nios II system then determines the winner or loser of the rock-paper-scissors game for each round, maintains the timing of the game, and performs printing of all relevant messages to the VGA screen. The two VGA outputs (SDRAM and Nios II) are multiplexed by switch 17 on the DE2, which allows users to easily switch between camera mode and game execution mode.

The actual move detection is done through a custom module, which takes calibrated data for each of the three moves and the current move on screen as inputs. These calibrated points are the sum of all the skin-tone pixels of a screen buffer, saved using the KEY[2] pushbutton and a combination of switches. When switch 0 is high and KEY[2] is pressed, the system saves calibration data for rock. When switches 0 and 1 are high and KEY[2] is pressed, the system saves calibration data for paper. When switches 0, 1, and 2 are high and KEY[2] is pressed, the system saves calibration data for scissors. Once the user has finished calibrating their moves, they can see their move at any time on the seven-segment displays, and their current move is constantly transmitted to the other board.

C code

The software part of the game is written in C and is run on the Nios II processor. It is operated through a state machine, which determins what to print to the screen based on whether the user is ready, the opponent is ready, and what move has been played.

Screen showing waiting for opponent

There is a gap of 1 second between each move cue (“Rock!”, “Paper!”, “Scissors!”, “Shoot!”) to simulate the actual game; this one second gap is generated using the parallel IO VGA_VS pulse, operating at 60 Hz. On the “Shoot!” stage, the program analyzes the transmitted data to see what move the user played as well as the received data to see what move the opponent played. These two moves are then compared to determine the winner of a point; rock beats scissors, paper beats rock, and scissors beats paper, with two of the same moves resulting in a tie. After the winner of the point is displayed on the screen, the score of the winner is incremented by 1 and the game begins again from “Rock!”. Once a player hits 3 points, the game ends and displays the winner.

Screens showing winner and loser as well as scores

Results    top

Performance

At the end we could say that our project is working without any major or minor errors. The skin recognition is very well; it even recognizes the people’s skin from pictures. Detecting the movement from motion capturing is also flawless. Once you calibrate it, it detects the movements perfectly. The speed of execution is really good as the communication is done every micro second and the maximum delay between the two users is only 2 microseconds, which is undetectable by humans. Since the users are the humans in our project, this delay is also negligible.

The video display of the hand has slight noises; however, it does not affect the algorithm. The skin detection is not perfect but it is good enough as far as this project goes. The synchronization of the VGA between the NiosII processor and the hardware is really good as they are working on the same clock. It is accurate enough to play the game without ever finding any bugs. Algorithm finds the winner perfectly as there is no corner case where the project will choose the wrong player to win.

Safety

There are absolutely no safety issues as the project only has the FPGA, a video camera, a monitor, and a serial communication cable. The only problem could be the electricity; however, that has nothing to do with our project specifically and there is nothing we can do to avoid it.

Usability

The game is very user-friendly with its feature that you can constantly change the display between the video camera display and the status of the game display. You see what your move is on the screen; constantly aware of the score and the delay of the “rock paper scissors shoot” makes the game look real as it is how you would play in real life. Calibration of the movements may be the only disturbing feature to the usability of the game, but we tested the system with people of varying skin tones and the system still works well. Any person can play and enjoy the game.

Full System