Introduction

---

Our project design is an exquisite PIC32-based gaming console that supports NTSC video output, audio output and takes input from a NES Controller. Also we have fully developed an easy-to-use game engine that PIC hobbyists can rapidly develop any favorite game on the console. With the console comes with a sample game that we implemented with the engine, Rope Jumper.

We have always been enjoying the entertainment from game consoles, since the 1970s when the home consoles emerged to the market. However, we have never explored the possibilities of realising the game on a small microcontroller. Technology constraints occur at many places: memory is the top concern of building anything with graphic interface; limiting number of accessible hardware, such as timers and DMA channels will also impede on the available functionality of the device; finally, the core frequency of the MCU dictates how many pixels we can synthesize and process for video output, and it constraints the possible resolution of the output.

Therefore, we started by looking into the methods of synthesizing video output from the microcontroller. Then we moved on building the game engine itself with an incremental design approach. At last, we implemented the sample game, Rope Jumper, using the pre-built game engine and explored ways of reducing graphic artifacts on the display as well as relieving heavy memory usage caused by large chunks of audio data.

High Level Design

---

Gaming Console

The idea of the project comes from multiple aspects. Firstly, the existing Nintendo game “WarioWare, Inc.” is very famous of all its interesting minigames and how enjoyable it can be to quickly switch between different minigames with totally different mechanics just by a click of the thumb. Then, the existing game engines in the world, such as Unreal and Unity provides a very straight-forward and elegant way of managing the game into scenes and sprites and objects that the game itself, or “the director/manager” can load and instantiate. Also, all the objects are arranged in a highly hierarchical fashion so that loading the scene also loads the corresponding sprites within it. Finally, we were wondering the capability of the PIC32MX256F128B MCU and whether it can handle real-time video generation as well as operating under proper game mechanics. With all three aspects combined, we proposed to build a game console that arranges a composite video signal, a ~3000 sample/sec audio output and a polling input from a NES Controller. Running on the console lies the lightweight and powerful game engine that is derived from the modern game engines that elegantly loads and switches game scenes triggered by user-defined event.

---

Structure

Here is the overall structure of our gaming console.

---

NTSC Video

NTSC Signal

We are using NTSC (National Television System Committee) standard for generating black and white analog signals that display image on closed circuit TV. As electrons sweep across TV screen, intensity at each point varies so that images are formed.

So to generate an NTSC video, we need to: 1) Generate sync signals that meets NTSC standard; 2) Produce pixel-wise luma information of the image; 3) Use DMA to transfer luma information from PIC32 to TV. To transfer the luma information as quickly as possible, we used the internal SPI hardware on the PIC32 to perform the luma burst. The DMA will write to the SPI data register and the SPI hardware will take control of the sending by itself. Following Professor Bruce Land's instructions on synthesizing NTSC video on PIC32, we used a resistive network to add up the luma information and the sync signal as the diagram shows.

NTSC Video from PIC32

Considering the limitation of PIC32 memory, we use 1/0 of a bit to represent black/white of a pixel. To produce a 256x200 pixel image, we need only 6400 bytes, or 1600 integers as a frame buffer.

NTSC displays video system uses a interlaced scanning technique, which updates the odd number lines and the even number lines alternatively for reducing flickering. It involves both vertical sync signals that divides up even/odd and horizontal sync signals that divides up scan lines. NTSC frame frequency is 30/sec, and dividing into even/old fields, the frequency becomes 60 fields/sec Each frame includes a total of 525 scan lines, with 252.5 for each field. Because the vertical resolution of our image is only 200 pixel, we are not distinguishing even/odd fields. So we are essentially generating images at the rate of 60 frame/sec.

---

Audio Generation

In order to generate audio output during gameplay, we adapted the method of audio playback using DMA block transfer of audio data onto the voltage reference pin controlled by a timer. Due to the nature that the VREF register only takes a short (2 bytes) as input and only four bits of the value is the voltage information, we had to truncate and fit the original audio data into 16 voltage level bins. Also, since the MCU only has 128Kb flash, we have to resample the audio file into a much lower sample rate so that the MCU can hold as many audio data as possible while still being capable of playing the audio with acceptable quality.

To rapidly produce the header files for audio data, we modified the Matlab script that was provided in ECE4760 Lab 2 for converting audio data into header file. In the modified script, we resampled the original file with sample rate of 44100 down to 3000. The script can be found below.

    // Matlab script for audio data compression
    clear all
    % WAV file produced by the very good ATT text-to-speech utility
    % http://www2.research.att.com/~ttsweb/tts/demo.php
    title = '270329__littlerobotsoundfactory__jingle-lose-00.wav';
    [s, Fs] = audioread(title);

    s = s(:,1);

    %OPTIONAL !! drop the sample rate to 8 kHz IF you can.
    % I find with numeric digits "six" is not clear
    % at 8 kHz
    s = resample(s,3000,Fs);

    s = s(1:10420);

    % now scale to 0-to-15 for the 4-bit DAC
    min_s = min(s);
    max_s = max(s);
    %s = s - min_s;
    %s = fix(s * (15.5/(max_s-min_s)));
    % and byte pack the codes
    for i = 1:1:length(s)-1
    packed(i) = fix(15.5*(s(i)-min_s)/(max_s-min_s)) + 32864;
    end

    %make a textfile with PIC XC32 source code in it.
    fname=strcat(title(1:length(title)-4),'.h');
    fid = fopen(fname,'w');
    %static constant means to store it in flash
    fprintf(fid,'#define BGM_SIZE %d\r', length(packed));
    fprintf(fid,'static const short bgm_audio_buffer[BGM_SIZE]={\r');
    for i=1:length(packed)-1
    fprintf(fid,' 0x%s,\r',dec2hex(packed(i)));
    end
    fprintf(fid,' 0x%s};\r',dec2hex(packed(end)));
    fclose(fid);
                

---

Controller

In our gaming console design, we adapted a Nintendo NES Controller to interface with the player. The NES Controller uses a serial interface to transmit a packet of data from the controller to the MCU. The reader set the latch pin high to begin transaction and as the clock pulses on the clock pin, the data pin will return the current status of a corresponding key. An unsigned char with 8 bits length will contain information for the eight keys on the controller.

---

Graphics

Since our project involves display on screen, we must take graphics into account. Coming to pixelated display, there are several points that we have to consider when implementing a graphic driver.

• How do we elegantly fit a line with slope onto pixelated display screen.
• How do we draw a curve.
• How do we fill a shape.

Looking into the two points above, we came up with our solution for the project through researching online and referring to the previously implemented graphic libraries for TFT screen.

From the book “Procedural Elements of Computer Graphics” written by David Rogers in 1985, we picked up this block of text to be extreme useful to solve the first concern.

... The process of determining which pixels provide the best approximation to the desired line is properly known as rasterization... Bresenham's algorithm is equally suited for use with CRT raster devices. The algorithm seeks to select the optimum raster locations that represent a straight line. The algorithm always increments by one unit in either x or y, depending on the slope of the line. The increment in the other variable, either zero or one, is determined by examining the distance between the actual line and the nearest grid locations, and this distance is call the error ...

The second concern was to generate a curve on pixel display. In order to achieve so, we looked up how Bezier curve is generated in the first place. On the wikipedia page, we were able to find valuable animated images that visualizes the generation of parametric curves.

By referring to the wikipedia page, we were able to incorporate the quadratic bezier curve function,

and come up with a way to display curves on the screen. The programming details will be explained later.

The third concern, however, is a problem that we yet have to solve. In our current project design, we filled a shape pixel-by-pixel. And by doing so rapidly, we can fill up a shape without being noticed by eyes.

Hardware Design

---

Attached below is the overall hardware schematic design diagram.

Looking at the schematics diagram above, one can see the wiring configuration of the gaming console. The VIDEO pin and SYNC pin is connected via a resistor ladder and combines into the composite video output. The NES Controller has five pins: CLK, LATCH, DATA, VCC and GND. Besides the voltage reference pins, the rest of the pins are wired to the IOPORTB pin 0-2 of the MCU. The audio output is connected to the CVREFOUT pin.

On the PIC32 MCU, a series of peripheral hardwares also need to be configured. In order to properly generate the horizontal sync pulses, we ran a timer and configured an output compare module that is dedicated with generating the interrupt. However, the luma burst doesn't start until the horizontal sync and the front porch, so we need to configure another output compare module that is dedicated to trigger the starting of DMA transfer. Within the ISR of the first output compare, we set the DMA to be trigger by the interrupt flag of the second output compare interrupt, and in the second output compare's ISR, we configure the DMA to be continuously triggered by the SPI transfer's ready signal.

The literature, Programming 32-bit Microcontrollers in C: Exploring the PIC32 by Lucio Di Jasio, has introduced another way of synthesizing NTSC video by chaining two DMA channels together so we can save an interrupt service routine. However, here we follow Professor Bruce Land's approach of using another output compare module. Surely an additional ISR will affect performance, but saving another DMA channel can be greatly helpful such as synthesizing audio output and handling other tasks.

Program Design

---

In order to run a full-blown game engine on the PIC32, a series of libraries and data structures has to be declared and put into use. Since every game runs on an update function and an init function, we designed our game engine to mainly focus the two aspects throughout the design.

---

ProtoThreads

Our game engine is completely isolated with the outside main function - every scene and sprite is defined inside within the game engine and attached to the game manager object. In the main function, the game manager is initialized and being called into update continuously. Since we also need to handle the polling of the controller, we adapted ProtoThreads, a lightweight C library that supports multithreading. One of the threads we use is the update thread, and inside the thread, only the update function of the game manager will be called. The other thread is the controller thread. Inside the controller thread the series of serial latch and clock pulses will be sent periodically and the MCU will read the data from the controller. Then the controller reading will be updated into a global variable, which is shared by the game engine.

---

Modularized Game Engine Structure

As we introduced above, our game engine is highly structured based on the key aspects of an ordinary game engine: init and update.

Game Manager

Game Manager is the one and only object that manages the game engine itself. It has a master init function which will configure all the required hardware settings of the MCU to enable NTSC video generation, audio generation and countdown timers. Then the game manager also contains an array of scene objects that it can manage. Normally the game manager will have a current scene pointer that attaches to the currently selected scene. The game manager has an update function which will be called from the thread directly, and inside the update function, the game manager runs the update function of the currently selected scene.

Scene

Scene is a data structure that resembles the notion of a "world" that has its own objects, display and mechanics. The scene object contains two function pointers that point to the init function and the update function of the scene. It also contains a list of sprites that the scene contains. The init function initializes the scene and all the scene's sprites and the update function will update the scene and all its children sprites.

Sprite

Sprite is really the notion of "object" in the game engine. It represent some object on the display that has a position and dimension. It also has a data array which stores bitmap data of itself. It contains two function pointers, and those point to the init and update function of the specific sprite.

Integrating Game Manager, Scene and Sprite

Although the three key data structures described above looks familiar, they handle different part of the game. For example, in a very basic world which displays a user interface panel on the screen showing the current time and a ball in the middle of the screen. When user presses the controller, the ball bounces. This can be done in the following integration of the three data structures: The Game Manager selects its current scene to the new scene we created. Inside the scene, at initialization the scene draws the UI panel on to the screen buffer and updates the current time to the panel. The scene has one sprite object attached - the ball object. The ball object has a defined position information and width/height information. However, the ball does not need a data array for bitmap - it can keep drawing a circle using video_drawCircle() in its own update function every cycle. Inside the update function of the ball, it checks the current keypress status variable that is constantly updated by the controller thread. If a keypress is detected, the ball will decrease its y coordinate (move up).

---

Graphics Implementation

The way we implemented helper functions to render graphic objects onto the display buffer is very eclectic: we looked up references and code snippets from other available graphic libraries and recraft those ideas to fit the current scenario. Moreover, we use an incremental design approach to tackle graphic problems one-by-one.

Painting Pixels

In order to access a pixel and write it high or low, we use the predefined video display buffer which contains 1600 integers where every bit of the integer represents a corresponding pixel on the screen. From the video library provided by Professor Bruce Land, we managed to obtain the easy way of quickly accessing a pixel in the buffer.

Drawing Lines

From the literature, we also found a very informative pseudo-code that helped with implementing the drawLine function.

    // the Sign function returns -1, 0, 1 as its argument is < 0, = 0 or >0

    //initialize variables
    x = x1
    y = y1
    ∆x = abs(x2-x1)
    ∆y = abs(y2-y1)
    s1 = Sign(x2-x1)
    s2 = Sign(y2-y1)

    interchange ∆x and ∆y, depending on the slope of the line
    if ∆y > ∆x then
    	Temp = ∆x
    	∆x = ∆y
    	∆y = Temp
    	Interchange = 1
    else
    	Interchange = 0
    end if

    //initialize the error term to compensate for a nonzero intercept
    e = 2 * ∆y - ∆x

    //main loop
    for i = 1 to ∆x
    	setpixel(x,y)
    	while (e > 0)
    		if Interchange = 1 then
    			x = x + s1
    		else
    			y = y + s2
    		end if
    		e = e - 2 * ∆x
    	end while
    	if Interchange = 1 then
    		y = y + s2
    	else
    		x = x + s1
    	end if
    	e = e + 2 * ∆y
    next i
                

From the pseudo-code, we understood that Bresenham’s algorithm basically judges when to increment x or y depending on which side has the larger slope. Also, it keeps incrementing the error by the slope to see whether the line has passed the midpoint of two pixels.

Drawing/Filling Rectangles

Drawing a rectangle is the same as drawing four lines that connects all the four corner points of the rectangle. Therefore, we simply wrote the drawRect function as drawing four lines.

Filling a rectangle is a tricky problem. Our solution was to continuously call drawLine and increment the y coordinate of the line by one every time until we fill the display buffer with all 1s. However, when we are dealing with larger rectangles, plotting every point at a time will severely cause visual artifacts (image blinking) and impair the game experience. We dodged the problem by not using large filled areas, but however there are fewer ways to compensate for the problem:

• Only update the region of movement of a large bitmap, so we can confine the region of visual artifact as small as possible.
• Rewrite the fillRect function to block write the display buffer instead of writing one pixel at a time.

Drawing/Filling Circles

Luckily, we managed to obtain the code of drawing and filling circles from the Arduino TFT library, and we modified the SPI write to the TFT display to the pixel write to the display buffer, and we successfully plotted circles on the screen.

Results of the design

---

In order to demonstrate the robustness and functionalities of our game engine design, we have implemented a sample game on our gaming console, Rope Jumper. This game resembles one of the minigames in Nintendo's famous game: WarioWare. WarioWare is a collection of minigames and each of the games have different mechanics and rules. To support different minigames in one game, we developed our game engine to support multiple scenes which can run completely different game mechanics and load different sprites.

Hopefully our design can prove the possibility for our game engine to achieve similar quality as production-ready games like WarioWare.

Gameplays from WarioWare

Gameplays from Rope Jumper

Rope Jumper: The Design

The objective of Rope Jumper is to keep the character jumping when the rope is swiveling past the ground. The player controls the character's jump by pressing A on the controller. The character's jump speed and interval is limited so the player cannot blindly spam the controller to hover above the ground. The speed of the rope will also change when player enters a certain level.

Rope Jumper: Game Structure

The Game Manager contains multiple scenes in its list. The first scene is the main menu that the game goes into. Inside the main menu scene there are three sprites, each representing a corresponding button. User can use the arrow keys on the controller to switch from different buttons. The second scene is a countdown scene that elegantly counts three second and switch to the game scene. The third scene is the main game scene which contains four sprites: rope, character, man who grips the rope and a UI tooltip sprite. Finally, there is a pause menu scene which renders a pause menu on top of the current game and contains three buttons in it.

Rope Jumper: Game Scene Implementation Details

The game scene consists of four main sprites: rope, character, man and tooltip. Before loading the four sprites, the scene also loads its own configuration to draw a large box around the display to contain the game and some help text. Then the scene actively polls for controller input inside its update so that if user presses start key during game, the scene will cause game manager to switch to the pause menu scene. Also, the scene continuously checks for game over condition that can be computed by checking the rope and character sprite's location.

Rope

The rope sprite is a quadratic Bezier curve that contains three way points. When we defined the rope, we fixed the starting and ending point of the curve and vary the middle point of the curve up and down so we can simulate the elastic rope. We also used a sine table to map the y position of the middle point so we can obtain a smooth and realistic rope animation. In order to compensate for gravity, we increased the falling speed of the rope by 1.5x compared to the rising speed by changing the number samples of the sine table we skip.

Character

The character sprite is a round rectangle with two text on it. All the displayed objects are drawn at init of the sprite and constantly wiped and redrawn during update process. The sprite also listens to the controller and adds a upward thrust to the object whenever a keypress on key A is detected. The thrust is a vector force and the character's velocity and position will be computed from the force. The force will soon drop back due to gravity and so does the character.

Man

The man sprite draws the two men holding the rope on the sides. The two men are pixel arrays that is generated from Matlab by reading an existing image file found on Google. The image array contains multiple frames so that the men's arm can move up and down with the rope's position. The update function of the sprite compares the current rope's rotation position with its fram number so that if the rotation exceeds the given threshold value it increments its frame number by 1.

Our first approach to the man figure was to constantly remove and redraw the figure when the rope moves. However, since there are a large chunk of array data being redrawn pixel by pixel, we easily noticed visual artifacts and very blinky image. Therefore, in order to reduce visual artifact as much as possible, we altered the code to only update the middle portion of the figure that contains the man's moving arm so that the top and bottom part of the figure does not need to be updated at all. By doing so we were able to minimize visual artifact and optimize game experience.

Tooltip

The tooltip sprite is used to display floating text tooltips that will appear whenever there is a speed up, difficulty change or the player loses one life. It is implemented as a state machine so that whenever a new tooltip request comes in, it resets the floating text's position and gradually shift the position of the text overtime. It also contains an internal counter to remove the text after a certain time interval expires.

Conclusion

---

In the end, our project met our expectation and functioned exactly as we proposed. We were able to run the game engine on the PIC32 MCU without any glitches and minimal artifacts. Although we initially proposed to synthesize colored video, but we realized that it would be too harsh on the timing of the color bursts. So we ended up using the black-and-white video display.

By using an hardcoded interrupt service routine to handle analog signal bursts, we successfully conform to the NTSC standard, especially the timing requirements. Actually successful Video generation under the NTSC standard is an essential foundation for all the work we have done in the project.

Our game engine design is a safe tool for developing recreational applications. The embedded Rope Jumper game is simple but entertaining game, and its content is healthy and appropriate for all user groups, with the game itself being a beneficial measure for attention and reaction speed training. So the project obviously matches with public welfare and interest.

Initially constructed a game engine, the infrastructure of our work makes it truly a versatile toolbox that can be easily expanded to a variety of graphical applications way beyond the scope of s game. So our project demonstrates an improvement in understanding of technology. Since this project in mainly digital, the undesirable effect it can produce to the environment is completely negligible, and it is very difficult to be used as any means for malicious purpose.

The implementation of our design does not involve any dangerous process and can be completely accomplished in a teaching lab environment, so we are confident of our technical competence in all the work we have done.

We adhere strictly to technical ethical code throughout all stages of the project, and we would definitely maintain the ethical code in all future work.

This project does not involve any legal restrictions besides what have been addressed in previous Intellectual property considerations sections.

Our project is more than welcome to suggestions for improvements and criticism on mistake if there is any. And we have credited all sources we referenced to and acknowledged all contributors of the project.

Appendix A

---

The group approves this report for inclusion on the course website.

The group approves the video for inclusion on the course youtube channel.

Appendix: Parts List

---

• Microstick II with PIC32MX356F128B - Provided in Lab - $15
• Perf board - Provided in Lab - $2.50
• RCA Connector - Amazon - $5.00
• 3.5mm Auxiliary Cable - Target - $3.49
• RCA Cable - Amazon - $7.99
• 3.5mm Auxiliary Connector - Provided in Lab - $0.10
• Acrylic Casing - Cornell RPL - Salvaged from scrap

Appendix: References

---

• How to generate video signals in software using PIC
• ECE4750 NTSC video PIC32
• Analog Video 101
• Decoding NTSC Video - MIT Lecture Slides
• Adafruit TFT Library
• NES Controller for Arduino
• Nintendo NES and SNES joystick controller pinout
• Procedural Elements of Computer Graphics
• Writing a Game Engine from Scratch

Appendix: Schematics

---

Appendix: Source Code

---

The source code of our project can be downloaded here.

Source Code

Appendix: Work Distribution

---

Dongze Yue

dy85@cornell.edu

Designed and implemented the graphics driver as well as the hardware of the gaming console. Wrote the helper functions for the NTSC display. Designed Rope Jumper.

Yixiao Zhang

yz624@cornell.edu

Drafted and coded the data structures of the game engine, including scenes and sprites.

Yuqing Sun

ys499@cornell.edu

Implemented part of the game engine about global messaging.

Appendix: API Documentation

---

In this section, all the major components of the game engine will be explained.

---

---