The Webcam Mouse

Software Design and Implementation

Matlab Code

As briefly discussed in our high-level design section, our Matlab code takes care of the pre- and post-processing of the mouse control. To communicate with the webcam, we utilize the Image Hardware Acquisition Toolbox (imaqhwinfo) and create a videoinput object. This allows us to trigger a function call at a fixed interval faster than the frame rate ensuring we never miss a new frame to be analyzed. We referenced previous work by MathWorks’ FileExchange for red object detection and Jerome Lecoq for real-time analysis (referenced in the Appendix) in developing the image processing and real time tracking code explained below.

Object Tracking and Centroid Extraction

In the call to the triggered MouseControl() function, we complete all of our image processing and then communicate with the microcontroller. After flipping the image so it acts like a mirror, we begin by extracting all data that isn’t red with the following built-in Matlab function:

   data = flipdim(getsnapshot(vid), 2)
   diff_im  imsubtract(data(:, :, 1), rgb2gray(data))

Data holds the snapshot (the current frame) for all x, y, and z data. Here, we keep all x and y data and only extract the first z value for each which corresponds to the red information in the red/green/blue (RGB) format. It then converts this to grayscale (binary) data, essentially mapping the specific color to a number. After this step, we then apply a median filter to the data in a 5 x 5 pixel area which removes noise. After experimenting with different areas, we found that 5 x 5 was a sweet spot for this specific webcam.

With this accomplished, we can begin finding the objects within the frame. This involves converting the image to a binary image by setting a threshold for 0’s and 1’s:

   diff_im = im2bw(diff_im, 0.20)
   diff_im = bwareaopen(diff_im, 16)
   bw = bwlabel(diff_im, 8)  

This threshold (0.20) was also heavily experimented with. If this is too low, parts of the frame with very little red content are grouped together. If it’s too high, it aggressively removes part of the image which we want to keep. The bwareaopen() command analyzes the binary data and finds connected components of red, i.e. distinguishes significant red objects. This is why the threshold of 0.20 is so important as it allows us to find the correct amount of discretion when analyzing the data. Finally, we label the connected components with the bwlabel command.

At the end of all of this image processing, we have found all significant red objects on the screen. The list in bw ranks them by their size and picks between objects of equal size by its proximity to the top-left of the screen. Using the regionprops function, we can extract the centroid of each object and send the corresponding x and y positions to the microcontroller.

   stats = regionprops(bw, 'BoundingBox', 'Centroid')

For debugging and development, we displayed each frame and its centroid with bounding box in order to find the correct parameters for our image processing. In our final code, we do not display the video and only extract the centroid to avoid the bounding box calculation.

Serial Communication

With the absolute position information extracted, the function then uses the serial object below to send the position packet to the microcontroller:

   s = serial('COM8', 'Parity', 'none', 'FlowControl', 'none', 'BaudRate', 9600)
   fprintf(s, '%d %d\n\r', [round(bc(1)) round(bc(2))])  

The serial object created is the most crucial connection of our project as it allows the microcontroller and Matlab to communicate. The settings above allow us to use the fscanf() and fprintf() functions to send packets and receive them with a 9600 baud rate. We experimented with other rates, but 9600 proved to be the most reliable. The arguments bc(1) and bc(2) are the x- and y-positions of the tracked object and are the entirety of the packet we send to the microcontroller. This was a design choice because we wanted the microcontroller to calculate all of the mouse controls. After sending the position packet, Matlab waits on processing done by the microcontroller and return data with a fscanf() function call. We chose to have the function timeout after 2 seconds if it hasn’t received a packet which made debugging easier. After it has received the packet from the microcontroller, it unpacks it with the following command:

   in = fscanf(s,'%d %d %d %d %d %d')

From left to right, the packet above holds the following information: x-position, y-position, left click, right click, on-off control, and the scroll command. The on-off value comes from the switch on board the breadboard. If it is low, then Matlab can’t control the mouse. This lets the user keep the mouse stationary while typing or simply lets them turn it off for extended periods of time.

Mouse Control

We used the fact that Matlab can natively run Java to create a Robot object which directly manipulates the computer’s mouse. If the scroll command is positive 1, Matlab knows that to scroll one scroll unit up on the screen. If it is negative one, it scrolls one scroll unit down. If it is zero, the user has either not selected the scroll option or has not moved more than 10 pixels up or down. If the mouse is on, the code will send clicks and scrolls to the Java Robot object along with an absolute position which controls it in real time. After timing this operation with tic and toc, we found that these commands execute very quickly as we expected. The code finally updates the state and exits the function call. It does nothing until the next time-triggered event.

Testing & Debugging

Our first bug involved Matlab crashing due to its memory allocation filling up. Without the timing object to trigger the function call, all of the frames that are analyzed sit in the computer’s RAM and fill it up very quickly. The function call and timer object prompt Matlab to garbage collect these frames, freeing up memory usage. When scanning in the packet from the microcontroller, we spent over two full work days trying to understand why it wouldn’t work only to realize that we needed the \n\r terminator. Without this terminator, the microcontroller’s fscanf() function will never detect an end on the input stream and as a result will wait indefinitely. Another interesting Matlab bug involved the Matlab’s packets to the microcontroller staying in the buffer even after they have been read by the microcontroller. More specifically, the x and y integers need to be read by Matlab before we can extract the microcontroller’s response. This is a confirmed bug that other groups have encountered this year which was solved by simply making two fscanf calls on the data.

Microcontroller Code

TRT Kernel

The microcontroller is running the Tiny Real Time (TRT) kernel which schedules two real-time tasks: one for serial communication with Matlab and a second which listens for button inputs from the user. Although the two pieces of information are acquired separately and must be protected via semaphore, the frequency of their execution is on the order of single milliseconds which allows seamless combination of the information in the Matlab communication task. In this communication task, the microcontroller calculates the scaled position in number of x and y pixels to control the mouse. It then accesses the user input variables to detect a right, left, or scroll request from the user along with an on-off switch selection.

Both tasks execute in a few milliseconds and have a small amount of local variables, which minimizes the overhead of the context switch. Although we could achieve the same functionality by writing solely C code and using ISRs, the TRT tasks both run quickly enough to be executed by the time the next frame is received (33ms). We also found TRT easier to use when developing and debugging the project.

Initialization

Before running the two TRT tasks, the microcontroller must setup its ports and serial communication. We utilized Port B as an input port with pull-up resistors. These allow the pins to return quickly to a value of logic high after discharging to zero volts upon a button press. We utilize trtUart.c, trtUart.h and trtSettings.h provided by Bruce Land (written by Dan Hennriksson and Anton Cervin) for our previous labs to communicate with the PC. We then create and start the TRT tasks with an enough stack space to call fprintf() and fscanf(). We ensured that the pccomm() task had enough space to do floating-point calculations as well.

PC Communication and Screen Position Calculation

After initialization, our first task, pccomm() executes every millisecond according to our TRT release and deadline times. The pccomm task first reads in the x and y position data that Matlab has sent over UART which is setup in initialization as follows:

   fscanf(stdin, " %d %d", &x, &y)

Because we have hardcoded the screen resolution of the lab computers, the microcontroller code can use the x- and y-pixel coordinates extracted from the packet to calculate the new absolute mouse position. The accuracy of our device is very high as we have 640 points of precision in the x direction and 480 points in the y direction from the webcam. This means that for a typical monitor display, the mouse can be positioned with 2 pixel precision using floating point arithmetic. This is more than enough to accurately point anywhere on the screen. Before the microcontroller can send its return packet to Matlab, it must encapsulate any user input received.

Because we use a semaphore to protect the user input, the pccomm() task must wait at most a millisecond (plus context switching) to access these data. The buttons used to not need debouncing, so we can directly access the value of PINB to check the bit value of the on-off switch, left click, right click, or scroll. For example, if the user has clicked the left-click button, Pin B’s bits will equal 00000010 indicating that the Pin B.1 is at zero voltage. The essentially sees if the value of Pin B.X is low and if so, either a button has been pushed or the on-off switch it connected to ground. The microcontroller then completes the scroll calculation before sending its packet to Matlab.

Scrolling Calculation

In order to implement touchscreen type scrolling, the microcontroller tracks the most recent y-pixel position. If the change between function calls is at least 10 pixels and the scroll button is held, the microcontroller stores a scroll up (1) or scroll down (-1) value to send to Matlab. This way, the user must move their hand while holding the scroll button to scroll up or down. If they stop moving, it will stop scrolling just like any touchscreen. After this calculation, the microcontroller combines all of this information and sends a packet back to Matlab with the data needed to entirely control the mouse as shown in the format below:

Button Communication

The buttoncomm() task is very short and thus very fast. It has a single state variable which tracks which button has been pressed. We decided to only allow one button be recognized with priority given to left click, then scrolling, then right click. This was after careful consideration of the possible use cases. Very rarely with a traditional mouse does one left and right-click at the same time, or click and scroll at the same time. This made our task faster and the state machine very simple. From state 0 (idle), the initial state, we first check for left clicks, then scrolling clicks, and lastly right clicks in a similar fashion as the left click. We invert Pin B because the buttons are active low. After observing the voltage characteristics of the buttons with an oscilloscope, we decided not to debounce them because they are very robust. For each of the three clicks, we transition to a unique state representing that it had been clicked. In each of these three states, we check to see if the button is no longer pressed. If so, we return back to state 0. This state machine is running constantly and is only interrupted upon context switches to pccomm().