EEG Error Correction Interface

Rebecca Bell (rb683), Emma Kaufman (eck66), Chloe Kuo (cck78)

Introduction

Our project was an EEG-controlled brain computer interface that allowed a user to correct errors in machine behavior. The project was modeled as a trial-based “game.” In each trial, a solid colored green block or dotted yellow block was placed on a conveyor belt. One end of the conveyor belt had a solid colored green box and the other end had a dotted yellow box. It was the user’s goal to get the block to move to the side of its corresponding patterned/colored box. In each trial of the game, the conveyor belt began to move in a randomly chosen direction. If the random direction was correct, the user needed to stay calm in order to allow the block to travel fully to the correct side and fall into its collection bucket. If the random direction was incorrect, the user needed to attempt to correct the direction by spiking their anxiety as determined by data from an EEG headband.

Design

The initial conception of our project can be attributed to our general interest in EEG-controlled robotics. Upon research in this topic, we came across work done at MIT’s Computer Science and Artificial Intelligence Laboratory regarding robot control using brainwaves and hand gestures. Our intent was to take inspiration from this research and demonstrate that we could create a consumer-friendly, inexpensive, EEG-controlled error correction interface with a timeline of five weeks.

To make this product consumer-friendly, we used the Muse 2 as our EEG headband. The Muse 2 has seven EEG electrodes, with the four main electrodes located on the forehead and behind the ears, and three additional reference sensors. The Muse 2 headband was donated to us by the Salk Institute in San Diego. First, data was sent over Bluetooth from the Muse headband to a phone. Then, the data was transmitted via OSC (Open Sound Control) to a computer. This data path was necessary due to the budget constraints and additional hardware necessary to receive Bluetooth data directly on a laptop. Then, the data was sent over serial to the PIC32, and an FFT was performed on the data (Figure 2). The conveyor belt moved in a random direction to start the game. Once the FFT was performed, we checked for anxiety spikes, and if the user was anxious, the conveyor belt switched directions. Figure 1 shows a block diagram of the system.

Figure 1
Figure 1: Block Diagram
Figure 2
Figure 2: High-Level Muse-to-PIC32 Connection Diagram

The OSC stream protocol worked by transmitting an OSC packet to an OSC server through either TCP (Transmission Control Protocol) or UDP (User Datagram Protocol), which are both IP based protocols. In this lab, we used UDP because if small sections of the data were not transmitted correctly, we could throw out the data and still obtain the correct result. This also simplified the code because UDP did not require the user to recognize when the data was received. UDP is specified by the Internet Engineering Task Force (IETF) via STD 6: User Datagram Protocol. This standard explains how application programs can send messages to other programs via a transaction oriented protocol.

Hardware

Figure 3 shows a wiring diagram.

Figure 3
Figure 3: Hardware Wiring Diagram
Figure 4
Figure 4: Motor Control Circuit Diagram
Figure 5
Figure 5: Motor Control Circuit

Our hardware design was broken up into 3 parts: serial hardware, motor control hardware, and game play hardware.

Serial:

For serial, we used the serial-USB cable. The UART receive pin (U2RX) was wired to RA1 on the board and the UART transmission pin (U2TX) was wired to RB10 on the board. Additionally, the serial-USB cable was grounded to the MCU ground, and the USB Vcc was not wired to anything.

Motor Control Hardware:

For the motor, we used a continuous rotation parallax servo. Since DC motors can cause inductive spikes that can blow out the transistors on the board, we used capacitors to reduce any chance of inductive spikes. We connected a 0.1 μF capacitor across the external 5V power supply and ground, and then a 1 nF capacitor across ground and the signal wire of the servo. The signal wire was connected to OC3, which is mapped to RPB9 (pin 18) on the board.

Game Play Hardware:

The game play hardware consisted of 3 buttons and 3 LEDs. The 3 buttons corresponded to start game (grey button), correct bin (blue button), and incorrect bin (red button). The start game button was wired to RA2 on the board and ground, the correct bin button was wired to RA3 on the board and ground, and the incorrect bin button was wired to RA4 on the board and ground. The LEDs corresponded to whether or not the player got the block in the correct bin as well as which bin was correct.

Mechanical Design

We had several design considerations before mechanical construction of our conveyor belt.

Because eye movement is observed on EEG measurements, we ensured that the horizontal length of the conveyor belt was relatively short (~12 inches) so that when the user’s eyes followed the traveling blocks, the EEG data would remain unaffected. Additionally, we picked a continuous rotational servo with low torque (38 oz-in) due to the fact that we did not need the belt to move at a high speed. Knowing that our project required lots of iterative testing, we chose to use 2020 aluminum T slot tracks as the material for the frame for our conveyor belt for durability and universal hardware mounting. Lastly, we used multiple rubber bands as the belt to keep our budget minimized and to allow for easy replacement during testing.

We made a 3D model of the conveyor belt that we intended to construct (Figure 6). Using this model and our design considerations, we assembled our conveyor belt (Figure 7).

Figure 6
Figure 6: Model of Conveyor Belt Concept
Figure 7
Figure 7: Final Constructed Conveyor Belt

Software

Our software for the PIC was broken down into 2 functions, FFT function and change_direction function, and 4 threads, the timer thread, motor thread, serial thread, and FFT thread, as well as an ISR. We also had a python script that sent data from the headband to the PIC over serial. On a high level, data was received in the serial thread from the python script running on a computer. In the serial thread, raw EEG data was placed into arrays and pre-FFTed data from the Muse itself was saved into single variables. Once an array of raw sensor data was full, the array was run through an FFT in order to update a running average of baseline EEG activity. In the motor thread, the pre-FFTed data was compared to this running average and, if the pre-FFTed data was much greater than the baseline, the motor thread determined that an anxiety spike was occuring. This changed the direction of the conveyor belt if a trial of a game was actively occurring. The details of each part of our software are explained below.

FFT Function:

For the FFT function, we used Bruce’s FFT algorithm from the FFT spectrum analyzer with TFT-LCD output. This function was optimized for use with fixed point numbers. We performed FFTs on arrays of size 256, so we defined N_WAVE as 256 and LOG2_N_WAVE as 8. This function returned an average of the magnitudes of all FFT bins in order to update our running average of baseline EEG activity which was used in other threads to determine anxiety spikes. To obtain this average, we added the power magnitude of all frequency bins together and divided by 128. The rest of this algorithm was consistent with Bruce’s implementation.

Change Direction Function:

This function set the duration of the PWM for the servo motor. The PWM signal was updated using this duration in the ISR. We changed the pulse width using a variable we called duration, and we used a switch statement using a variable called direction, where 0 corresponds to stop, 1 corresponds to counterclockwise rotation, and 2 corresponds to clockwise rotation. When direction was 0, the duration was set to 12000. When direction is 1, the duration was 12100, and when the direction was 2, the duration was set to 11900. These numbers were determined based on trial and error tuning of our specific servo motor.

Timer Thread:

The timer thread printed the time since reset at the top of the TFT. We kept track of time in order to determine when we believed we had enough baseline EEG data to begin playing the game (we typically had enough data about a minute after transmission began).

Motor Thread:

The motor thread controlled the motor and game play. First, we read the values from port A, bits 2, 3, and 4, which corresponded to the 3 buttons. The values read-in on these pins determined whether we were starting a trial, completing a trial when the correct outcome had occurred, or completing a trial when the incorrect outcome had occurred. We also saved the current state of the game and the previous state of the game, in order to determine whether we should complete actions that were only supposed to be carried out at the beginning of each round. When the start game button was low (meaning the button was pressed), we set valid_round high and score_incremented low. Score_incremented ensured that the score was only incremented one time per round even if the buttons were pressed more than once. We also printed “Start Game” so that the user would know that the game had started. Then, if either button_right or button_wrong went low at any point, we set valid_round equal to 0 to signal that the game had ended. We also increment the number_right or number_wrong based on which button was pressed and printed the updated scores. If button_right was pressed, we lit up an LED corresponding to the color of the collection bucket based on the direction that the conveyor belt was moving when the game completed. If button_wrong was pressed, we lit up a red LED in the middle of the conveyor belt to signal that the block landed in the incorrect bin. The next section of the thread checked if there was a valid round occurring. If a valid round was occuring, and it was the first cycle in this valid round (as determined by comparing valid_round to its previous value), we started the conveyor belt in a random direction from a randomly generated seed based on the current time. We compared our gamma value (from the pre-FFT data) to our running average from the FFTed raw data, as completed in the FFT thread. If the gamma value was at least 0.2 higher than the running average for 4 cycles, then we considered this an “anxiety event”. After an anxiety event was detected, it took 3 cycles of gamma data below the threshold for the debouncer to consider the EEG activity “not an anxiety event”. This ensured that the user had actually calmed down, instead of the PIC just receiving a transmission error. If there was not a valid game, meaning that the “correct” or “incorrect” end game buttons had been pressed, we set direction of the conveyor belt equal to zero and called change_direction(0) so the conveyor belt stopped. We also updated change, previous_change and the debounce counters to be ready for the next round.

Serial Thread:

The serial thread saved both the raw data and the pre-FFT data sent from the headband via the python script. The python script sent data every time it received an exclamation point. We used the thread PT_DMA_PutSerialBuffer, because this function used DMA for serial communication, which was fast. To use this thread, we spawned a child thread, which blocked the serial thread from executing while PT_DMA_PutSerialBuffer was executing. Next, we used PT_GetMachineBuffer to get the data from the python script. We used PT_GetMachineBuffer so that the data could be sent without a user inputting any data. We used a termination time of 5 seconds and a termination character count of 5 characters. If we received data, we scanned PT_term_buffer and saved the data value into cmd and its tag into a variable called value2. The tag indicated whether we were receiving raw data or pre-FFTed data, as well as which sensor the data was being received from in the case of raw data. If we do not receive anything, we set cmd to 0. Then, we divided cmd by 1000 and checked to see if it was less than 0.1 or greater than 0.999 to ensure that the values were in the correct range. If they were not in the correct range, meaning we received corrupted data, we set the value of cmd equal to its previous value. Though this caused us to lose some accuracy, we took this approach because EEG readings do not tend to change very quickly, and the timing of our system was slightly more important than making sure every data point was exactly correct. Then, we had a switch statement based on the value2 tag to save the data into the correct variables. Case ‘1’ was our gamma values (pre-FFTed values from the Muse headband), so we saved cmd into the variable ‘gam’. We ultimately decided to only save raw data from the two Muse sensors located on the forehead because these were the 2 main sensors associated with collecting data for gamma waves. Each of these sensors had two arrays. While one array was being filled, there was an FFT being performed on the other array. To ensure that we were filling the correct array, we use the variable ‘lock’ to indicate which array should be filled, and changed this value once the FFT had been performed. We also had counters for each sensor so that we ensured that we were filling in the correct index and that the array was full before we performed an FFT on it. Case ‘7’ was our AF7 sensor, so we saved the value into the respective array based on lock. If lock was equal to 1, this meant that there was an FFT being performed on F1af7 and F1af8, so we put cmd multiplied by the window into F2af7 at ctr3, which is the counter for the AF7 sensor. If lock was equal to 0, there was an FFT being performed on F2af7 and F2af8, so we put cmd multiplied by the window (initialized in main) into F1af7 at cr3. Lastly, we incremented ctr3 if it was less than 255, since the size of F1af7 andF2af7 was 256. Case ‘8’ was our AF8 sensor, so we saved the value into the respective array based on the lock. If lock was equal to 1, this meant that there was an FFT being performed on F1af7 and F1af8, so we put cmd multiplied by the window (initialized in main) into F2af8 at ctr4, which was the counter for the AF8 sensor. If lock was equal to 0, there was an FFT being performed on F2af7 and F2af8, so we put cmd multiplied by the window into F1af8 at ctr4. Lastly, we incremented ctr4 if it was less than 255, since the size of F1af8 and F2af8 was 256.

FFT Thread:

The FFT thread first checked if either set of raw data arrays contained 256 data points. If a set of arrays was full, it would call the FFT function on the filled arrays while allowing the other, unfilled arrays to be used in the serial thread to collect data. Once the FFT function was complete, all FFT bins were averaged together, and this average was used to update the running average baseline for anxiety thresholding. The running average was then printed at the bottom of the TFT display.

ISR:

The ISR changed the motor direction by adjusting the length of the PWM pulse. The desired motor direction was set in the Motor Thread.

Main:

In main, we initialized all of the threads and scheduled them based on round robin scheduling. We also initialized the motor and set the input pins. Additionally, we created the sine lookup table and the window array to help scale the data values to the correct range.

Python Script:

The python script first set up a server to receive the OSC stream from a phone. The phone was connected over bluetooth to the Muse Headband to receive both the raw data and the pre-FFT data. Once this server was established, methods were added to the server in order to receive data from specific transmission channels (i.e. in order to receive specifically from the raw sensor data channel or specifically from the gamma frequency data channel). These methods added the received data to predetermined queues, either the gamma queue or a raw sensor queue, given that the data actually contained a number as expected (instead of just garbage). Once the methods were added to the server, serial communication was established with the PIC32. We then used a multithreaded python script in order to receive OSC data and add it to queues, while simultaneously preprocessing and truncating some significant figures and sending this data with an appropriate tag over serial to the PIC32. We added character tags to all of our serial transmissions in case we could not guarantee the order of data transmission. For our analysis, it was important to identify whether we were sending pre-FFTed or raw data, as well as which sensor the raw data originated from.

Testing and Results

Figure 8
Figure 8: Conveyor Belt Undergoing a Trial
Figure 9
Figure 9: Display on TFT

We first tested our ability to communicate over serial. Our original implementation of serial caused us to get a large amount of unexpected and inaccurate data on the PIC32. To remedy this, we decided to only send data when the PIC32 was ready to receive, as indicated by it sending an exclamation point over serial to the computer. We then noticed that our serial communication was hanging at random times. To fix this, we went into the protothread header files and cleared all serial errors each time a read or write was performed. This solved many of the hanging problems, yet we still could not reliably establish communication for more than about five minutes. We then began to flush the serial buffer in the python script, as well as lock our python thread whenever we performed a read/write, which finally fully fixed the problem and allowed us to communicate for hours at a time.

We then noticed that, despite having a baud rate of 115200, we were becoming massively delayed in our serial transmission and creating queues of steadily increasing size so that any data sent was actually delayed from real-time by thousands of data points. Because our project relied so heavily on quick responses to fairly noisy data, we decided to average together three data points in the server methods before adding data to a queue. This allowed us to keep up with transmission and keep our queue size between 1 and 0 at all times.

Once each thread functioned as desired and all data was properly transmitted, we began to analyze speed and performance. Given the size of our FFT, we ultimately ended up updating our running average about every 3 seconds. This was an appropriate rate given that the average should not rapidly change from millisecond to millisecond, but instead should hold as a fairly steady baseline once the program had been running long enough to collect several data points. We then analyzed our speed in receiving the pre-FFTed gamma data. We updated this data slightly faster, about once every half second, allowing for many data points to be received during any given trial of the game. Because of this data rate, we adjusted our conveyor belt to move fairly slowly. This allowed for appropriate debouncing of anxiety spikes and ample time for users to respond to the conveyor belt motion in order to correct errors.

We then focused on thresholding. In order to achieve clean results, we added a debouncing component to detecting anxiety spikes. For activity to be called an anxiety spike, an anxiety state needed to be held for four data points. Similarly, in order for a spike to be marked as complete, activity would need to be below the threshold for more than 2 data points. The threshold was determined by trial and error. However, by using the running average of all FFTed data as the baseline for this threshold, the game was relatively transferable to different players. If a player had a higher baseline, they would need to cause a higher spike in order for an anxiety event to occur. After tuning the threshold, we found that our accuracy in getting blocks into correct bins was over 90% for data taken from one player over about 60 trials. We are interested in further investigating this accuracy for different players, as well as taking more structured data for the original player. A video of our early testing can be seen below:

To test motor control, we programmed a PWM signal to output on RB9 (using a timer-based ISR), and used an oscilloscope to see the duration of the pulses and how far apart the pulses were. For continuous rotation servos, the length of a pulse determines the stop, clockwise, and counterclockwise rotation. A decrease in the pulse width causes the servo to rotate faster in the clockwise direction, and an increase in the pulse width causes faster rotation in a counterclockwise direction. We tuned our servo to rotate at a desired slow speed by changing the duration of pulses in our code.

Figure 10
Figure 10: PWM Servo Control (Source: here)

Conclusion

Through the completion of this project, we were able to design, build, and demonstrate machine error corrections in real time using EEG signals. Our design generally met our expectations. We were highly satisfied with our data analysis abilities and the cleanness with which we could ultimately label anxiety spikes. We were slightly disappointed in the mechanical performance of our conveyor belt, and would have liked to rebuild it given more time. In terms of future improvements, we would like to add a learning component to our game. If a user marks a trial as wrong, we would like to have functionality which can adjust our thresholding to reduce the likelihood of an incorrect round in the future. We would also like to further investigate how EEG data can be used in human-machine interactions, maybe by feeding EEG data into neural networks to see whether we could discriminate thoughts such as “left” or “right.” However, given the time and budget constraints in this project, we believe we created a robust project which successfully met nearly all of our expectations.

Our final product met the general standards for this course as it had more than 50% of its data processing on the PIC32 (raw sensor FFT, thresholding, and spike-debouncing was done on the PIC32 whereas only specifically gamma-band FFTs were done elsewhere). All game-play and motor logic was also implemented on the PIC32. The design was under-budget (see Appendix C), and our group spent a large amount of time in-lab working to make sure each individual component of the hardware and software performed robustly.

This project used the Muse 2 EEG headband, and used the Mind Monitor app in order to receive the raw sensor data and pre-FFTed gamma data over OSC. The python written for this report was based on the open-source project, “Telekinesis Car,” created by Devin Delfino. The link to the Github repository containing the code for this project is linked in the reference section of the report.

Our design was very safe. The only component attached to a physical person was the EEG headband, which is commercially available and powered by a charged battery. It also sends data over bluetooth, so there are no extraneous wires attached to any person. Additionally, The conveyor belt was built from sturdy material, and the servo was strongly attached to the conveyor belt, so no parts were at risk of coming loose. Our design also had accessibility considerations. We chose yellow and green as our block colors, and dotted our yellow blocks/bin, in order to be accessible to those who are color blind.

We therefore believe we are in compliance with the IEEE code of ethics. Our project did not endanger the environment or any person who actively used the game. We do not have any conflicts of interest, did not fabricate any performance data, and did not participate in bribery of any kind. We have fully credited contributors to each part of this project, and we believe that our project beneficially highlights the growing potential and promise of EEG-based human-computer interfaces.

Figure 11
Figure 11: Final Project Setup

Appendix

Appendix A: Approval for Inclusion on Course Website

The group approves this report for inclusion on the course website.

The group approves the video for inclusion on the course website.

Appendix B: Budget

Item Vendor Part# Cost
Microstick II 1.00
Big Board 10.00
White Board 6.00
Large Solder Board 2.50
Power Supply (x2) 10.00
PIC32MX250F128B 5.00
I/O Expander 5.00
TFT LCD 10.00
Header/Socket Plug 3.70
Jumper Cables 0.30
ReliaBot 2PCs 8mm x 500mm Case Hardened Chrome Plated Linear Motion Rod Shaft Guide - Metric h8 Tolerance ReliaBot (from Amazon) IT-RD048 11.39
Sydien 8mm Inner Bore Ball Mounted Pillow Block Flange Micro Vertical Bearings 4 Pieces (8mm) Sydien (from Amazon) 43237-2 8.99
Antrader 12pcs T Slot L-Shape 2020 Aluminum Profile Interior Corner Connector Black Joint Bracket for 2020 Alu-Profile 3D Printer with Screws Antrader (from Amazon) AZ18101105 10.99
20mm X 20mm T-Slotted - Four Open T-Slots (Cut to 18 inches) 80/20 Inc. 2020 8.46
20mm X 20mm T-Slotted - Four Open T-Slots (Cut to 6 inches) 80/20 Inc. 2020 13.86
Blocks 0.30
Paper 0.20
Rubber Bands 2.00
Tape 5.00
LEDs 0.12
Buttons 1.20
Total $113.81

Appendix C: Task Breakdown

Rebecca:

  • Established serial communication between the computer and the PIC32
    • Worked on python script and serial thread
  • Integrated FFT function and wrote FFT thread to analyze data received from serial thread
    • Thresholded pre-FFTed data with running average of all raw data
  • Set-up motor thread for game play with trials and changing conveyor belt directions
  • Built game-play hardware and motor control circuits
  • Report

Emma:

  • Established serial communication between the computer and the PIC32
    • Worked on python script and serial thread
  • Integrated FFT function and wrote FFT thread to analyze data received from serial thread
    • Thresholded pre-FFTed data with running average of all raw data
  • Set-up motor thread for game play with trials and changing conveyor belt directions
  • Built game-play hardware and motor control circuits
  • Report

Chloe:

  • Constructed conveyor belt
  • Set up continuous rotation servo with PWM control
    • Wrote change direction function
  • Set-up motor thread for game play with trials and changing conveyor belt directions
  • FFT debugging
  • Built game-play hardware
  • Report and website

Appendix D: Commented Code

EEG data receiving, FFT, and motor control:

Python serial script:

References

  • Github containing basis for python code: here
  • Website for app used to connect with Muse2 over Bluetooth: here
  • Website for Muse EEG headband: here
  • Paper that served as original inspiration for project: DelPreto, Joseph et al. “Plug-and-Play Supervisory Control Using Muscle and Brain Signals for Real-Time Gesture and Error Detection.” Robotics: Science and Systems(2018).