David Pirogovsky (dap263), Alec Newport (acn55), Phillip Douglas (pdd35)
The Constellation Glasses allow you to find out what you are looking at in the night sky with the click of a button. On one side of the glasses, there is an accelerometer- magnetometer mounted, which allows us to determine the position of your head. In a separate box, connected to the glasses by a null modem cable, a GPS module informs the PIC32 of your position, calculates your right ascension and declination based on the accelerometer and GPS data, and sends a request to a locally hosted Python server through the WiFi module. The Python server finds the closest star to your viewpoint, converts the name of the star and the constellation it is to speech, and returns that information to the WiFi module to be played on the speakers on the DAC.
HIGH LEVEL DESIGN:
Our system consists of five main parts: the WiFi module, a locally hosted Python server, a GPS module, the PIC32, and an accelerometer/magnetometer module.
Figure 1: High Level Diagram
Figure 2: PIC32 Pinout
The WiFi module requires up to 500 milliamps of current at peak consumption, and can draw 250 milliamps consistently, and requires inputs ranging from 3.3 volts to 6 volts. The small development board26 has a 3.3 volt regulator on the output, but it cannot source enough current. Luckily, the board also has two pins that output the unregulated voltage directly from the power supply. Since we were going to use a 9 volt battery, we needed a 5V regulator. The 5V regulators in lab can only source 100 milliamps, so we built our own regulator, designed by Bruce Land, using a TIP31 power transistor, a zener diode, two resistors, and two capacitors. This regulator is still capable of powering the WiFi module on a 5V input, even though it outputs about 3.8 volts with a 5 volt input.
Figure 3: Voltage Regulator Circuit
Figure 4: KMX62 Connection to PIC32
The KMX62 eval board found online and linked in the datasheet provides incorrect pin assignments. The pinout seen above has been corrected.
The KMX62 can communicate on an I2C bus through the SDA and SCL lines. The first 7 bits of the KMX62 slave address are factory set to 000111X, where X is a bit that is set by the ADDR pin on the device. For the purposes of our project, we wired ADDR to ground, thus the first 7 bits of the address were 0001110. The last bit is determined by whether or not the master is reading or writing the device. If the master is writing the last bit is a 0, and if the master is reading the last bit is a 1. In order to make life simpler, we defined the read and write addresses as constants, as seen in Figure 4.
Figure 4: I2C Read and Write Addresses
Our I2C read and write functions, as seen in Figures 5a-c, were simply translated from the KMX62 datasheet1. The datasheet did a very good job of completely outlining the steps that needed to be taken. In addition, we used both the Blimp-O-Project2 and Pose: an Arm Tracking System3, two older ECE 4760 projects, as reference for our I2C read and write functions
Figure 5a: I2C Wait Helper Function
Figure 5b: I2C Read Function
Figure 5c: I2C Write Function
Before we ever use any of the values coming out of the magnetometer or accelerometer, we send them through a digital low-pass filter, seen in Figure 6 below, in order to filter out the higher frequency noise. Beta determines the cutoff frequency and responsiveness of the low-pass filter. A lower beta value leads to a better low-pass filter with a very slow response time.
Figure 6: Digital Low-Pass Filter
In order to know what position in the sky the user was looking at, we need to figure out the angle of their head with respect to the horizon, also known as pitch, as well as what cardinal direction they were facing, also known as yaw. Both of these measurements can be obtained from the KMX62. We used the accelerometer to find both roll and pitch angles of the device, as these values will be necessary in future calculation. All of these measurement directions are shown in Figure 7 below.
Figure 7: Roll, Pitch, and Yaw Directions
Roll and pitch can be calculated easily from the following equations:
We negate X Accel in the pitch calculation to follow a predetermined coordinate system.
Using the magnetometer proved much more difficult than the accelerometer. In order to be useful, the magnetometer must be calibrated for all EM noise. There are many ways to do this. We chose the simplest method. We have the user spin the sensor in a circle on startup, and average the X, Y, and Z the magnetometer values over about 5000 cycles. The end result is our X, Y, and Z “hard iron” offset values. If these offsets are subtracted off the raw values from the magnetometer, the resulting values are the “real” magnetometer readings of the Earth’s magnetic field, as described by the following equations:
The final step is to calculate cardinal direction (yaw) from all of our previously calculated values. If we assume that the sensor is to stay completely flat, ie. roll and pitch are both zero, the calculation is simple:
However, due to the nature of our system, we cannot assume that the device will always stay flat. Thus we have to compensate for pitch and roll.
Throughout all of this calculation, we heavily used STMicroelectronic’s AN3192 application note on how to create a tilt compensated electronic compass4 as well as FreeScale Semiconductor’s article on “Implementing a Tilt Compensated e-compass using Accelerometer and Magnetometer Sensors”5.
To convert from the local altitude/azimuth coordinates output by the accelerometer to the absolute right ascension/declination coordinates in the start database we used, we needed temporal and position data. Both of these are transmitted by GPS signals, so we purchased the Adafruit Ultimate GPS Breakout board to acquire the data and convert it to a form usable by the PIC32. This board collects the freely available GPS data from satellites orbiting Earth and converts it to standardized NMEA sentences, which it then sends out over a UART interface. We initially connected to the UART on the board using a USB to serial cable and PuTTy running on a PC. This way, we were able to view which of the NMEA sentences this particular module transmitted. The output on PuTTy with a GPS fix is shown in Figure 8. We found a website6 that lists the information contained in each sentence. We decided to use the GPRMC string for our purposes, because it contains all of the information we need: date, time, latitude, and longitude.
Figure 8: GPS Serial Output with Fix
To extract this data on the PIC32, we connected the GPS board to UART1 on the PIC. We initially tried modifying the UART code provided to us in the file pt_cornell_1_2_3.h, but we were unable to get the string to display properly on the TFT on the Big Board we were using for debugging. Instead, one of our TAs, Vidya Ramesh, told us that her group had used the same GPS module for their ECE 4760 project, Borkbit: A WiFi-Enabled Smart Collar7. They wrote a file, GPS_UART.h, that contained two functions, PT_GetSerialBufferGPS and parse, that we based our own GPS code off of. They used the GPGGA string instead, so we needed to change these functions to work with the GPRMC string we are using.
The primary thread that deals with the GPS data in main.c is protothread_GPS, shown in Figure 9 below. Approximately every 800 msec, it spawns a child thread that runs the function PT_GetSerialBufferGPS. Once the child thread has completed, it checks if a RMC sentence was found in the serial data. If it was, it parses the data into manageable variables. It then checks to see if the GPS module has a fix. If the status of the fix has changed, it alerts the user using the appropriate sound data output to the speaker.
Figure 9: GPS Thread
PT_GetSerialBufferGPS is the thread that actually gets the data being sent over the serial line from the GPS module. We had some trouble with this function due to the fact that the GPS sends numerous NMEA strings all at once, each separated by a newline. Since we need exactly one of the lines from each burst, and there is no delimiting character between bursts, we found the best way to find the GPRMC sentence was to always acquire a large number of characters from the serial monitor each time the thread is executed (we chose 256 characters), have no termination character, and search the returned data for the sentence ourselves. The code we came up to do this, once the 256 characters are stored in PT_term_buffer_GPS, is shown in Figure 10. It searches PT_term_buffer_GPS for the substring “$GPRMC”, which denotes the beginning of our desired sentence. If that substring is found, we then search from that point onwards in the buffer for a newline character, which denotes the end of that string. If we find that as well, we zero-terminate the string and copy it over to a new buffer called PT_term_buffer_GPS_RMC, so that we have only the information we need. We then set the flag named GPRMC to denote that we have found the string.
Figure 10: Finding GPRMC Sentence
The function parse_RMC takes care of parsing the whole GPRMC sentence into variables for each of the bits of information it contains. We store this data in global variables so that it can be accessed elsewhere in our code. Some of the data is transmitted in slightly obscure formats, so we translate it into more manageable units using the code shown in Figure 11. First, we convert the character denoting whether the string contains valid data, stored in GPS_valid, into a boolean value, stored in GPS_fix. In the NMEA string, the character ‘A’ indicates the string contains valid data, and ‘V’ indicates invalid data, so we convert ‘A’ to true and anything else to false. Next, we have to convert the latitude and longitude from degrees, minutes, and a direction to positive or negative degrees and fractional degrees. In the received data, the latitude and longitude values, stored in GPS_dmLat and GPS_dmLon respectively, have minutes in the ones and tens place (as well as the decimal places), and degrees in the hundreds place and up. Using this information, and the fact that there are 60 minutes in a degree, this value is converted into degrees. The direction of the latitude and longitude are transmitted separately in the NMEA sentence. We temporarily store these in GPS_NS and GPS_WE. Since we want signed degrees, if the latitude direction is south, denoted by an ‘S’ character in GPS_NS, or the longitude direction is west, denoted by a ‘W’ character in GPS_WE, we make the degree value of the corresponding variable negative. Then, we convert the time into three separate variables for hours, minutes, and seconds. The time is transmitted with hours in the hundred thousands and ten thousands places, minutes in the thousands and hundreds places, and seconds in the tens, ones, and decimal places. We do not need the time to be accurate to less than a second, so we discard the fractional seconds. The date is stored in a similar format to the time, with days in the hundred thousands and ten thousands places, months in the thousands and hundreds places, and years since 2000 in the tens and ones places. All of these values are separated out and placed into their individual variables.
Figure 11: GPS Unit Conversion
We had a problem for a little while where the incorrect values were being stored in the variables. We realized this was because when the GPS doesn’t have a fix, it sends nothing in the string where data values should go, i.e. there would be portions of the GPRMC sentence that were just series of commas in a row. Because of how sscanf works, some of those commas were being pattern matched into some of our variables. The workaround we devised to fix this problem was to only look at the data variables when the GPS string was valid. Since the string type ($GPRMC), time, and valid character are always transmitted regardless of the validity of the data, we can always be sure that the valid character contains the correct value from the string. Therefore, if the valid character isn’t ‘A’, we ignore the rest of the data.
Button Presses and Coordinate Conversion:
The user interface for the Constellation Glasses consists only of a single button. All of the code for a button press is contained in protothread_button. This thread is structured as a state machine, shown in Figure 12, based on the state of the button to debounce the input (avoiding the same speech being played multiple times on each button press). Every approximately 30 msec, the button pin is read and the state changed accordingly. 30 msec is enough time for any oscillations caused by pressing or releasing the button to die down. If a button press is confirmed, we call the function AltAz2RaDec to convert the current head position to right ascension and declination, and then we send the data to the WiFi module using ESP_request_data.
Figure 12: Debouncing FSM
With the data from the GPS and accelerometer/magnetometer in global variables, the AltAz2RaDec has all of the information it needs to convert the local altitude/azimuth coordinates to the absolute right ascension/declination coordinates that can be used to search through the star database. This involves some fairly complicated formulas, which we found on a forum post8 in comments made by users Surveyor 1 and Range 4. The declination can be calculated directly from the altitude, azimuth, and latitude, but the right ascension requires some intermediate steps. First, we calculate the Julian Day, which is the number of days since noon on January 1st, 4713 BCE9. We then calculate the current Universal Time in hours and fractional hours. Since the time given by the GPS is already in Universal Time, this is a simple unit conversion. We then use these two values to calculate the days and fractional days since the start of the current epoch. The epoch is the “reference time” that we use for the right ascension/declination coordinate system10. We are currently in epoch 2000, meaning that we measure days since January 1st, 2000 CE at noon UT. The next value that we need is the sidereal time. Sidereal time is the time relative to the background stars, rather than that relative to the sun. It can be used to calculate the local sidereal time, which is the right ascension coordinate currently passing directly overhead11. This, combined with the hour angle, which is the angle between a circle passing directly overhead and over the poles and one that passes through the observed coordinate and over the poles12, is used to calculate the observed right ascension.
We started with a ESP-01 module that was lab surplus, but gave up because we were unable to determine the firmware on the chip, and couldn’t find firmware to update it to that had enough documentation with it to actually determine which commands were usable. We decided to buy a the ESP8266 HUZZAH module from Adafruit so that we would know exactly what the hardware details were and have better documentation for it. This came with four programming options: controlling it with AT commands from Espressif (the Chinese company that manufactures the ESP8266 chips), using the Arduino IDE, using NodeMCU Lua, or using micropython. We decided that since we were not sure how much we would want to change on our WiFi module, we would use the NodeMCU firmware with the Lua interpreter on the chip so that we would have more flexibility.
The steps to initialize the module are as follows (All python code was run on Python 2.7.12 :: Anaconda 4.1.1 (64-bit)):
> python esptool.py --port <COMx> --baud 78400 write_flash -fm dio 0x00000 <downloaded_file.bin>
After updating the firmware, you can open PuTTY, or any other serial terminal on your computer, and change the settings to the COM port the serial cable is on, select serial and select a baud rate of 78400. Make sure to enable “Implicit LF in every CR”, which sets the carriage return properly so that you can press enter and get a new line. Then, you can start reading the NodeMCU documentation13 and begin testing out your WiFi module.
Initially, we just used PuTTY to test code, but this quickly became tiresome. We found ESPlorer (see references), which allows you to upload files, write snippets and run them, and gives a much more convenient graphical interface for interacting with the module. This significantly sped up testing, debugging, and development, and made it much easier to upload code. We used the init.lua file from the NodeMCU examples16, and added in a line at the end to set the UART configuration to 256000 BPS to match the PIC. The init.lua that we use comes with code to allow three seconds to make changes to the file system before it finishes executing, so that if any code written after that causes a “PANIC” (power-on reset), there is enough time to delete it and recover from an infinite boot loop.
Our goal was to request information from the local server and have it returned in the form of speech. Our first approach was to try to use the play_network.lua example code from the NodeMCU GitHub16. However, this would not run properly, because we were unable to turn on the Lua Flash Store (LFS). Enabling LFS requires uncommenting a line in a user_config file13 and building the firmware yourself, but we were unable to install Docker, the software needed to execute your own customized build, on my version of Windows 10. We decided to try modifying this file not to require LFS, and was eventually able to write to a file without any problems, but realized that this was useless, since reading from a file on the SPIFFS file store was too slow to support audio streaming. We also followed the pcm examples to try to use the 8-bit Sigma-Delta generator on board the WiFi module, but could find no useful documentation as to how it worked (the only mention in the Espressif datasheet is to say that it exists), and was getting unexpected behavior when we were trying to play back square waves at different frequencies. We then added the SPI module and tried to read from a file and send the data line by line through SPI directly to the MCP4822 DAC, but the filesystem was still much too slow to support playback by this method. From my experience, it seems much easier to use a separate chip with better documentation to do actual processing, and have the ESP do the bare minimum of the necessary communication between other parts, at least when flashed with Lua.
The last step we tried before transitioning to that approach was using the GPIO module to control a passive 8-bit R2R DAC with 8 GPIO pins. The only code used was if else blocks with GPIO writes, but this could only generate a 31 Hz square wave. When we decreased it to 4 pins, we were able to get around 300 Hz, which is still basically unusable. According to the GPIO documentation, there is a way to generate accurately timed pulse trains across multiple pins, but this would still not be helpful for an unknown pattern of sound, and would cost a lot of computation and time to generate from the full speech file before playing it back. Also, this required the same edit to the configuration files as enabling the LFS.
Finally, we gave up on using the WiFi module as a standalone unit. we realized that the print statement on the WiFi module automatically outputs to the onboard UART, and decided to stream the speech to the PIC. Since Lua is designed to be an embedded language, most asynchronous processing is done through callbacks, rather than normal functions, in order to decrease RAM usage. The full Lua code used is shown in Figure 13. The first callback function terminates the sent data with an ASCII control character, which is registered on the PIC as an end of packet character, and just breaks out of the receiving while loop. The second callback function sends the received packet (about 1500 bytes), and if it is the first packet, sends an ASCII control character that will start the dma channel reading from the input buffer. The UART is configured at 256000 bits per second, so, assuming no WiFi connection delays, the buffer will be filled about 5 times faster than it is read from. Currently, the size of the input buffer is 15000 shorts, meaning it can store about 2.7 seconds of playback. We intended to add support for continued playback by using more ASCII control characters, and possibly a GPIO pin between the modules, but more bugs popped up on integration that were a higher priority.
The WiFi thread calls ESP_setup() before entering an infinite while loop, which registers the two callback functions. The thread then spawns a receiving thread on UART2, using the PT_GetMachineBuffer method, and yields to other threads. ESP_request_data(message) takes an input message to send to the server, which must start with either “tts: “ or “ra: “. For tts, the message is whatever the user wants converted to speech, and for ra, the message is the right ascension, followed by a comma, followed by the declination. Figures 13a and 13b present both of these functions in their pure Lua representation, but in the C code they need to be wrapped in printf statements, with all the quotations and backlashes escaped with a backlash, and every line followed by a “\r\n” for the carriage return.
Figure 13a: Lua code in ESP_setup()
Figure 13b: Lua code in ESP_request_data(), “Hello, World” replaced by message
The Python server is fairly simple. We combined the code for generating the text to speech from three sites17, and used the python wave library to determine that the wave file was a mono, 16-bit signed pcm signal at 22.05 kHz. This method has only been tested on Windows 10 and requires you to use “pip install comtypes”. We first tried to brute force convert the frames, which were represented as two hexadecimal bytes, to one unsigned byte, but was unable to figure out the formatting. We found two lines of code18 (Figure 14) to convert the frames from the struct format to signed shorts, and then convert them from signed shorts to whatever formatting we wanted. To remove the sign, we added 215 to them, then shifted right by 9 to put them in the 7-bit range, and added 32 so that none of them would register as ASCII control characters. Initially, this was converted to a string of binary, because the formatting necessary for the WiFi module was unclear to me, but converting to ASCII characters was more efficient and accomplished the same goal. Each of these is stored as a NumPy uint8 number25 to ensure proper formatting. We also downsampled here by only keeping every fourth sample in the file. They are then converted to the ASCII representation by calling the Python method chr() and concatenating them all into one string.
The CSV file is downloaded from the HYG Database24 website. Every time the server is started, csv_parser.setup_csv() is called, which goes through the CSV row by row using python’s built in DictReader class19 and adds each column to a list if the star has a proper name. After the lists have been created, the right ascension and declination are converted from spherical coordinates to absolute x, y, and z coordinates and added to their own lists. When searching for a star based on an inputted right ascension and declination, these coordinates are converted to cartesian as well, and the minimum difference between the input and something in the database is returned, with the text to speech method automatically being called to return the result to the WiFi module in speech form. The database initially just had abbreviations for each constellation, so we went through and replaced the abbreviations with the full names, so that the parser could return in the format: “<Star> is in <Constellation>”.
Figure 14: Wave File Conversion Code18
RESULTS OF DESIGN:
The speech produced from the WiFi module has an annoying clicking that occurs about 7 times a second, which we believe is caused by periodic bursts in activity from the WiFi module. We first noticed this when on our first success with speech output, hearing it and seeing a 1.5 volt drop on the output of the DAC. When we built the 5V regulator for the module, there was a periodic drop with a similar frequency but much smaller magnitude. The speech is not possible to analyze thoroughly using an oscilloscope, since the bursts are much to short, but downsampling to 5.5 kHz definitely decreased the ease of understanding it.
There was about 2.7 seconds of a delay between pressing the button and the DAC beginning to output.
The IMU returned pitch and roll values accurate to +-5 degrees. The tilt compensated compass returned a heading value accurate to +-5 degrees when flat, and accurate to +-10 degrees when at 90 degrees tilt. Ideally we would like all of these measurements to be more accurate, however, at this accuracy level the product should work as intended.
Our device conforms to the UART standard, using a standard baudrate and the proper hardware channels, the I2C standard, by using the proper pull-up resistors, and the applicable WiFi standards are enforced by the WiFi module. The ASCII control character definitions and HTTP standards, however, were partially ignored for the sake of convenience, with random control characters being used, and the HTTP header being removed from the server response.
The results of our design did not quite meet our expectations. Most of the design steps, especially with the KMX62 and the ESP8266, were much more convoluted and time consuming than we anticipated, causing our full system integration to be pushed back, and leaving some bugs in the final design unresolved. Since we wanted to wait for working designs before we moved everything to solder boards, we did not take our design outside until the last minute, and were unable to test the server with the GPS from inside, because the antenna was too weak to get a fix inside. The system would take a few seconds after receiving a GPS fix to output correct information, and would output correct information on the first reading, but after that would repeat this definition for an undetermined, variable length of time, which suggests that one of the threads would be left hanging, or the accelerometer/magnetometer was not able to return a new reading. We would have liked to improve the responsiveness of the system as a whole, and to add flow control between the WiFi module and the PIC in order to allow speech playback with no time limit. This would have allowed us to read back the full Wikipedia page for a constellation, and provide other kinds of information, on a user input like a long button press.
Safety was of the utmost importance when design the physical system to be worn by the user. Our final system was fully insulated, and the user could not come into contact with any wiring.
Throughout the entire project, we consulted with TAs in lab as well as assisted other students in lab whenever the opportunities arose. Over the course of our project, we determined that our data limitations were still sufficient to provide the user with a good idea of what they were observing in the sky.
This project could have potentially useful implications for the blind community. The overall cost is very low, and this design could be integrated with the Google Maps StreetView API to give you voice information on your surroundings, potentially informing the user of a crosswalk, or when they have reached their desired location.
About half of the code that was used for the WiFi module and Python server was modified from the official Python or NodeMCU online documentation. The basis for the server code, just enough to support a request from a client, was copied from an Operating Systems lecture taught by Robbert van Renesse. The NodeMCU examples are on an MIT license.
The mathematics used to calibrate the magnetometer and implement the tilt compensated compass was adapted from scholarly articles written by STMicroelectronics and FreeScale Semiconductor and compliant with their rules and regulations.
Our project did not make use of any patented material.
The Adafruit HUZZAH ESP8266 Breakout is FCC tested and certified.
The Adafruit Ultimate GPS Breakout is FCC tested and certified. Our use of the Adafruit Ultimate GPS is compliant with all FCC regulations.
The KMX62 is FCC tested and certified.
The group approves this report for inclusion on the course website.
The group approves the video for inclusion on the course youtube channel.
APPENDIX B: BILL OF MATERIALS
Small Development Board
Adafruit Huzzah ESP8266 Breakout
Adafruit Ultimate GPS Breakout
Normal Solder Board
Small Solder Board
9 volt battery
DB9 Null Modem Cable
APPENDIX C: CODE
All of our code can be found on github:
APPENDIX D: WORK DISTRIBUTION
David Pirogovsky worked on the python server, csv parsing, text to speech, figuring out how to play speech with the WiFi module, setting up the DMA for audio output to the DAC, and debugging the UART receive code from the WiFi module.
Phillip Douglas worked on I2C communication with the KMX62 Accelerometer/Magnetometer, pitch and yaw calculations, physical glasses design.
Alec Newport worked on the GPS module, the UART connection with both the GPS and WiFi modules, the code to convert between altitude/azimuth and right ascension/declination, button debouncing, and the final, physical construction of the protoboards, electronics housing, and glasses/electronics interface cable.
APPENDIX E: REFERENCES
APPENDIX F: PHOTOS
Photo 1: Glasses, Button Side
Photo 2: Glasses, Accelerometer/Magnetometer Side
Photo 3: Internal Circuitry (Top to bottom: Phono out, DAC, 5V regulator, Small Development Board, WiFi Module, GPS Module)
The basis for the python server code was taken from an Operating Systems lecture by Robbert van Renesse. Additionally, we would like to thank all of the TAs, especially Nick and Vidya, for all of their support, and Bruce Land for his abundance of suggestions and solutions to many of our problems.