Wearing the Muse
Assembling everything on the can
Developers are so hard working people. Look at us - we're working 36 hours now, and after the Red Bull #10 we decided that there must be a way, a smart way, to track, suggest and help our kind while they program on their IDE.
What it does
Breakpoint is a fatigue sensing mechanical debugging tool that integrates with IDEs to help developers debug software in an ergonomic fashion. It pairs with a Muse headband sensing brain activity in realtime and notifies you when you’re too tired to productively code.
Using Muse (5 band EEG brain sensor), we track the users' mental alertness. We can tell when she is tired, needs a break, or just when she lost her concentration. This data is also displayed in real time in the developers' IDE. Feedback is given then via the LCD screen mounted the Red Bull can (pun intended). Users can control his IDE using the Joystick and the Plugin we developed for that. All the data is then being transferred using web sockets and is displayed nice and tidy to the user in form of graphs...because who doesn't like them?
We calculate fatigue based off of three measures derived from the EEG signals and compare readings over the past time window to the overall programming session to see if programming took a turn for the worst.
How we built it
This project contains 3 major systems, which we will discuss:
- Arduino - end user experience system
- IDE Plugin
- Muse server code
The Arduino software and hardware is the end user system. It contains the LCD giving feedback to the user. Using XBee Transmission device, we are writing and reading data from the Serial (received from the IDE Plugin), analyzing it and deciding on a proper action to take on the LCD screen. Wiring diagram is soon to come.
Muse server code
The Muse headband provides a 10 Hz stream of 5 band EEG data. I wrote a multithreaded Python server to capture this data, and a stream processing algorithm that calculates fatigue by looking for Increases in α and (θ + α)/β, and decreases in θ/α waves. These values (and calculated fatigue) are passed along using a web socket connection.
Our VS Code plugin is setting up multiple websockets. The muse python client is connecting to one of these, the visualization to another. VS Code forwards and analyses on the run. The data connection to the actual debugging device is done through a XBee Pro, so you could actually go as far aways as some kilometeres. All of that is orchestrated with different hook functions of the VS Code api and some hacks where these weren't enough.
Challenges we ran into
Most of the project went smoothly without major challenges. One challenge was the small memory Arduino Nano offers (2k bytes). To overcome this, we transmitted short byte arrays representing the messages. All the code written is a memory efficient code. Moreover we found the integration between the IDE and the Muse server somewhat challenging. We used web sockets to overcome this challenge.
Accomplishments that we're proud of
We started the event clueless. after 24 hours we had a breadboard with some wires and LCD screen. 2 hours later we had some graphs showing brain activity. The road to the Red Bull can you see on the picture, was fast and efficient. Controlling the IDE from across the room, as a show off, was amazing! It was stunning to see a full and functional product comes to life overnight from a Red Bull can.
What we learned
We learned tons. Firstly - about Muse and Arduino. We learned about the needs of a programmer, wireless communication, machine learning and much more.
What's next for Breakpoint
The next step for us is to commercialize those cans. We're going to put all the wires inside the can, and give it a clean non-hacky look and feel. Replace the 9V Battery with a rechargeable one and make it robuster with more sensors. Once we make such a can, we have basically a go-to-market product. In addition a GUI and special IDE will be developed.