Inspiration
For the estimated 7.42 million people worldwide with neck-level spinal cord injuries [1], the millions of people with cerebral palsy who have severely limited mobility [2], and others with conditions such as ALS, mobility solutions are not just prohibitively expensive, but also very difficult to get access to.
Comparing the devices available to people with paraplegia who retained arm function, we found the devices available to those without leg and arm function were not even publicly quoted, as just specialized eye monitoring hardware used for their wheelchairs cost 2 orders of magnitude more.
Example) Tolt Technologies' Ability Drive [3] costs over $22,000 in specialized hardware alone, excluding the wheelchair, and requires:
- Finding a specialized medical equipment distributor.
- Undergoing a professional assistive technology evaluation.
- Obtaining a custom quote and compatible equipment.
- Arranging professional installation and programming.
What it does
Our project allows those who cannot use their hands to navigate using only their eyes. We combine eye tracking technology with EMG signals measured from electrodes placed around the head, in order to do this in an accessible way that does not require extensive personal calibration.
(We named it after Charles Xavier, the X-Men professor who gets around by intent instead of hands.)
How we built it
Reading brain signals
We initially explored SSVEPs (a technique where looking at a flickering box produces measurable EEG activity at the same frequency of the flickering) [4]. We used canonical correlation analysis (CCA) to identify these frequencies [5], placing electrodes on occipital lobes by measuring the head.
We tested four stimulus frequencies (11, 13, 15, and 17 Hz), recording 15-second intervals for each. After several recorded sessions of 10+ minutes, our offline frequency classifier correctly identified the target in, on average, 9 of 44 analyzed windows (20.5%), below the 25% chance baseline. This was due to substantial electrical interference and signal fluctuations, which were likely due to noise from electrode contact through hair, muscle interference, and variations in placement from our measurements.
We pivoted to jaw-clench detection, using the OpenBCI hardware to capture electromyographic (EMG) activity. Because of the noise produced by muscle activity, we decided to use them as the primary signal [6].
Object detection
EyeTrax, an open-source webcam gaze library is used to isolate objects that the user may want to move towards. We use YOLO and ByteTrack to box the objects in the camera feed, so that the box the user looks at is the object they choose to move toward.
Eye tracking
We use eye-tracking to find the object the user looks towards (if they do not look at the screen they remain stopped). Before moving them though, we confirm by feeding the signal from the electrodes through our ML model to check whether they are saying yes or no to moving. If yes, we send a signal to move the chair!
Moving the chair
Those signals feed a safety state machine that runs 20 times a second, defaulting to a stopped state. It sends an updated wheel speed to an ESP32 every 50 ms, using our custom Python library. The ESP32 turns the signal into 9-byte frames for the hoverboard's two boards. We erased the factory firmware and flashed our own build to be able to control the board's movements (https://github.com/RoboDurden/Hoverboard-Firmware-Hack-Gen2.x).
Challenges we ran into
**Mapping the pinout of the firmware ourselves with a multimeter, probing the pads and reading their voltages one until the hall sensors, the serial lines, and the battery sense lined up. **Noise interfering with the brain recordings, and many false positives as a result until we refined our model. **Aligning the eye-tracker, OpenBCI signals, and ESP32 signals
Accomplishments that we're proud of
- We reverse engineered a piece of hardware (the hoverboard's PCB, for which there was no online specs), programming it with our own firmware. This allowed us to control the hoverboard's motors from our ESP32, and we created an extensible Python library to call different functions of the hoverboard from our microcontroller
- It was all of our first time working with OpenBCI, and we trialed several brain monitoring approaches, moving from EEG signaling to EMG signaling and troubleshooting high levels of noise and impedance. After significantly improving the signal to noise ratio we trained a custom ML model and set up an API to provide live inference
- Setting up the chair was more complicated than we expected, since the hoverboard caused the chair to rock back and forth, adding noise to both our eye tracking mechanism and brain signals. We ended up jerryrigging our chair with a few coat hangers providing a stable base. We also ran into a lack of hardware, setting up a custom switch because we couldn't find one.
Use of CoStar
Navigating to a certain room in a building is something many of us take for granted, but for a person with mobility issues, there are several limitations to the paths they can take. CoStar allowed us to create a novel application of digital twin technology that enables people with mobility impairments to find the fastest accessible path to their desired destination within a building. We use floor plans and spatial data retrieved from the Matterport API, along with the wheelchair's current location, to calculate routes using the A* pathfinding algorithm.
We use Gemini to label features in CoStar's spatial data, such as whether a passage contains stairs, a ramp, or an elevator. We then compare the dimensions and mobility requirements of the user's wheelchair against the available building data to identify possible routes, eliminate those with accessibility hazards or uncomfortable passages, and select the most suitable path.
CoStar allowed us to transform Xavier from a wheelchair that simply assists its user into one capable of autonomous navigation, demonstrating a novel application of real estate data in accessibility technology.
Use of Gemini
We use Gemini to analyze spatial data from CoStar's Matterport digital twins and identify accessibility-relevant features, including stairs, ramps, elevators, and potential obstacles. These labels help us determine which routes are suitable for a user's specific mobility device.
By combining Gemini's understanding of the environment with our A* pathfinding algorithm and wheelchair-specific constraints, we can prioritize routes that are not just short, but accessible and comfortable to navigate.
Use of Vultr
We used a Vultr machine to train a random forest regression model from all the EMG data we had collected. We then served a FastAPI endpoint for inference, feeding in the OpenBrain telemetry data live in order to get live responses on whether or not the user had approved a movement.
Use of Amazon
We used Amazon's polly as a voice agent within our guidance
MLH - SpaceXSI
We wrote the pieces in Cursor: the gaze loop, the vision pipeline, the safety machine, and the bridge firmware, then stitched them into one path from a look to a wheel.
MLH - DEV
https://dev.to/vani_ramesh_dbaf9093a790c/hackprinceton-making-charles-xavier-143a
What's next for Xavier
We would like to expand from our use of EMG to EEG signals in order to get more precise signals. Our prediction model currently uses the signals measured from 2 electrodes as well as the difference among them as its only features, but as we introduce more electrodes we would be able to do more intricate things as well as include more options for the user (e.g. they are shown more potential rooms to navigate to from the getgo).
We believe that repurposing cheap electronics into accessible systems has a massive impact, considering that only around 3% of people with accessibility needs in third world countries have those needs met in some way. Utilizing data and existing electronics to help those people is something we feel strongly about.
References
[1] https://pmc.ncbi.nlm.nih.gov/articles/PMC12446318/ [2] https://pmc.ncbi.nlm.nih.gov/articles/PMC1919364/ [3] https://www.tolt.tech/_files/ugd/0a876c_799177bb843145879463e78e30856ad5.pdf [4] https://doi.org/10.1167/15.6.4 [5] https://doi.org/10.1109/TBME.2006.889197 [6] https://doi.org/10.3389/fnhum.2013.00138
