Signal2Speech

Inspiration

People with conditions like ALS, stroke, or cerebral palsy can lose the ability to speak or type, even though they still know exactly what they want to say. Most assistive tools either make them spell out every letter, which is slow, or guess for them, which takes the decision out of their hands. We wanted to build something that gets them to an answer quickly without ever putting words in their mouth. That is where our main rule came from: the app suggests what to say, and the wearer decides what they actually want to say.

What it does

Signal2Speech lets someone answer questions using an EEG headset (a cap that reads electrical activity from the brain) instead of their voice or hands. A conversation partner types a question, and Claude offers two short replies based only on facts the app has verified. The wearer picks one by looking at its card while the cards flash. A preview then shows the exact words, and the wearer confirms by closing their eyes for about two seconds. Only after that does the app speak, so nothing is ever said by accident. If the wearer wants something else, they can ask for new options, switch to saved phrases like "Yes." and "No.," or cancel. A user or caregiver can edit the saved phrases without touching any code, and holding one key for two seconds says "I need help, please." from any screen.

The same app also has a NASA air-quality example, which shows it can do something on screen and not only speak. The question is which Michigan county's Air Quality Index (AQI) rose more when wildfire smoke reached southeast Michigan in June 2023. When the wearer confirms "Compare counties," the app plays NASA satellite images from June 24 to June 28 (clear ground, three cloudy days, and then the smoke) next to the EPA's daily county AQI, and reads the result out loud. Washtenaw rose 186 points, and Wayne rose 168 points, from June 26 to June 28. A NASA fire map adds each day's fire detections, which shows that the fires were actually in Quebec and Ontario and only the smoke reached Michigan.

How we built it

The app is one Python process built on g.Pype, g.tec's Python toolkit for EEG, with a PySide6 (Qt) interface. EEG goes through g.Pype's filters and OSCAR artifact removal (cleaning out noise from blinks and muscle movement). To pick a card, we use a P300 chooser. The P300 is a brain response that shows up about 300 milliseconds after the thing you're watching flashes. We cut out the EEG around each flash, filter it to 1 to 30 Hz, average five flashes per card, and classify them with Linear Discriminant Analysis (LDA) from scikit-learn. To confirm, we use an alpha switch. Closing your eyes raises the brain's 8 to 13 Hz alpha rhythm at the back of the head, so one two-second closure sends exactly one Confirm. AI Answering uses Claude Sonnet 5.5 through the Anthropic API. Each request includes a "Verified facts" block built from our data, and Claude is told to say "I don't know" for anything outside it. Replies are spoken with Piper, a natural-sounding neural voice that runs fully on the laptop, and the app falls back to the operating system's voice on machines without it. The NASA example uses NASA MODIS imagery, NASA FIRMS fire detections, and EPA AirData, all bundled locally. Only the Claude replies need the internet, and everything else runs with Wi-Fi off.

Every input (keyboard, simulator, recorded EEG, and the headset) goes through the same selection contract. The contract's rules live in a shared scenario file that the code has to pass, so everything after the input is the same code. Since our headset never arrived, the app's default input is real human EEG that g.tec recorded on their own amplifier. Each card flash is answered with one real recorded brain response, and the same pipeline and classifier the live path uses calibrate on them and pick the card. GitHub Actions runs more than 500 tests on Windows, macOS, and Linux, and a pull request can't merge until they pass. Shared git hooks scan for secrets and run the tests before anything is pushed. We also used Claude as a coding agent throughout, with a shared CLAUDE.md so every teammate's agent started from the same instructions.

Challenges we ran into

The first big challenge was a pivot about an hour and a half in. We first planned to build in Unity with g.tec's starter, but that would have hidden the signal pipeline that the g.tec judges actually want to see. We moved the whole project to Python and g.Pype, and the selection rules carried over with no changes because they were already written as a test file. The bigger challenge was hardware. Our g.tec headset was lost in transit, so we borrowed PiEEG Octopus 16 boards from the venue library and tested on two people overnight. The board connected and recorded well, but we could not find a usable brain signal. Eyes-closed alpha actually went slightly down (0.96 and 0.87 times the eyes-open power, where a usable switch needs at least 2 times), and a flickering square gave no response. Our app was built around the Unicorn's 8-electrode cap, and the Octopus 16 is a small puck whose reference pin sits within 2 cm of its signal pins. That likely cancels a brain rhythm before it is measured, although we did not prove this.

We also ran into bugs that were easy to miss. Before we added grounding, every model we tried answered "Wayne" to the demo question, which is wrong because Washtenaw rose more. That is hallucinated information, and in an app that speaks for someone, a confident wrong answer is the worst kind of reply to offer. The P300 chooser and the alpha switch both numbered their events the same way, so the app rejected every alpha confirm as a repeat of the P300 preview, and the live flow could never actually confirm. Our backup voice also spoke the first reply and then silently skipped every one after it, which we only found because the board session's spoken cues went quiet.

Accomplishments that we're proud of

We are proud that the whole choose, preview, and confirm flow works end to end, and that it is built safely. Since the app speaks for someone, the worst failure is a repeated or late signal saying something the wearer didn't choose. Only one place in the code can speak, a decision left over from an old round is thrown away, and a preview cancels itself if nobody confirms within 15 seconds. Cancelling also takes a two-second hold, the same as confirming, so a stray tap can't throw away what the wearer chose. The timeout follows the fail-safe defaults principle from Saltzer and Schroeder: when something is unclear, the app does nothing instead of guessing. We backed this up with a fuzz test (a test that throws huge numbers of random inputs at the code), which ran 3,000,000 random operations with zero cases of speaking twice. An hour-long soak test also passed with 169 confirms, 169 spoken lines, and zero doubles, even after we killed the app on purpose.

We are also proud of how honest the demo is. A row of labels on screen always shows whether the input is Keyboard, Simulated, Recorded, or Live EEG, and nothing from the PiEEG board or a simulator is ever called live. On g.tec's recorded data from five people, our chooser picked the right card out of two 94% of the time from one flash and 100% from five. Each test used a part of the recording the classifier didn't learn on, and with the labels scrambled, the same code dropped to chance (42 to 46%). OSCAR made the biggest blink and movement swings 14 to 65% smaller while barely changing the rest of the signal. Finally, we kept raw EEG out of the repo with git hooks, and only the typed question is ever sent to the AI. EEG is personal health data, so there is absolutely no reason for it to end up in a public repo or be sent to an outside service.

What we learned

We learned that separating "choose" from "confirm" matters more than how accurate any single step is. A P300 pick can be wrong, but if the wearer has to deliberately confirm it with a different signal, a wrong pick costs a few seconds instead of the wrong words being said. We also learned that AI has to be grounded before it speaks for someone. The models sounded confident about Wayne, and the only fix was giving them the actual data and having the spoken numbers come from the dataset instead of from the model. Writing the rules as a shared test file first also meant four people could build the keyboard, simulator, recorded, and headset inputs in parallel and still have them behave the same.

On the technical side, most of us were new to brain-computer interfaces. The PiEEG tests taught us that the hardware's design decides what signals you can actually measure, and simulated data can't tell you that because it does exactly what you tell it to. We also learned how P300 and alpha signals show up in EEG data, why artifact removal like OSCAR is needed, and that OSCAR delays the signal by about a third of a second, which costs up to 0.14 accuracy if the flash markers aren't shifted to match.

What's next for Signal2Speech

The live headset path is built, so the next step is running it with a g.tec Unicorn on an actual wearer and measuring live accuracy with our two-card layout. So far, it has only passed on simulated signals. On the live path, we also need to handle OSCAR's delay and detect when the headset goes silent without fully disconnecting, which the app doesn't catch yet. We would also like to give Claude more verified data so it can answer more questions with real facts instead of "I don't know." Longer term, we would like to test with people who actually use assistive communication tools, since they would know best what a tool like this needs.