BrainSync

Inspiration

Our main inspiration was Parkinson's disease. As it progresses, the tremor and stiffness make typing, pointing and tapping a screen harder and harder. Many people with Parkinson's also find their voice growing quieter and harder to understand. The mind stays sharp, but the ways it reaches other people slowly close.

The same is true of late-stage ALS, locked-in syndrome and severe cerebral palsy. Eye trackers help some people, but they need steady eye control and careful calibration, and medical brain-computer interfaces cost far more than most families can pay.

We wanted to know how far a cheap, dry-electrode EEG headset could go. The answer depends on two things: how reliably the headset tells a few on-screen targets apart, and how much each choice is worth. Our idea was to let the brain make only small, reliable choices and let an LLM do the rest.

What it does

BrainSync turns where you look into actions and words, without your hands or voice. The interface uses SSVEP: each choice is a square flickering at its own frequency (11 to 17 Hz). When you look at one square, your visual cortex follows its flicker. An 8-channel Cerelog X8 EEG headset picks that up, and a training-free decoder works out which square you are looking at.

BCI Voice is the main demo:

The conversation partner talks normally. Speech recognition runs on the partner's laptop or iPhone.
Claude (Haiku 5.5) writes three short replies in the user's own voice, ranked most likely first, using the conversation and the user's profile.
Each reply sits beside its own flickering square, and a fourth square, NEXT, asks for three more. Every option is always visible.
Looking at a reply opens a confirm screen ("Say it" / "Back"). Nothing is ever spoken without that second look.
The reply is spoken aloud with a Cartesia voice.

The partner sees a mirror of the user's screen, the conversation and a big Stop button. That runs in Chrome or in our native iPhone app, which has a Look to Speak-style setup.

The same headset and decoder also drive:

BrainPoint: a Chrome extension where Claude finds the clickable elements on any web page. The three best become flickering targets you click by looking. BrainPoint Desktop does the same for the focused Mac app through Accessibility.
BrainScroll and BrainSwipe: scroll Instagram Reels, or swipe through invented dating profiles, with your eyes.
BrainChess: pick chess moves by looking at them.
A live brain map: a scalp map of your own EEG in the browser (alpha, beta, SSVEP strength, contact quality).

A client/server split lets a Jetson Orin Nano own the headset and decoder while any browser draws the targets. We also started BrainSyncOS, an installer image that sets a Jetson up as a BrainSync appliance.

How we built it

Signal. The decoder uses canonical correlation analysis (CCA). For a window of EEG $X$ from the posterior electrodes (P7, P8, O1, O2) and a sine/cosine reference $Y_f$ at each target frequency $f$ and its harmonics:

$$\rho_f = \max_{a,b} \mathrm{corr}\left(a^\top X, b^\top Y_f\right), \qquad Y_f(t) = \left[\sin 2\pi h f t, \cos 2\pi h f t\right]_{h=1}^{H}$$

Filter-bank CCA (FBCCA) repeats this on several band-passed copies $X_n$ of the signal and combines them as $\rho_f = \sum_n w_n \rho_{f,n}^2$, with $w_n = n^{-a} + b$.

The 1/f background of EEG favours low frequencies even when nobody is looking at anything. So every score has a resting baseline subtracted, measured on a still screen and scaled by its spread $\sigma$:

$$s_f = \frac{\rho_f - \mu_f^{\mathrm{rest}}}{\sigma}$$

The online decoder scores the newest 1.5 s of EEG every 0.25 s. It commits a target only when the best score leads the runner-up by $0.5\sigma$, sits $0.5\sigma$ above its own rest level, and wins 4 windows in a row. Otherwise it abstains. We chose those settings from a 576-setting grid replayed over our recordings.

Stimulus. PsychoPy draws the full-screen, frame-locked sine flicker. It measures the display's refresh rate and refuses to run unless it is within 1 Hz of 60 Hz.

Voice and UI.

A FastAPI server owns all state as an event-driven state machine.
The decoder side and the voice side talk over a versioned WebSocket protocol, with a contract checker each side can run alone.
Claude is called with one forced, strict tool call, and every answer is validated with pydantic. If Claude is slow, wrong or offline, keyword-matched fixed reply lists take over, so the app never stops working.
Speech comes from Cartesia's API and is cached on disk, so replies still play offline.
The iPhone app is SwiftUI with on-device speech recognition. It pairs with a link the server prints, over the LAN or a Cloudflare tunnel.

Hardware. The Jetson installs with one command and is reached through a Cloudflare tunnel. BrainSyncOS remasters NVIDIA's JetPack installer ISO. It ships a signed package repository on the ISO, the Cerelog BrainFlow fork rebuilt for arm64, and a CH340 USB-serial driver built for NVIDIA's kernel, which lacks one. Everything is developed on macOS and Linux; Windows works too, as an extra platform.

Challenges we ran into

The first four recording sessions showed nothing at all. The Mac's ProMotion display was quietly running near 120 Hz, so the flicker on the glass was not the configured frequency. A fixed 60 Hz brought the SSVEP straight back. Now the stimulus checks the refresh rate itself.
10 Hz was unusable. It sits on the user's own 9–10 Hz alpha rhythm, and a still screen "won" the 10 Hz test about as often as the flicker did.
Dry electrodes are noisy. They need about 25 minutes to settle, two electrodes never reached the scalp, and the laptop charger coupled 60 Hz mains noise in. We ran on battery.
Gaze drifts. Without a box around the target, the eyes wander to the other square after about 2 s. That works in a cued recording, but live use cannot hold a box on a target nobody has chosen yet, and this is still open.
Knowing when not to decide. Our first decoder committed on about half of all still screens, when nobody was looking at anything. Requiring a response above the resting level cut that sharply, at the cost of some correct picks.
Venue Wi-Fi. With 1–4 s pings, every Claude call first timed out. We added kept-open connections, a hedged second request, keyword fallback pages and offline speech.
Safety. Every frequency we use is in the photosensitive range. Every client shows a warning and draws no flicker until it is accepted, Esc stops the flicker locally, and the flicker stops whenever the connection drops.

Accomplishments that we're proud of

A working dry-electrode SSVEP decoder, with honest numbers. On three cued two-target recordings (15/17 Hz, both squares flickering), the best settings reached 89%, 82% and 93% (17/19, 14/17 and 13/14 trials), two clearly significant and one borderline after correcting for the settings we tried. Still-screen controls stayed at chance.
A decoder that abstains instead of guessing. Replayed over those recordings, the online decoder made 12, 10 and 8 correct picks with 0, 2 and 0 wrong. Its median time to a correct pick on the reference session was 1.24 s.
CCA and FBCCA that match the published benchmark. On the 12-target Nakanishi 2015 dataset they reproduce the published results within a point: 92.2% and 93.7% with 4 s windows.
An LLM that never decides for the user. Only the decoder (or keyboard mode) picks; the model only proposes. Every option stays visible, and nothing is spoken without confirmation.
About 2,900 automated tests. They cover the decoder, the protocols, the state machine, the servers, the browser and extension code, and end-to-end runs against a real-time fake headset.
A Jetson installer ISO that builds end to end. It is built from a pinned NVIDIA base, with its own signed package repository.

What we learned

Hardware details beat algorithms. The refresh rate, electrode contact, a charger cable and where the eyes rest mattered more than the choice of classifier.
Measure "nothing" too. A still-screen control and an eyes-closed recording caught classifiers that looked good but were really following alpha or the 1/f background.
Small samples lie. With 14–19 trials per recording, a still-screen control once scored 79% by chance. We report permutation p-values and treat gaps under about 10 points as noise.
The LLM is where the speed comes from. One reliable look is not worth much on its own. When the right reply is usually on the first page, a few seconds of looking becomes a whole sentence.
Fallbacks are a feature. Every network dependency (Claude, Cartesia, speech recognition) needed an offline path before the demo felt safe.

What's next for BrainSync

Four targets, measured. We still have to record the planned 4-target session (13/15/17/11 Hz in a diamond). It decides between showing 3 options at once and a 2-target list, and gives us data to train a per-user classifier (TRCA).
Better contact. Gel on O1/O2, and the floating electrodes moved to Oz/POz.
BrainSyncOS on real hardware. Boot it on an Orin Nano, run BCI Voice against a local llama.cpp model first, and check the flicker timing while the model runs.
Learning while in use. Every confirmed "Say it" is a labelled trial, so the decoder could recalibrate during a conversation.
Real users. Test with people who need it, with their consent and proper clinical guidance, and in their own banked voice where they have one.