Residual

Inspiration

Someone close to us, an uncle who is only 50, is living with ALS. ALS takes movement away one piece at a time, and the eyes and eyelids are usually among the last things that still answer. That made the problem personal, and it set our bar: this has to work for a real person, not just in a demo.

We started where everyone starts: eye tracking. We tried it ourselves, even Apple's. It was less accurate than everyday use needs, and holding your gaze on a target all day is tiring. If that strains us after a few minutes, it is hard to imagine relying on it for hours when you can't move anything else. So we chose two signals that are comfortable to repeat: blinking twice and raising your eyebrows. Everything in Residual is built from just those.

## What it does

Residual is an interface for a person with ALS, controlled by only two signals.

Two big choices on every screen. Blink twice for the left choice, raise your eyebrows for the right, and close your eyes for a moment to go back. Long lists split in two each time, most likely first, and the things he uses most rise to the top.
Learns his face first. A short setup measures his own blink and eyebrow range. No two people move the same way.
A whole day, not just care requests. Say something in a natural voice, message family ("I love you", "Call me tonight"), YouTube, Netflix, Reels, music, and any website through Browser Use.
It adapts to him, and asks first. When his eyebrows tire, Residual shows exactly what it saw and offers one-signal mode: the choices light up in turn and one blink picks the lit one. The interface changes, not the person.
A care circle of agents decides who to wake. Each family member has their own agent on Agentverse. When he asks for help they bid on who is best placed: free or busy, how far away, who has helped most this week. Only that person's phone buzzes.
Real messages, real replies. Photon sends the text, "on my way" comes back, and his screen says help is coming.
Emergencies skip the circle and go straight to the care plan's contacts.

## How we built it

Applied Intelligence: a local model on his eyes and eyebrows. A MediaPipe Face Landmarker model runs entirely on the laptop. Its weights and runtime are served by the app, never a CDN, so frames are locally shared to the DeepSeek model and never leave the machine. From each frame, the model reads five numbers: eyeBlinkLeft, eyeBlinkRight, an eyebrow score from browInnerUp and browOuterUp, face presence, and head pose.

A pretrained model is generic, so the applied part is personalization and decision logic that DeepSeek customizes on top of it:

Per-person calibration. The setup stage allows DeepSeek to come up with unique threshold configurations per user.
A switch state machine. The model makes deliberate gestures and classifies them as one action, and holding never repeats. Back is a 0.6 to 1.0 s closure; help is three closures of 1.5 to 4 s, with a dead zone between them.
Adaptation that asks. Each eyebrow attempt is scored against his threshold. If Deepseek analyzes weak signals from eyebrows, it offers one-signal mode and never switches silently.

Fetch.ai uAgents on Agentverse. Four agents run in one uAgents Bureau: the Relay, Maya's, Raj's, and a Concierge. Each has its own mailbox and is discoverable, and the Agent Chat Protocol makes them reachable from ASI:One. A custom residual-care-circle protocol carries typed messages, and the care circle is contract-net style bidding.

Maya and Raj are caregiver uAgents which have the schedules of both the

The rest of the stack.

Messaging: Photon for iMessage and SMS, with idempotency keys so a retry never texts twice, and a signed webhook for replies.
Web: Browser Use drives a real browser and stops one click before anything final.
Voice: ElevenLabs.
Interface: Electron, with a page helper that turns real sites into two-choice lists.
Design: Paper.

## Challenges we ran into The hardest part was making two signals feel natural. Calibrating to one face, and making The hardest part was making two signals feel natural. Calibrating to one face, and making sure a slow camera frame or an accidental long blink can never trigger the wrong action, took the most time. We solved it with per-person levels, a frame-gap guard and tests that cover every closure length. The second challenge was agents talking to agents: language models on the other side reply to everything, so the Relay learned to recognize its own conversation coming back and never act on it twice.

## Accomplishments that we're proud of

A real text to a real phone, a real reply all handled and orchestrated by FetchAi uagents and share the answer on his screen, driven by a blink and an eyebrow raise.
A care circle where each family member's availability stays with their own agent.
An Agentverse interface that adapts to the person, with the user's consent, instead of asking them to adapt to it.
Everything the patient needs in a full day, from "I love you" to the game, just from two comfortable signals.

## What we learned

Comfort is a feature. Gaze is powerful but tiring, and the better choice for long use was the quieter one. We were working on a Kalman and changepoint engine that taught us how to track a failing signal rigorously, but the two-signal interface taught us that comfort and simplicity matter more day to day. We also learned that for something this important, safety logic should stay deterministic and AI should work around it, never in charge of it.

## What's next

We want to put Residual in front of people living with ALS and their speech therapists, and keep shaping it around what they tell us. Next steps are more agents (pharmacies, doctors), and voice banking, so Residual can speak in a person's own voice. It is a prototype, not a medical device, and it was built for someone we love.

I changed only the formatting: headings, bold lead-ins, bullets and code formatting for the identifiers. Every word is yours. The unfinished sentence about Maya and Raj and the DeepSeek lines I flagged are left exactly as you wrote them.