Our patient mobile view!
the skills marketplace for your agent
Patient Eye Gaze only mode!
Home page -
Clinician view
Docotor portal, log data per patient

aide 🫶

Independence at the speed you used to have.

Inspiration

We all use the internet every day. Booking a doctor, appealing an insurance denial, refilling a prescription. It's friction, but it's our friction, and we move through it at our own velocity.

ALS takes that velocity away one system at a time. About 32,000 Americans live with ALS today (9.68 per 100,000, per an ALS Association co-authored study) and 300,000+ people worldwide, with US prevalence projected to hit 11.21 per 100,000 by 2040. Up to 95% lose the ability to speak. For bulbar-onset ALS, functional speech is lost after a median of 23 months from symptom onset and intelligibility drops below 85% by 32 months (Eshghi et al., Scientific Reports, 2022). Speech can decline as fast as 20% per year.

And nobody designs for what comes next: every login, every liveness check, every "say your passphrase," every button smaller than a fingertip assumes a stable adult who can speak, see, and tap. People with ALS fail those checks at exactly the moment the stakes are highest. The fallback today is a caregiver logging in as them, which means losing privacy, protection, and a piece of yourself.

When we talked to the accessibility community, losing a voice was devastating, but losing the ability to handle your own life was what people described as losing themselves. That's the problem we built for.

What it does

aide is a companion app for people with ALS, built on three commitments: verify, understand, act.

🔐 Verify. Every user is a Persona-verified human. Clinicians go through a separate Persona flow, and community membership requires a clinician-confirmed diagnosis, so the community has no bots, trolls, or impostors. Every agent action is tied to your verified identity and scoped like an access control.

🎙️ Understand. Once a week, an ElevenLabs conversational agent checks in on how you're doing. Those conversations, plus every other voice interaction in the app, are stored and analyzed for recognition confidence, speaking rate, pauses, filler words, and response latency against your own rolling baseline. Motor signals (target time, miss rate, jitter) and vision signals (zoom behavior, gaze stability) come from ordinary use. Together they produce progression values clinicians can follow between visits instead of waiting for the next ALSFRS-R.

📐 Adapt. As motor control declines, buttons grow and spread out. As vision declines, type scales up. We default to light mode because a brighter screen constricts the pupil, which measurably improves webcam eye tracking, and eye tracking is a first-class input, not a fallback. The interface never gets harder. It gets easier.

🗣️ Preserve. Every check-in quietly builds a voice bank. With ElevenLabs voice cloning, aide reconstructs your voice from your own recordings so that when speech goes, your voice doesn't.

⚡ Act. A self-hosted personal agent completes real tasks: forms, calls, refills, appeals, scheduling. A community skills marketplace lets users share workflows (Medicare enrollment, a power wheelchair appeal) so one person's hard-won process becomes everyone's shortcut.

🧑‍⚕️ Connect. A verified ALS-only community, and a clinician view that receives progression trends with the patient's consent.

How we built it

Web (Next.js): App Router with server components, custom GLSL fragment shaders via WebGL for ambient and progress visuals, and a typography system built on HCI principles: 44px minimum targets scaling past 88px, WCAG-AAA contrast, and fluid type driven by a capability profile instead of viewport width.

Native (React Native): Expo-managed with Swift modules bridged for camera, low-latency audio capture, and haptics, so eye tracking and recording behave consistently across iOS devices.

Backend (FastAPI, Python): Async services for auth, audio ingestion, metrics, and the agent gateway. Postgres for structured data, encrypted object storage for audio, and a job queue for ASR and feature extraction.

Identity (Persona): Hosted-flow inquiries with reference IDs so each user maps to one Persona account, a separate clinician template, and inquiry.completed webhooks driving account state. Agent tool permissions are scoped per verified identity and every privileged action is logged against who authorized it.

Voice (ElevenLabs): Conversational Agents for the weekly check-in with a server tool that writes structured data back to our API, voice cloning over stored check-in audio for voice banking, and TTS in the user's banked voice.

Speech pipeline: audio → ASR with per-token confidence → features (WPM, pause ratio, mean pause length, filler rate, response latency, confidence mean/variance) → per-user rolling baseline → EWMA and z-scored drift → progression score, with every score linked to the recording that produced it.

Eye and motor tracking: Webcam gaze estimation with per-session calibration and dwell-to-click; fixation stability feeds the vision signal. Every tap is instrumented for time-to-target, miss rate, and jitter, modeled against Fitts's law to separate "slower" from "less precise."

Adaptive UI engine: Degradation signals resolve to a capability profile (motor, vision, speech) that maps to design tokens (target size, spacing, type scale, contrast, input modality) consumed by every component. Adaptation is continuous and reversible.

Agent (self-hosted Hermes): Behind our own gateway with a tool registry, per-tool permission scopes, human-in-the-loop confirmation for irreversible actions, and full audit logs. Community skills are versioned workflow definitions the agent loads on demand.

Security: Health data encrypted at rest, access scoped to the verified identity, and the agent receives only the minimum context each task requires. We framed the "aid me" features as access controls first, the way HIPAA thinks about them.

Challenges we ran into

Eye tracking is unforgiving. Gaze estimation drifts with head movement and lighting. Per-session calibration and light mode were the difference between usable and useless.
Speech metrics are noisy across microphones. Raw ASR confidence varies by device, so everything had to anchor to each user's own baseline instead of absolute thresholds.
Agents need guardrails before features. Giving an agent health-adjacent tasks forced us to design permission scopes and confirmation gates first.
Tone. ALS is terminal. Every screen and sentence had to communicate capability, not decline.
⏱Latency. Voice check-ins only feel human under a few hundred milliseconds of turn latency, so we tuned the agent and our tool round-trips to stay there.
Sandbox realities. Adaptive liveness for late-stage patients doesn't exist in Persona yet, so we documented it as the collaboration we want rather than pretending it away.

Accomplishments that we're proud of

A real end-to-end pipeline: Persona verification → weekly voice check-in → speech, motor, and vision metrics → capability profile → adaptive UI → agent action → clinician sharing.
An interface that visibly gets easier as the user's abilities change, shown across three capability stages.
Speech progression tracking from ordinary conversation, with every number tied to the audio behind it.
A community where authenticity comes from identity and diagnosis, not moderation after the fact.
Voice banking that happens as a side effect of being cared for, not as homework.

What we learned

Building agents is mostly building trust. The hard part was never the tool call; it was deciding what the agent must confirm, log, and refuse.
Eye tracking is a physiology problem before it's a software problem. Pupil size and lighting mattered more than any model choice.
Accessibility is not a font setting. Talking to the community pushed us to find a true problem, loss of independence, instead of shipping a feature list.
In healthcare, security and privacy are the product. Nobody will put their voice, their decline, and their finances into an app that doesn't take identity seriously. Persona is why the community and the agent can be trusted at all.
Authenticity requires proof. An online community for a vulnerable population can't be open by default. Clinician-verified membership is a feature, not a barrier.

What's next for aide

Beyond touch and gaze: tongue-based input, non-invasive EMG/EEG, and eventually brain-computer interfaces so the agent can be driven with no movement at all.
Accessible verification with Persona: liveness and re-verification modes that adapt to what a patient can still do, so identity is never another thing ALS takes away.
Clinical integration: FHIR-based progression sharing into neurology clinics and ALS multidisciplinary teams.
More conditions: Parkinson's, MS, and stroke aphasia share the same need for an interface and an identity that adapt to a changing body.
More voices: growing the skills marketplace from the workflows patients and caregivers already know by heart.