Cadence

A real-time communication companion that helps people with ALS join fast conversations in their own voice, with ready replies, control, and one-tap speech.

Comment

Inspiration

A conversation can move at 100 to 140 words per minute, while many AAC users communicate far more slowly. The problem is not only typing speed. It is timing, personality, repair after a misunderstanding, and the ability to start a topic instead of always reacting.

Cadence is a real-time communication companion for people with ALS and other AAC users. It listens to a conversation, prepares a small set of possible replies in the user's own style, and keeps the user in control of every word that is spoken.

We were inspired by a simple question: what if someone could join a fast dinner conversation, make a joke, say “I love you,” hold the floor, or start a topic of their own before the moment moves on?

What it does

Cadence turns a live conversation into ready-to-review communication choices.

  • Live captions: Uses the browser's Speech Recognition API for no-cost, on-device live transcription.
  • Prepared replies: Uses GPT-5.6 with structured JSON output to create diverse reply options grounded in the latest conversation, the user's saved voice style, personal details, and local memory.
  • User control: A person can speak a suggestion, edit it, make it shorter, ask for something more like them, save it, reject it, or mark the context as wrong.
  • Initiation, not only reaction: “Start something” prepares conversation openers so users can bring up a story, concern, question, affection, or news on their own terms.
  • Fast communication boards: Quick reactions, feelings, care needs, saved phrases, and “Hold the floor” are always easy to reach.
  • Offline resilience: Needs, feelings, saved replies, phrase boards, and instant device voice remain useful even when online AI is unavailable.
  • Voice options: Users can choose OpenAI text-to-speech or instant device speech, with tone-aware delivery.
  • Accessible input: Large tap targets, keyboard support, screen-reader labels, scanning mode, and experimental local eye-gaze focus support different access needs.
  • Privacy-first local memory: Voice style, personal details, preferences, remembered people and topics, session data, phrases, and access settings are stored locally on the device. Memory is visible and clearable.
  • Safety controls: Cadence never speaks automatically from a model suggestion or gaze focus. The person confirms with touch, keyboard, switch scanning, or Select.

Why Cadence is different

Many communication tools help compose text faster. Cadence focuses on participation in the whole conversational turn:

  1. Be ready before the moment passes.
  2. Choose language that sounds like the person.
  3. Repair a wrong suggestion or misunderstanding.
  4. Hold the floor when more time is needed.
  5. Initiate something meaningful, not only react.

The goal is not simply faster typing. It is timing, choice, and connection.

How we built it

Cadence is built with:

  • Next.js 14 App Router and TypeScript
  • Tailwind CSS for a responsive, tablet-first interface
  • OpenAI Responses API with gpt-5.6-luna, low reasoning effort, lean prompts, and structured JSON outputs
  • OpenAI TTS for server-side speech synthesis with configurable voice and tone instructions
  • Browser Web Speech API for live captions
  • MediaPipe Face Landmarker for experimental, local eye-gaze focus
  • localStorage for style cards, personal details, phrase boards, remembered people/topics, sessions, access preferences, and optional local debug recordings
  • Upstash Redis for distributed per-IP rate limiting in real mode
  • Vercel for deployment, analytics, and production hosting

The language layer is intentionally separated behind interfaces and API routes:

  • predict() creates grounded candidate replies
  • expand() turns a short idea into full replies
  • toneAdjust() rewrites a message in warm, firm, or funny tone
  • initiate() prepares conversation starters
  • speak() supports OpenAI speech or instant device voice
  • transcribe() supports browser captions

This makes the system replaceable and testable. MOCK_MODE=1 runs offline mock behavior without model or speech costs. MOCK_MODE=0 enables the protected real AI path.

Privacy and safety by design

Cadence is an early prototype, not a medical device or emergency service.

We designed for meaningful control:

  • No account is required.
  • Personal voice data, details, memory, phrases, and sessions stay on the device.
  • Camera video and gaze samples are processed locally for the experimental eye-gaze feature and are not uploaded or stored by Cadence.
  • AI receives only the limited conversation context needed to prepare a reply.
  • The server validates input size, requires explicit real-mode consent, rate-limits requests per IP, and keeps API keys server-side.
  • Eye gaze is an experimental focus method, not a general mouse replacement. It never triggers speech automatically.
  • Care phrases such as “I need suction” do not alert caregivers or emergency services. They only speak the selected phrase.

Challenges we ran into

The hardest challenge was balancing speed with control.

A system that reacts too slowly misses the conversational moment. A system that reacts too early can misunderstand an unfinished sentence. We addressed this with speculative prediction from interim captions, a short final-caption debounce, cancellation of outdated requests, and clear “preparing replies” states.

We also learned that reliability matters more than a flashy demo. When real AI, network access, or TTS is unavailable, Cadence does not leave the user without communication. It keeps quick needs, feelings, saved replies, phrase boards, hold-the-floor phrases, and device voice available.

Eye gaze was another important lesson. Browser camera tracking is sensitive to lighting, camera position, head movement, calibration, and device performance. We keep it clearly labeled as experimental, local, confirmation-based, and optional.

Research that shaped Cadence

Cadence is informed by publicly available AAC, ALS, accessibility, and communication-participation research.

  • AAC users can face high abandonment rates because of training burden, fatigue, partner burden, and effort. Cadence reduces friction with one-tap prepared replies, scanning, and no required partner app.
  • Communication partner instruction can improve AAC communication. Cadence does not require a partner to install anything, while still offering a concise optional partner tip.
  • AAC design must account for cultural and linguistic context. Cadence supports user-controlled language preferences and a “do not translate or normalize my wording” preference.
  • Recent AAC UI co-design research emphasizes that researcher-only design misses real access barriers. Cadence includes a voluntary tester feedback process and treats participatory co-design as an ongoing requirement.
  • Research on communicative participation in ALS supports measuring more than speed. Cadence tracks replies spoken, topics initiated, time-to-response, repairs, rejected suggestions, and whether a reply sounded like the user.
  • Voice banking and voice identity research informed our emphasis on user choice, device voice fallback, tone, and configurable speaking voices.

Selected sources:

These sources guide our product decisions. They do not prove clinical effectiveness of this prototype.

Accomplishments that we are proud of

  • Built a complete live conversation flow from caption to prepared reply to confirmed speech.
  • Made initiation a first-class feature, so users can lead conversations rather than only respond.
  • Kept user agency central: AI suggests, the person chooses.
  • Built offline-safe communication tools instead of depending completely on cloud AI.
  • Added meaningful access options: touch, keyboard, scanning, instant voice, and experimental local eye-gaze focus.
  • Added local-first personalization, memory, needs, feelings, phrase boards, and session recovery.
  • Added production safeguards including structured outputs, consent checks, input caps, CSP, rate limiting, and server-side secret handling.
  • Built a responsive experience for desktop, tablet, and mobile use.
  • Added unit, integration, and end-to-end coverage for core flows including onboarding, local sessions, offline fallback, rate limits, privacy, scanning, and input validation.

What we learned

We learned that a strong AAC experience cannot be reduced to a text box and a faster model.

The most important design decision was making Cadence a system for conversation participation, not just message generation. The person needs to be ready, recognized, able to correct the system, able to express care or urgency, and able to decide when not to speak.

We also learned that accessibility features must be designed with failure states in mind. A tool is more trustworthy when it explains uncertainty, keeps a clear off switch, and still offers communication when a network or model call fails.

What is next for Cadence

Our next priority is co-design with people who use AAC, people living with ALS, caregivers, speech-language pathologists, and assistive-technology professionals. We want to learn what feels useful, inaccurate, tiring, or missing in real conversations.

We plan to improve reply relevance, voice match, personal profiles for family, care, work, and medical settings, editable captions, custom names and vocabulary, multilingual support, and clearer confidence when listening is uncertain.

We will continue strengthening access through Bluetooth switches, configurable scanning, and experimental target-based eye-gaze focus with better calibration and validation. Eye gaze will remain clearly labeled as experimental until it is tested reliably with users.

We also plan to expand offline communication boards, saved replies, care phrases, conversation repair, and user-controlled session history, so communication remains available even when AI or network access is unavailable.

Most importantly, we will measure participation, not only taps saved: meaningful replies, topics initiated, repairs completed, fatigue, time-to-response, and whether the user feels the words sound like them.

Cadence is an early prototype, not a medical or emergency tool. Our goal is to build responsibly with the community, so more people can be present in the conversations that matter, in words that remain their own.

We will use feedback to improve:

  • Reply relevance, voice match, and rejection/repair behavior
  • Partner-tip content and communication setup
  • Multilingual and code-switching support
  • Per-person and per-setting communication profiles
  • Target-based eye-gaze reliability, calibration, fatigue, and validation
  • Accessibility for switches, eye-gaze hardware, and assistive browsers
  • Outcome measures centered on participation, autonomy, fatigue, satisfaction, and “this sounded like me”

Cadence is early, but the direction is clear: help more people be present in the conversations that matter, in words that remain their own.

Built With

  • aac
  • accessibility
  • als
  • gpt-5.6
  • mediapipe
  • next.js
  • openai-api
  • openai-tts
  • playwright
  • pwa
  • tailwind-css
  • typescript
  • upstash-redis
  • vercel
  • web-speech-api
Share this project:

Updates