SureSay: nothing happens until you choose

Voice assistants mishear slurred speech, then act anyway. SureSay shows its 3 best guesses as big cards and does nothing until you choose. For people with dysarthria, on Alexa+ and AWS.

Comment

Why it matters

After a stroke, or with Parkinson's, ALS or cerebral palsy, speech can come out slow, soft or slurred. This is dysarthria, and the mind behind it is exactly the same. These are the people who would gain most from running their home by voice, and voice assistants fail them most: on real dysarthric speech, a plain recogniser got a sentence's words exactly right only a third of the time.

Getting it wrong is not the worst part. Acting on it is. "Lock the door" heard as "unlock the door" unlocks it. As Alexa+ gains the power to act, a misheard command that acts is worse than no command at all. So people stop talking to it.

What it does

SureSay is a voice assistant that knows when it isn't sure, and never acts on a guess.

  1. Say it your way, or type it. Recognition is primed with your own phrasebook.
  2. See three guesses. He says "fan on"; SureSay hears "van on". Instead of guessing, it shows its three best guesses as huge numbered cards and reads them aloud. If it isn't confident at all, it says so and asks again.
  3. Choose. Say "two", "the second one", "haan", "do number", or tap, press a key or use one switch. Risky actions (unlock, order, call) ask a second time.
  4. Only then does it happen, and a consent receipt records who confirmed what, when and how.

Around that core:

  • "Don't" never becomes "do." "Don't unlock the door" sounds like "unlock the door", so a negated command offers no cards: SureSay says what it heard and that nothing will change.
  • Talk to someone. Be understood by people, not only machines. He says what he wants to tell the person in front of him, picks the right sentence, and SureSay says it clearly out loud and shows it huge, turned around for someone across the table.
  • Speak for me. One tap on everyday sentences, in English and Hindi.
  • Practice. A few sentences a day and an honest clarity score showing which words came through. Every try teaches SureSay his voice. On TORGO the score follows severity (rank correlation 0.85).
  • Help. One press offers to call or message family, and they get an alert.
  • Timeline for family, with "changes without you: 0".
  • Teach SureSay your own words; it also learns from your confirmations, but never a phrase that already means something else.
  • Your devices: every screen, Alexa link and assistant that can show your cards, each one tap from unpaired.
  • Accessible by design: WCAG 2.2 AA, voice, tap, keyboard or a single switch, three themes, calm mode, TV mode, and Hinglish confirmations.

Try it now, no sign-up: "Try it now" opens a private demo home of your own.

Built for Alexa+

SureSay is a self-hosted MCP server built to the Alexa+ add-on requirements, and also a working Alexa skill today.

  • MCP spec 2025-11-25 over Streamable HTTP, at /mcp on AWS Lambda.
  • Account linking: OAuth 2.1 with PKCE (S256) and Protected Resource Metadata. Linking means entering your SureSay pairing code; unpairing the link ends every token at once.
  • MCP Apps: understand returns ui://suresay/cards.html, so the three cards render on an Echo Show or Fire TV.
  • An add-on package, with icons, carousel, privacy and terms.
  • A classic Alexa skill, paired as the user's own voice device: Alexa reads the cards, and his spoken "the second one" goes back as signed evidence.

A linked Alexa+ client gets exactly the agent's powers: it can understand, read and propose, never confirm.

How we built it

The core rule: the AI can ask, but only the user can approve.

  • Every write tool on the MCP server only creates a pending card set.
  • The set's one-time code goes only to the user's paired device, over a separate HMAC-signed channel, and never into any MCP result.
  • confirm runs the action only if the evidence is the user's own tap, or his own recording transcribed on the server, and it passes 13 named checks: right device, right card set, fresh, unused, clearly chooses a card, action unchanged since shown. The code is then consumed by a single DynamoDB conditional write.

A prompt injection can make the agent ask. It can never make anything happen.

On AWS:

  • Amazon Bedrock AgentCore Runtime hosts a Strands agent, which calls the MCP server over Streamable HTTP.
  • AgentCore Memory keeps per-user phrasebooks and long-term facts.
  • Amazon Nova 2 Lite ranks the three meanings (code ready, waiting on quota).
  • Amazon Nova 2 Sonic is a second speech recogniser, and the real-model agent in the safety tests.
  • Amazon Transcribe streams recognition with per-user custom vocabularies.
  • Amazon Polly reads the cards aloud and speaks for the user.
  • DynamoDB holds the gate ledger with one-time conditional writes; S3 stores private, expiring audio.
  • Lambda runs the hub and the Alexa skill; SNS sends family alerts; CloudWatch carries a safety dashboard.
  • One command deploys and tears everything down.

The web client is React and TypeScript, and confirms through the MCP TypeScript SDK. The safety layer is open source on its own: confirm-gate-mcp (MIT) puts confirm-before-act in front of any MCP server.

Proof

Real dysarthric speech from the TORGO corpus: thresholds tuned on two speakers, results on six others it never saw.

Plain assistant SureSay
Misheard words taken as a "yes" or a choice 62.8% 2.3% (430 decisions)
Acted on a wrong phrase without asking 84.3% 0.0%
A real AI agent (Amazon Nova 2 Sonic): changes nobody confirmed 11 / 35 0 / 35
Adversarial attacks (injections, near-misses, forged confirmations) that changed anything 35 / 35 0 / 50

Every number comes from scripts in eval/ and is copied from eval/REPORT.md, including a section on what doesn't work yet. The worst case is the most severe speakers: SureSay asks them to repeat more often, and its "maybe" cards win most of that back.

Challenges we ran into

  • Model access on a new account. Nova 2 Lite was authorised, but every on-demand quota was 0, and AgentCore started at 0 agents. We built every Nova path against stubs, kept an offline ranker, and measured Nova 2 Sonic, which did run, as both a recogniser and a real agent.
  • MCP SDK 2.x broke the old server code, and Strands pins mcp<2.2: a port and a version dance.
  • Hearing "yes" safely. "Ek second" (one moment) was read as "option 2"; "ok hang on" as a yes. Every one became a failing test, then a fix.
  • Three adversarial security reviews found real bugs, including pairing brute force through the public demo and an Alexa answer landing on a card set the user never heard. All were fixed test-first.
  • Two speech SDKs needing different versions of the same native library, and IAM changes that only took effect after a redeploy.

Accomplishments that we're proud of

  • False yeses on real dysarthric speech fell from 62.8% to 2.3%, and SureSay never acted on a wrong guess.
  • A real model as the agent: 11 unconfirmed changes without the gate, 0 with it.
  • A working, accessible product anyone can try in one click, an Alexa skill, and an Alexa+-ready MCP server.
  • An open-source gate any MCP developer can use, with 355 tests and a strict mode for servers you don't control.
  • More than 450 tests, honest metrics, and a "what doesn't work yet" section.

What we learned

  • For people who are misheard most, the safest answer is often "I'm not sure, which of these?", and that question has to be fast, big and dignified.
  • Safety you can't measure isn't safety: the attack suite changed the design more than any review.
  • An AI that can act needs a second channel for consent. A yes inside the chat can be forged; a code it never sees can't.

What's next

  • A full Alexa+ integration as access opens, with the cards on Echo Show and Fire TV.
  • Per-user thresholds and recognition that adapts from each person's own confirmations.
  • Indian languages beyond Hinglish confirmations.
  • Work with speech therapists and stroke-recovery groups to test with real users. ```

Built with

Type each into the "Built with" field:

python, typescript, react, vite, model-context-protocol, mcp-apps, strands-agents, amazon-bedrock, amazon-bedrock-agentcore, amazon-nova, amazon-transcribe, amazon-polly, amazon-dynamodb, amazon-s3, aws-lambda, amazon-sns, amazon-cloudwatch, aws-iam, alexa-skills-kit, oauth2, starlette, playwright, whisper

"Try it out" links

https://fig5s4lcztnov5vd3us22pnmoi0srphu.lambda-url.us-east-1.on.aws
https://github.com/Prashant-thakur77/SureSay
https://github.com/Prashant-thakur77/confirm-gate-mcp

Built With

Share this project:

Updates

Submission history