Say It Your Way: Alexa+ for people with broken speech

About the project Inspiration

People recovering from a stroke, living with Parkinson's, aphasia or a stammer often speak slowly, in pieces, and correct themselves halfway. Voice assistants are built for one clean sentence: a pause ends the request, the pieces are lost, and the person has to start over. The people who need a voice assistant the most are the ones it understands the least.

What it does

Speech-Resilient Alexa is an Alexa+ add-on, built as a self-hosted MCP server.

Keeps every piece. "Tomorrow…", then "doctor…", then "ten" become one request: a reminder for the doctor tomorrow at 10:00. Corrections like "no, Mom" change only that part. Asks only for what's missing, and never acts without a clear yes. Remembers across sessions. Close the app halfway, and next time Alexa offers to finish the request instead of making the person repeat it. Older history is summarized so the brain stays fast. Knows a new request from a correction. "Call Mom" in the middle of a reminder makes Alexa ask before switching, so nothing gets merged or lost. Emergency SOS. "I fell", "help help" or "emergency" start a 5-second countdown. "Stop" cancels it; silence sends it, so someone who fell and can't speak still gets help. Their helpers get a Telegram message with the street address from Amazon Location and a map pin. Talks back in a natural Amazon Polly voice, and listens through Amazon Transcribe. How we built it

An MCP server in Python with FastMCP, over Streamable HTTP, MCP spec 2025-11-25. Five tools: understand_speech, do_confirmed_request, cancel_request, resume_unfinished, emergency_sos. An LLM brain (Groq; Amazon Bedrock wired in) with structured output, plus fixed word lists checked first, so yes, no, SOS and stop work even when the LLM is down. Memory in SQLite: every turn, the request in progress, a rolling summary, an action log, and the people who get emergency alerts. AWS: Amazon Transcribe streaming for speech in, Amazon Polly neural voice for speech out, Amazon Location ReverseGeocode for the alert's address. All behind the server, so AWS keys never reach the browser. A web page that simulates the Alexa+ host: it streams the mic, sends one turn per pause, speaks the replies, and runs the SOS countdown. Telegram Bot API for alerts: a helper links themselves by opening a link and pressing Start. 156 Python tests and 34 web tests, all running against the real server over HTTP, with a recorded mock brain, a fake Telegram and a fake Amazon Location, so they need no keys or network. Challenges we ran into

Telling a correction from a new request. With broken speech, "no, Mom" and "call Mom" sound alike. We stopped guessing: Alexa asks before switching. The microphone hearing Alexa. We pause the mic while she talks, and start the SOS countdown only after she finishes, otherwise there's no time to say "stop". Amazon Bedrock stayed closed to our account (see the friction log), so we moved the brain to Groq's free tier and kept Bedrock as a provider in the code. SMS to Egypt needs a registered sender ID, so emergency alerts go through Telegram, behind one notify() so SMS can be added later. Accomplishments that we're proud of

A request said in five broken pieces, across a restart, still ends up as one correct reminder. The SOS works with no LLM at all, and the alert reaches a real phone with a real address. What we learned

Speech accessibility is mostly about patience and memory, not a better speech model. Asking one more question is cheaper than one wrong action. What's next

Usual requests: learn what each person asks for most, so "c… Mom" becomes "Call Mom?" faster. A per-person Transcribe custom vocabulary (names, corrections). Yes/no mode after a few failed tries, and noticing frustration with Amazon Comprehend. Running on a real Alexa+ device.