-
-
Answer with a nod, a gesture, or a tap. Your choice becomes Matilda’s voice.
-
Matilda speaks for you when speech is difficult, helping you call the places you rely on.
-
Follow the conversation through live captions, so you always know what’s being said.
-
Go beyond preset replies. Mouth your words or type exactly what you want to say.
Inspiration
Our team came to Hack The Nest wanting to build something meaningful. We briefly considered curing cancer. Two people, 24 hours. We kept brainstorming.
We talked about Muhammad Ali’s experience with Parkinson’s, then Hector Salamanca from the TV show Breaking Bad, who communicates by ringing a bell after losing his speech. That sparked a question: what if a small movement could become a complete spoken response?
The problem extended beyond any one condition. People can lose speech or struggle to communicate because of ALS, Parkinson’s, stroke, or other conditions. We wanted to help them stay part of the conversation.
What it does
Matilda is an iPhone app that uses AI to speak on calls for people who cannot speak clearly or at all.
Family members can set up frequently contacted places, call goals, and answers to common questions. Matilda handles saved factual details and presents choices when a decision is needed. Users respond through nods, thumbs up or down, taps, or silently mouthed words.
Picture calling a pharmacy: Matilda provides your prescription details, asks about delivery, and speaks the answer you choose with a nod. The same interaction can support someone with speech difficulties or someone who is nonspeaking, using whichever input works for them.
How we built it
- AirPods motion sensing recognizes nods and head shakes as confirmation or rejection.
- Computer vision uses Apple Vision to detect hand joints and recognize thumbs up and down.
- Machine learning combines ARKit mouth tracking with Auto-AVSR, MediaPipe, and ONNX Runtime to transcribe silently mouthed words.
- Gemini and ElevenLabs connect question understanding, response suggestions, and conversational speech.
A SwiftUI app, TypeScript backend, and Next.js web demo line coordinate the conversation. The caregiver portal includes MongoDB Atlas and SSOJet integrations.
Challenges we ran into
A listening nod should not accidentally answer a question. We built question-specific input handling, gesture filtering, and a cancellation window before responses are sent.
Lip-reading can also misinterpret words. We use conversational context to prepare a proposed sentence, then let the user review and correct it before Matilda speaks.
We also designed multiple ways to respond because speech loss does not look the same for everyone. A gesture that works for one person might be difficult for another.
Accomplishments that we're proud of
We implemented a conversation workflow connecting head motion, hand gestures, visual speech recognition, and an AI voice.
We also built the less visible pieces that make participation possible: caregiver setup, large answer controls, live captions, a practice screen, response rejection, and call history.
The interaction is built around how someone can communicate, rather than requiring a particular diagnosis.
What we learned
The difficult part is giving someone a comfortable way to decide what the AI says.
Accessibility shaped the whole interaction: how users follow a question, choose an answer, correct a mistake, and take the time they need. Supporting different abilities meant building flexibility into the experience from the beginning.
What's next for Matilda
We want to test Matilda with people who have a range of speech and communication disabilities, including people with ALS, Parkinson’s, stroke-related speech loss, and people who are nonspeaking. Their feedback will help us improve recognition, reduce effort, and understand which inputs work best in different situations.
We also want to support more languages and make setup easier for users and families.
Phone calls are one part of everyday communication. We see Matilda expanding into face-to-face conversations, appointments, shopping, and other situations where someone knows what they want to say but needs another way to say it.
Giving a voice to the voiceless, literally.
Built With
- airpods
- apple
- arkit
- atlas
- auto-avsr
- elevenlabs
- fastify
- gemini
- mediapipe
- mongodb
- motion
- next.js
- onnx
- opencv
- python
- react
- ssojet
- swiftui
- typescript
- vision
- webrtc
- websockets