Kira - AI Agent for the Speech and Hearing Impaired

Kira gives deaf and mute users a real voice — reading signs, classifying sounds, detecting danger, and speaking for them via Amazon Nova Lite, Nova Sonic, and Amazon Polly. Built for 70M+ Africans.

Comment

Kira - an AI Agent for the Speech and Hearing Impaired with Amazon Nova.

Inspiration

I was standing in a pharmacy in Ibadan when I saw it happen.

A man, maybe 35, wearing a faded blue kaftan, walked up to the counter holding a folded piece of paper. He handed it to the pharmacist. She read it, shook her head, and pointed to something on the shelf that was not what he needed. He pointed back at the paper. She called a colleague. The colleague shrugged. A small queue formed behind him. Someone sighed loudly. The man looked at the floor.

He left without his medicine.

He was deaf. And in that moment, with the entire might of modern technology sitting in everyone's pockets, not a single tool could bridge the gap between him and a pharmacist three feet away.

I know what it looks like when a system was not built for you. I have seen it in many forms, in many places. But what I saw in that pharmacy stayed with me differently. Because this was not a resource problem or an infrastructure gap. This was a human being standing in a room full of people who simply could not make himself heard.

There are over 70 million people globally who are deaf or hard of hearing, with millions more who have speech impairments. The vast majority of AI tools being built right now were designed with none of them in mind. When the Amazon Nova AI Hackathon opened with its focus on multimodal AI, voice AI, and agentic reasoning, I saw an exact match between what Nova can do and what these communities desperately need.

I built Kira for that man in the pharmacy. And for everyone like him.

What It Does

Kira is an AI Agent for people who are deaf or have speech impairments. Its companion is Bunnie 🐰, a warm, always-present AI character that becomes the user's voice, their ears, and their safety net.

My Voice is where Kira speaks for you. Type a message and Bunnie naturalises it into flowing, human speech before reading it aloud through the Nova Sonic and Amazon Polly neural voice pipeline. Or show a sign to the camera. Nova multimodal reads the gesture and adds the word to a sentence builder. For the first time, someone who has never been able to speak can walk into a shop, a hospital, or an office and have a real conversation.

My Ears is where Kira listens for you. People with hearing impairments cannot hear a fire alarm, someone calling their name, a car horn, or a baby crying. My Ears runs a continuous sound monitor. When something significant happens, an alert fires instantly then Bunnie enriches it in the background: transcribing nearby speech, detecting if the user's name was called, and specifically identifying dangerous sounds like fire alarms or sirens. For high-urgency sounds, Bunnie does not stop at telling the user. She acts.

My World is where Kira sees for you. Point your camera at anything, a document, a scene, a person's expression, and Bunnie describes what she sees. Safety score, emotional context, who is present, what is happening. She reads it aloud in Joanna's voice.

My Safety is where Kira protects you. Register your emergency contacts. In a crisis, hold the SOS button for two seconds. Bunnie locates you, writes a personalised emergency message, and sends it to every contact at once. No typing. No navigating menus. Two seconds.

Talk is where Kira converses for you. A two-way conversation assistant: you type what you want to say, Bunnie speaks it aloud. The other person talks, and you see their words as text. Bunnie suggests three contextual quick replies so you always know what to say next.

Caregiver Dashboard is where Kira connects your support network. Every day, Bunnie generates a summary for caregivers, family members, and support workers: what the user did, how independent they were, any alerts that fired, a personalised wellness tip. This screen makes Kira useful not just for the primary user but for everyone around them, turning a personal tool into a shared care system. A caregiver in a different city can stay connected to someone they love without surveillance, just understanding.

And then there is BunnieAgent, the part that takes Kira beyond a simple tool.

When My Ears detects a danger sound with urgency 7 or above on our 10-point scale, BunnieAgent activates. Four Nova reasoning steps fire in sequence, automatically:

  1. Nova assesses the threat. Is this a genuine emergency? What is most likely happening?
  2. Nova composes the SOS. A personalised message, under 160 characters, that names the user and explains they cannot call for help.
  3. Nova writes a caregiver notification. Separate, warmer, specific: what happened and what the caregiver should do.
  4. SMS is dispatched to all emergency contacts. Done.

Someone who cannot hear or speak and is in an emergency cannot navigate menus. Cannot call 999. BunnieAgent exists because in that moment, the AI must act, not wait to be prompted.

How We Built It

The core stack is React Native on Android, talking to AWS Bedrock via a hand-rolled AWS Signature Version 4 implementation in JavaScript. No SDK. Every Nova call goes through HMAC-SHA256 signing built from scratch: canonical request, credential scope, signing key derivation. It took time to get right. One wrong byte in the signing chain returns a silent 403 with no explanation. But the result is a zero-dependency, fully portable signing implementation that runs in any JS environment.

The voice pipeline works like this: text travels to our Nova Sonic proxy server (Node.js, deployed on Render), which opens a bidirectional session with Amazon Nova 2 Sonic for text processing and enrichment, then hands the output to Amazon Polly for neural speech synthesis. The MP3 returns to the phone where react-native-video plays it. The voice users hear is Joanna, a warm and natural neural voice that sounds nothing like robotic TTS.

BunnieAgent is a pure JavaScript async pipeline. Each step feeds its output into the next. The full chain-of-thought log is stored in the app so you can see exactly what Nova reasoned at each step. That transparency matters for debugging and it matters for trust.

State lives in React Context with useReducer. No Redux, no Zustand, just plain React. Bunnie's animations use counter-based triggers so the same animation can re-fire on repeated events without getting stuck.

Challenges We Ran Into

Audio without amplitude. react-native-audio-record does not give you a volume number. It gives raw base64 audio chunks. We built a proxy: chunk length correlates with volume because loud audio compresses less. It is unorthodox but it works on a real device in a real room.

Signature V4 from scratch. Every field in the canonical request is exact. The signing key is derived from four successive HMAC rounds. We hit silent 403s for hours. But the result is a signing implementation we fully understand and fully control.

Name detection for non-English names. Standard phoneme matching does not understand the tonal quality of many names from Africa, Asia, and other regions. We fixed it two ways: fuzzy prefix matching on the first three characters, and a prompt that explicitly tells Nova to consider regional name pronunciations.

Screen freezing during audio listening. The JS thread was drowning in state updates. The fix: throttle wave animation to 200ms intervals, process audio every 5 seconds, and yield to the UI thread before any Nova enrichment call.

Nova Sonic architecture. Nova 2 Sonic uses InvokeModelWithBidirectionalStreamCommand, a binary bidirectional streaming protocol that required building a full proxy server. The deployed server represents real infrastructure that demonstrates how Nova Sonic can be integrated into mobile accessibility applications.

Accomplishments We Are Proud Of

Eleven distinct Nova touchpoints across four Nova products in a single, coherent, genuinely useful app. Not eleven features bolted together. Eleven moments where AI makes life meaningfully better for someone who has been left behind by every previous wave of technology.

A 4-step agentic emergency pipeline that acts without being asked.

A Nova Sonic WebSocket proxy server, fully deployed and processing real sessions.

Amazon Polly neural voice output that sounds genuinely human, personalised across six voice personalities.

Built and tested on a real physical Android device. Not a simulator.

An app built for people who did not have one.

What We Learned

Nova's multimodal capabilities are stronger than we expected for accessibility use cases. Sign recognition from a still photo is not perfect but it is good enough. And when the alternative is silence, good enough is transformative.

The most important design lesson: people who cannot hear a loading sound need every wait to be visible. Every critical alert must be instant. That is why Kira uses the instant-local-plus-background-enrichment architecture: a fast local result fires immediately, and Nova makes it smarter a moment later. This is the right model for accessibility AI on mobile. Probably for most AI on mobile.

Agentic AI is most valuable when it removes friction at the worst possible moment. BunnieAgent does not ask for confirmation. It acts. That is not a bug in the UX. It is the entire point.

What's Next for Kira

Nova Act to automate emergency service web forms, because people who cannot speak also cannot call emergency services.

Real-time video sign language, with frame extraction and sequential Nova multimodal analysis.

Nova multimodal embeddings to build a personal sign vocabulary, so users can teach Kira their own signs.

A companion caregiver app sharing the same Nova-generated logs.

Support for more languages including Yoruba, Hausa, Igbo, Swahili, Spanish, Mandarin and other major languages spoken by communities with significant deaf and hard-of-hearing populations globally.

A formal partnership with deaf advocacy organisations for real user testing and iteration.

Offline mode. Because the people who most need Kira are often in the places with the worst connectivity.

The man in the pharmacy deserves better than what exists today. Kira is the beginning of giving it to him.

Built With

React Native 0.73 (Android), Amazon Nova Lite (amazon.nova-lite-v1:0), Amazon Nova 2 Sonic (amazon.nova-2-sonic-v1:0), Amazon Polly (neural TTS, Joanna voice), Amazon Bedrock (us-east-1), AWS IAM, AWS Signature Version 4 (custom JS implementation), Node.js WebSocket proxy server (Render.com), react-native-video, react-native-fs, react-native-audio-record, react-native-tts, react-native-image-picker, react-native-geolocation-service, react-native-sms, React Navigation 6, React Context with useReducer, react-native-linear-gradient, crypto-js, JavaScript ES2022

Built With

  • amazon-bedrock
  • amazon-nova-2-sonic
  • amazon-nova-lite
  • amazon-polly
  • aws-signature-version-4
  • react-native-0.73
  • react-native-video
  • websocket-proxy-server
Share this project:

Updates