In My Words

Inspiration

In KW Habilitation's Inclusive Living homes, some residents don't communicate verbally. When a new staff member starts a shift, almost everything they know about that person comes from a paper binder that might be years out of date. When we talked with KW Habilitation, three things stood out: there are no AAC tools in use today, onboarding means reading a binder and shadowing for two weeks, and residents end up having their story told for them, over and over, by other people.

We wanted to flip that. The resident should tell their story once, in their own voice, and make their own choices.

What it does

In My Words is an AI-assisted communication app with three users in mind: residents, staff and managers.

Speak For Me (AAC): residents tap photo tiles to build a thought ("coffee" + "now"). AI suggests full sentences based on what it knows about that person ("Can I get a double-double from Tim's?"). The resident approves or rejects, and only then is it spoken out loud in a voice they chose.
My Story: a picture-based interview builds the resident's life story in chapters, narrated in their own voice. Told once, heard by every new staff member.
Onboarding Coach: replaces "read the binder." New staff ask questions like "What upsets Marcus?" and every answer shows where it came from: in Marcus's words, a staff observation, or the home's records.
Shift Handoff: a "since your last shift" summary of what happened, what didn't, and why.
Supported Choice: options shown as pictures with simple consequences. Logged as "Marcus chose" or "not decided yet", never "staff decided".
Memory layer: the app learns routines and preferences from everyday use. Anything it figures out is labelled "pattern noticed" until the resident confirms it.
Actions: when a resident wants something that can be done online, an agent prepares it (book an activity, add it to the calendar) behind a picture approval card. Anything that needs a person physically there becomes a staff task instead.

How we built it

Frontend: Next.js and React, designed tablet-first with large tap targets
AI: Gemini powers the agents: sentence suggestions, the Coach, story chapters, choices and action planning
Voice: ElevenLabs for resident voices, story narration and the voice agents
Data: Tiger Data (Postgres + pgvector) stores profiles, stories, choices and memories, with vector + keyword + recency search for the memory layer
MCP: four MCP servers (resident profile, memory, actions, and a mock of the home's records system). Agents only reach data through these servers, so the mock records server can be swapped for the real one without changing any agent code
Media: Cloudinary turns staff phone photos into consistent, accessible picture tiles
Deploy: the whole stack runs on a Vultr VM in Toronto over HTTPS, with CI auto-deploying every green push

Challenges we ran into

AI that helps without speaking over someone. Our rule: nothing is spoken, saved as the resident's words, or acted on without the resident's approval.
Keeping a profile honest. We separated "in the resident's words" from staff observations and AI-noticed patterns, so you can always tell who said what.
Building four MCP servers, a memory layer and voice agents in 24 hours, with zero sleep and one 6am pivot.

Accomplishments that we're proud of

Personalized sentence suggestions that sound like the person, not a generic AAC app
A memory layer that keeps the profile current instead of going stale like the binder
A clean architecture where every agent shares one data layer through MCP

What we learned

Supported decision-making changed how we designed everything. KW Habilitation told us well-meaning staff sometimes take decisions away from residents, so we built a system with no "staff decided" option at all.