CommuniCare

Inspiration

Across the world, millions of nonverbal children, stroke recovery patients, and individuals on the autism spectrum rely on Augmentative and Alternative Communication (AAC) picture boards to express their daily needs and emotions. However, for family caregivers, special education teachers, and speech-language pathologists (SLPs), making customized visual boards by hand is a mentally exhausting chore. Searching symbol libraries, resizing tiles, and color-coding clinical grammar takes 15 to 30 minutes every single morning. We built CommuniCare to automate this multi-step clinical workflow end-to-end using Google's newest agent technologies.

What it does

CommuniCare is an autonomous assistive intelligence agent that turns natural spoken or written caregiver messages into high-contrast AAC picture symbol boards in under one second.

Autonomous 5-Step Agent Pipeline: When a caregiver enters a message, the agent retrieves the recipient's profile from Google Cloud Firestore, uses Google Gemini 3.5 Flash to extract core concepts, queries the open ARASAAC catalog for pictograms, formats the board according to the clinical Fitzgerald Key color standard, and updates recipient memory.
Interactive Speech Synthesis: Tapping any card reads the word aloud. The Speak All button vocalizes the entire plain-language instruction without clipping.
Voice Tone Studio: Caregivers can customize speech audio across sensory-friendly personas (Child Friendly, Female Adult, Male Adult, and Gentle & Calm).
Multi-Turn Adaptive Learning: When a caregiver reinforces a card (Worked Well), the agent updates memory in Firestore. On subsequent routines, Gemini 3.5 Flash automatically prioritizes the individual's learned symbols.
Clinical & Offline Outputs: Features 1-click formatted physical Print Sheets for communication binders and distraction-free Presentation Mode for iPads and tablets

How we built it

Google GenAI SDK (google-genai): The primary Agent framework driving the 5-step autonomous pipeline and enforcing strict Pydantic JSON schemas.
Google Gemini 3.5 Flash (gemini-3.5-flash): Deconstructs messy caregiver instructions, extracts grammatical intent, and simplifies speech into clinical plain language.
Google Cloud Firestore: Provides multi-tenant persistent memory for care recipient profiles, vocabulary ceilings, and reinforcement feedback across sessions.
Google Cloud Run & Cloud Build: Serverless containerized deployment architecture (Dockerfile, cloudbuild.yaml).
Google Identity Services: 1-Click Google OAuth 2.0 and Google Authenticator 2-Factor Authentication (TOTP 2FA).
Clinical Standard: Fitzgerald Key color grammar (Yellow for People, Green for Actions, Orange for Things and Food, Blue for Descriptors, Pink for Feelings, Purple for Time).
Symbol Catalog: ARASAAC Open Pictograms (CC BY-NC-SA 3.0).

Challenges we ran into

Browser Speech Synthesis Race Conditions: Early browser TTS engines were dropping long sentences mid-speech due to garbage collection. We engineered an asynchronous queue sequencer with active audio utterance tracking that guarantees 100% of cards and sentences are spoken completely.

Clinical Grammar Alignment: Mapping dynamic natural language concepts to standardized AAC color categories required crafting strict multi-class system prompts with Pydantic schema validation.
Google Billing

Accomplishments that we're proud of

Executing a full 5-step clinical reasoning and symbol resolution pipeline in under 800 milliseconds.
Creating an interface that is 100% responsive from 320px mobile phones up to 4K displays.
Maintaining a 100% green automated test suite (24/24 unit and integration tests passing).

What we learned

How to leverage the Google GenAI SDK for structured schema outputs with Gemini 3.5 Flash.
The profound importance of adaptive memory in assistive technology and how learning a user's favorite symbols over time builds familiarity and trust.

What's next for CommuniCare

Multimodal Vision Input: Allowing caregivers to snap a photo of a physical scene (such as a breakfast table or classroom) and having Gemini automatically generate matching AAC communication cards.
Multilingual Translation: Generating dual-language boards (such as English + Spanish / French / Arabic) for bilingual homes and international clinical settings.