Cognitive Companion

Cognitive Companion Application
Cognitive Companion Admin Panel
Architecture Diagram

Inspiration

Cognitive decline doesn't have to mean loss of independence. Cognitive Companion acts as a careful, unobtrusive presence in the home, watching for situations where a gentle reminder or nudge might help, without automating away the small acts of daily life that give seniors agency and routine.

Rules are written in natural language and evaluated by on-premise vision and language models, so the system understands context rather than just triggering on rigid conditions.

What it does

A privacy-first AI watchdog for multi-generational households, designed to support seniors facing early cognitive decline while preserving their independence and dignity.

How We Built It

Built a FastAPI gateway that ingests camera and sensor events
Implemented a natural-language rules engine that:
- Filters rules by room and time contexts.
- Runs vision reasoning via Cosmos Reason2.
- Runs logic/decisioning via Gemma 3.
- Pushes notifications and Gemini Live prompts over WebSockets.
Added an event batching/cooldown layer so bursty camera events are grouped before analysis.
Integrated MinIO for media storage and presigned URLs.
Added occupancy and emergency detection using Home Assistant presence sensors, with escalation actions.
Built a Gradio debug console for rules, sensors, logs, translation, and vision testing.
Built a Progressive Web App UI with:
- Live transcripts and audio playback of Gemini Live responses.
- Visual alert images.
- Emergency alerts with “I am okay” / “Need assistance” actions and auto-escalation.
- An audio visualizer + VAD calibration for mic input.
- Deployed on GCP Firebase

Challenges We Ran Into

Real-time audio streaming and playback: converting mic PCM to Gemini Live input and playing back streamed PCM audio reliably with low latency.
Keeping Gemini Live sessions resilient: handling reconnects, keepalive pings, and preserving conversation context across disconnects.
Getting the rule pipeline stable across multiple AI services (Cosmos + Gemma + Gemini Live) while keeping response formats consistent.
Preventing notification spam: adding batching, cooldowns, daily limits, and scheduled rules.
Coordinating multi-channel outputs (PWA UI, alert images, Home Assistant audio, SMS/WhatsApp hooks) without overwhelming the user.

Accomplishments We’re Proud Of

End-to-end, privacy-first pipeline that runs on local infrastructure while still supporting high-quality vision and voice interactions.
Natural-language, context-aware rules with room/time filtering and rate limiting.
A live Tamil (and Tanglish) interaction loop using Gemini Live, with UI transcripts and audio responses.
A polished PWA dashboard for seniors and caregivers, including emergency escalation workflows.
A unified debug console that makes it easy to iterate on rules, sensors, and models.

What We Learned

Real-time, assistive AI must be forgiving: session reconnects, keep-alives, and graceful fallback behaviors are essential.
Contextual rules beat rigid triggers for dignity-preserving assistance.
Multimodal flows are only as good as their weakest link; getting consistent JSON and predictable translations is critical.
Seniors-first UX requires clarity, low-friction actions, and minimal cognitive load.

What’s Next for Cognitive Companion

Expand sensor support (more presence modalities, wearable integration, sleep/health signals).
Improve on-device inference performance and model orchestration to reduce latency further.
Add richer caregiver workflows (summaries, timelines, configurable escalation policies).
Strengthen personalization (preferred language tone, household routines, adaptive thresholds).
Formalize safety and compliance features for clinical/eldercare pilots.