NeuroDecode AI

A real-time, multimodal AI copilot providing live voice guidance to help caregivers or parents navigate autistic sensory meltdowns.

Comment

Inspiration

Watching a caregiver navigate a child's autistic sensory meltdown is heartbreaking and intense. In those high-stress moments, parents are entirely focused on keeping their child safe—they simply don't have the hands or the mental bandwidth to type symptoms into a text-based chatbot. We realized that traditional AI assistants are fundamentally disconnected from the physical reality of caregiving emergencies. Caregivers need a "second set of eyes and ears" and a calm, empathetic voice to guide them instantly. That realization was the spark for NeuroDecode AI: a real-time, hands-free, multimodal caregiving companion.

What it does

NeuroDecode AI is a hands-free caregiving agent designed for high-stress autism support moments. When a meltdown begins, the caregiver starts a live session and receives real-time voice guidance from Gemini. While the agent speaks, local audio and video observer models quietly detect distress cues and feed situational notes into the session. NeuroDecode also retrieves child-specific memory, such as known triggers or sensitivities, so the support is personalized instead of generic. After the session, it logs triggers and interventions, and when recurring patterns emerge, it proactively sends preventative guidance through in-app and FCM notifications.

How we built it

We built NeuroDecode as a mobile-to-cloud live agent system. The Flutter app handles the caregiver experience and live session controls. A FastAPI backend on Google Cloud Run manages Gemini Live voice sessions over persistent WebSockets. Firestore stores profiles, memory, and session history for retrieval during support moments. Custom TensorFlow/Keras audio and video models run as heuristic observer sensors, while Firebase Cloud Messaging powers proactive caregiver alerts when repeated distress patterns are detected.

Challenges we ran into

Building a real-time voice streaming architecture from scratch was incredibly unforgiving:

  1. The "Chipmunk" & Audio Stuttering: Piping raw PCM 24kHz audio chunks from WebSockets directly into Flutter caused massive buffer underruns, making the AI sound robotic or cut off entirely. We had to dig deep into native Android engineering to build a bounded queue.
  2. The 46KB Burst: We discovered that massive audio bursts from the backend were overflowing our mobile queues, causing the AI to lose half its sentences. We meticulously engineered a "sub-chunking" system to feed the native audio player bite-sized pieces (e.g., 5760 bytes), ensuring a smooth voice delivery.
  3. Resource Contention: Running a camera capture loop, native audio playback, and WebSockets simultaneously brought the mobile CPU to its knees. We optimized this by elegantly pausing the visual observer loop specifically when the AI was actively speaking.

Accomplishments that we're proud of

  • Taming the chaotic world of real-time native audio streaming to achieve a natural, conversational latency on mobile devices.
  • Successfully implementing a Rule-Based Proactive Engine with FCM push delivery, turning the app from a reactive tool into a proactive health assistant.
  • Layering custom Edge ML "sensors" on top of the Gemini Live API without causing "prompt leakage" (ensuring the AI maintains its empathetic persona and never reads secret system notes out loud).
  • Taking a deeply technical architecture and turning it into an accessible tool that can genuinely support an emotionally taxed community.

What I learned

  • Streaming is a different beast: Developing for a stateful, voice-first AI requires a completely different mindset than prompt-engineering a text LLM. Pacing, interruption handling, and buffer management are just as important as the prompt itself.
  • AI Ethics & Safety: We learned the vital importance of positioning our custom ML models as "assistive heuristic signals" rather than clinical diagnostic tools, ensuring the AI guides the caregiver rather than replacing their judgment.

What's next for NeuroDecode AI

The current build proves our core concept and proactive capabilities. Our next phase focuses on scaling the ecosystem:

  • Context-Aware Clinical Routing: Bridging the gap between digital assistance and real-world care. By integrating healthcare directory APIs (or a curated internal referral matrix), the proactive engine will connect caregivers with relevant professional help. For example, if the AI notes escalating behavioral patterns, it can gracefully route the caregiver to specialized ASD clinics, occupational therapists, or emergency telemedicine services tailored to their specific situation.
  • Time-Delayed Proactive Pipelines: Utilizing Cloud Scheduler to send gentle "check-in" notifications hours after a severe meltdown to monitor the child's recovery.
  • Longitudinal Analytics Dashboard: Exporting Firestore session data to BigQuery to help caregivers and therapists visualize trigger trends and intervention success rates over months.
  • iOS Support: Porting our hard-won native audio streaming logic to Swift to support the Apple ecosystem.

Built With

Share this project:

Updates