AI-NeuroGuardian

A multimodal AI companion that understands voice, vision, and context to support memory, wellbeing, and independent living.

Comment

🧠 NeuroGuardian – Multimodal AI Companion for Cognitive Support

Inspiration

Millions of elderly individuals and people experiencing memory decline, stress, depression, or early cognitive impairment struggle with everyday situations that often go unnoticed until they become severe.

These challenges include:

  • forgetting medication schedules
  • misplacing everyday items such as glasses or keys
  • experiencing confusion during routine tasks
  • emotional distress or loneliness
  • early symptoms of Mild Cognitive Impairment (MCI), dementia, or Alzheimer’s disease

While reminder apps and digital assistants exist, they typically rely on manual schedules or static notifications and cannot understand what is happening in the user's environment.

We asked a simple question:

What if an AI companion could observe the environment, understand context, and provide assistance in real time?

This idea led us to build NeuroGuardian, a multimodal AI companion designed to support elderly individuals and people experiencing cognitive challenges by combining vision, voice interaction, and contextual AI reasoning.

What the Project Does

NeuroGuardian is a multimodal AI assistant that combines:

  • camera-based environmental awareness
  • voice interaction
  • AI reasoning using Google Gemini
  • simple memory tracking
  • on-demand emotional comfort generation

The goal is to provide gentle, contextual assistance that helps users maintain independence while also supporting caregivers.

📷 Multimodal Environment Awareness

The system captures camera frames and sends them to the backend where Gemini multimodal reasoning analyzes the scene.

Example insight:

Glasses detected near bedside table
→ AI suggests checking the table

This helps users who experience forgetfulness or confusion when searching for objects.

The system uses a short observation window of multiple frames to avoid reacting to every single camera frame and instead analyze the environment more reliably.

🧠 Memory Assistance

If objects are detected in the environment, the system stores their last observed location.

Example workflow:

AI observes glasses on table
↓
Location stored in memory
↓
User asks: "Where are my glasses?"
↓
AI retrieves last known location

This feature helps people experiencing age-related memory lapses or early cognitive decline.

🗣 Voice Interaction

Users can interact naturally with NeuroGuardian using voice.

Example interaction:

User: Where are my glasses?
NeuroGuardian: Your glasses were last seen near the bedside table.

Voice input is captured using the browser SpeechRecognition API and analyzed by Gemini.

⚠ Emotional Awareness

The system analyzes voice input to detect possible emotional signals such as:

  • stress
  • sadness
  • confusion

If distress is detected, the AI provides calming responses designed to reassure the user.

📖 “Tell Me a Story” – AI Comfort Mode

NeuroGuardian includes a Tell Me a Story feature, which generates calming stories on demand using Gemini.

This feature is designed to provide emotional comfort for users experiencing:

  • anxiety
  • stress
  • loneliness
  • difficulty relaxing

Example interaction:

User: Tell me a story
AI: Imagine sitting beside a quiet lake surrounded by gentle trees...

Stories are generated dynamically and spoken aloud using browser speech synthesis, creating a more comforting experience.

How We Built It

NeuroGuardian is built using a cloud-based multimodal architecture combining web technologies with AI services.

Frontend

The user interface is built using:

  • React
  • Vite
  • Web Speech API
  • HTML5 Camera API

The frontend handles:

  • camera frame capture
  • voice interaction
  • displaying AI guidance

The application is hosted using:

Firebase Hosting

Live application:

https://armour-assistant.web.app

Backend

The backend is implemented using:

Python + FastAPI

It processes camera frames and voice inputs before sending them to the AI model.

The backend runs on:

Google Cloud Run

Backend API:

https://neuroguardian-backend-1024847090973.us-central1.run.app

AI and Multimodal Reasoning

The project uses:

Google Vertex AI
Gemini 2.5 Flash

Gemini performs multimodal reasoning, combining:

  • text input
  • voice transcripts
  • camera images

to generate contextual assistance.

Cloud Infrastructure

The system uses multiple Google Cloud services:

Component Service
Frontend hosting Firebase Hosting
Backend runtime Cloud Run
AI model Vertex AI (Gemini)
Container registry Artifact Registry
CI/CD Cloud Build
Logging Cloud Logging

Challenges We Ran Into

Handling Camera Frame Processing

Sending raw frames continuously created unstable results and large payloads.
We solved this by implementing an observation window that buffers frames before sending them to the AI model.

Managing AI Output Reliability

AI models do not always return structured JSON responses.
We added response normalization and error handling to ensure the backend always returns valid structured data.

Cloud Integration

Connecting Firebase Hosting with a Cloud Run backend required careful configuration of:

  • CORS policies
  • deployment pipelines
  • container builds

This provided valuable experience in deploying serverless AI applications.

What We Learned

Through this project we learned:

  • how to build multimodal AI applications
  • how to deploy serverless architectures on Google Cloud
  • how to integrate vision and voice interfaces
  • how to handle AI unpredictability safely

We also gained insights into how AI can support real-world assistive applications for aging populations.

Future Scope

NeuroGuardian currently demonstrates a working AI-assisted cognitive support prototype.

Future improvements could include:

  • improved multi-frame environment analysis
  • caregiver notification systems
  • mobile device support
  • secure long-term memory tracking
  • privacy-focused patient data management

The goal is to explore how AI companions could support independent living for elderly individuals while complementing caregiver support systems.

Built With

Languages

  • Python
  • JavaScript

Frontend

  • React
  • Vite
  • Web Speech API
  • HTML5 Camera API

Backend

  • FastAPI

AI

  • Google Vertex AI
  • Gemini 2.5 Flash

Cloud

  • Firebase Hosting
  • Google Cloud Run
  • Cloud Build
  • Artifact Registry

Try it out

Live demo:

https://armour-assistant.web.app

Backend API:

https://neuroguardian-backend-1024847090973.us-central1.run.app

Source code:

https://github.com/pkc533/neuroguardian-ai-agent

Built With

Share this project:

Updates

Submission history