Nöra – Multimodal AI Cognitive Screening Agent

A real-time, voice-interactive AI agent designed for preliminary cognitive screening, helping individuals assess potential decline risks early on.

Comment

Inspiration

Many individuals lack access to low-stress, affordable environments to monitor their cognitive wellness. Traditional clinical screenings are often expensive, require trained personnel, and can be intimidating for the elderly. We asked: what if an AI agent could provide a preliminary, conversational assessment accessible to anyone, anywhere? Nöra was born from the intersection of applied Generative AI and a deep societal need. We wanted to build a tool that democratizes cognitive health awareness, serving as an early indicator rather than a diagnostic device.

What it does

Nöra is an autonomous voice-and-visual AI agent that conducts a structured, clinically-inspired preliminary cognitive assessment through natural conversation. It guides users through 4 interactive exercises:

  1. Verbal Fluency Test — Naming words in a specific category.
  2. Story Recall Test — Retelling a dynamically generated short story.
  3. Visual Recognition Test — Identifying objects in AI-generated images.
  4. Orientation Test — Time, date, and spatial awareness questions.

At the end of the session, Nöra produces a preliminary risk assessment score and a downloadable PDF report. It is explicitly designed as a wellness tracking and early awareness tool, encouraging users to consult medical professionals if risk patterns are detected.

How we built it

We utilized a multi-agent architecture (coordinating 5 distinct AI agents: Nöra, BrainAgent, VisualTestAgent, DateTimeAgent, VideoAnalysisAgent) powered by the Gemini Live API for real-time, interruptible voice conversations via a WebSocket proxy. We integrated Imagen 4 to dynamically generate unique images for the visual recognition tests, ensuring no two sessions are identical and preventing memorization.

The frontend is built with React, Vite, and TailwindCSS, utilizing AudioWorklet for low-latency audio streaming. Our backend relies on Node.js, Express, and PostgreSQL, deployed on Cloud Run with automated CI/CD pipelines.

Challenges we ran into

  1. LLM Hallucinations in Assessment: Early on, the AI would miscalculate test scores. We solved this by strictly separating data collection from evaluation. We implemented deterministic scoring: the AI only collects data via tool calling, and all mathematical evaluations are computed algorithmically on the backend.
  2. Real-time Audio Constraints: Browsers couldn't directly connect to the Gemini WebSocket due to CORS. We built a custom Node.js WebSocket proxy to manage server-side sessions and used AudioWorklet for seamless audio capture.
  3. Multi-Agent Synchronization: Managing 5 concurrent agents required a robust centralized state machine to avoid race conditions and ensure smooth transitions between test modules.

Accomplishments that we're proud of

  • Zero-hallucination scoring: By separating AI reasoning from deterministic computation, we achieved fully reliable, algorithmic test scoring.
  • True real-time voice agent: Sub-second latency voice conversation with interrupt support, moving far beyond standard turn-based chatbots.
  • Dynamic test content: Every session generates unique stories and images.
  • Genuine societal impact: Building a functional, accessible tool that addresses a real-world healthcare awareness gap.

What we learned

We learned that real-time voice AI requires highly explicit system instructions to maintain consistency across long sessions. We also realized that while LLMs are exceptional at conversational data gathering (via tool calling), they should not be trusted with deterministic mathematical scoring in sensitive domains—separating these concerns is vital for reliable AI applications.

What's next for Nöra – Multimodal AI Cognitive Screening Agent

  • Clinical validation: Partnering with neurologists to validate our preliminary screening accuracy against established clinical tests (like MoCA or MMSE).
  • Longitudinal tracking: Allowing users to retake screenings over time to detect trends in their cognitive health.
  • Multilingual expansion: Adding support for more languages to expand our global reach and accessibility.
  • Caregiver dashboard: Developing a platform for family members to monitor screening results securely.

Built With

Share this project:

Updates