Nöra – Multimodal AI Cognitive Screening Agent

Nöra is a real-time multimodal AI agent that analyzes voice, facial cues, and cognitive responses to detect early signs of Alzheimer’s risk through natural conversation and interactive visual tests.

Comment

Inspiration

Alzheimer’s disease affects millions of people worldwide, yet early symptoms are often subtle and difficult to detect without clinical evaluation. We were inspired by the idea that everyday human interaction—such as speech patterns, response time, and visual recognition—can already contain signals of cognitive decline.

Our goal was to explore whether a multimodal AI agent could analyze these signals in real time and provide an accessible early cognitive screening experience. This idea led to the creation of Nöra.

What it does

Nöra is a real-time multimodal AI agent designed to perform early cognitive screening through natural interaction.

Instead of traditional questionnaires, users interact with the system through conversation, visual recognition tasks, and behavioral responses. The system analyzes multiple signals simultaneously, including speech patterns, response coherence, visual recognition ability, and facial focus through the camera.

The goal is not to diagnose Alzheimer’s disease but to provide an early cognitive risk indication that may encourage further medical evaluation.

How we built it

The system was built as a multimodal AI platform using Google’s AI and cloud ecosystem.

The frontend captures microphone and camera input and streams them in real time to the backend. The backend integrates with Gemini Live API to enable natural conversation with interruption handling.

Visual cognitive tasks are generated using image generation models, and the user’s responses are analyzed by backend algorithms.

The entire infrastructure runs on Google Cloud using Cloud Run for application deployment, Cloud SQL for structured data storage, and Secret Manager for secure credential management.

Challenges we ran into

One of the biggest challenges was building a low-latency system capable of handling multiple data streams simultaneously, including audio, video, and AI responses.

Another challenge was designing a reliable cognitive scoring approach. Instead of relying on LLM reasoning for scoring, we implemented deterministic backend algorithms to ensure more stable and explainable results.

Accomplishments that we're proud of

We successfully built a fully working multimodal AI agent capable of real-time voice interaction, camera-based observation, and interactive visual testing.

The system demonstrates how AI agents can move beyond simple text interfaces and create meaningful real-world applications that combine multiple human signals.

What we learned

Through this project we learned how powerful multimodal AI agents can be when applied to real-world problems.

We also learned that combining AI capabilities with deterministic algorithms can significantly improve reliability and interpretability in sensitive domains such as healthcare.

What's next for Nöra – Multimodal AI Cognitive Screening Agent

In the future we plan to expand the system with more cognitive test modules, improved behavioral analysis, and larger datasets for validation.

Our long-term vision is to develop Nöra into an accessible digital screening assistant that can support healthcare professionals by enabling earlier detection of potential cognitive decline.

Built With

Share this project:

Updates

Submission history