AURA

An AI memory prosthesis using Gemini vision to recognize loved ones and quietly whisper their names to Alzheimer's patients, restoring connection.

Comment

Inspiration

Every three seconds, someone in the world develops dementia. For 55 million patients and their families, the progression of Alzheimer’s slowly erases names, relationships, and shared histories. Conversing with a loved one becomes a source of high anxiety when a patient looks at their child or partner and sees a stranger.

Our team wanted to explore how passive, non-intrusive technology could serve as a real-time cognitive aid. We were inspired to build a system that acts as an invisible memory prosthesis—a gentle helper that quietly whispers context and names to patients right as their loved ones walk into the room, restoring comfort and dignity to daily interactions.

What it does

AURA is a hands-free, passive memory companion. The patient wears their smartphone on a lanyard with the rear camera facing outward, or places it nearby. Once started, the application requires zero user interaction:

  1. Passive Observation: The app streams live camera frames to our backend server.
  2. Intelligent Identification: The system analyzes the feed and cross-references visual data with a cached family history profile.
  3. Calm Audio Assistance: When a family member is identified, AURA does not make loud noises or display complex alerts. Instead, it gently whispers a conversation starter into the patient’s earpiece (e.g., "This is your brother, Chirag. You can ask him how his coding project is going.") and displays large, high-contrast text on the screen.

How we built it

AURA is designed as a lightweight, real-time application optimized for low latency and accessibility:

  • Frontend: Built using HTML5, CSS3, and Vanilla JavaScript (bundled with Vite). It utilizes navigator.mediaDevices.getUserMedia for passive camera access, an offscreen HTML5 Canvas for extracting frames, and native browser SpeechSynthesis (Text-to-Speech) for clear, pace-adjusted audio delivery.
  • Backend: Developed in Python using FastAPI and Uvicorn. The backend manages concurrent WebSocket connections and handles the pipeline between the client and the AI models.
  • Inference Engine: Powered by Google Gemini 2.0 Flash (gemini-2.0-flash-001) for vision analysis.

Frame Debouncing & Resource Optimization

To optimize API usage and avoid rate limits, we built a debouncing system. While the client captures camera frames at $1\text{ FPS}$ ($N_{\text{raw}} = 60\text{ frames/min}$), the backend enforces a cooldown interval of $t_{\text{cooldown}} = 5.0\text{ s}$.

This reduces active requests to: $$N_{\text{optimized}} = \frac{60}{t_{\text{cooldown}}} = 12\text{ frames/min}$$

This mechanism provides an $80\%$ reduction in token overhead and API calls: $$\text{Reduction} = \left(1 - \frac{N_{\text{optimized}}}{N_{\text{raw}}}\right) \times 100\% = 80\%$$

Challenges we ran into

  • API Quota Limitations ($429\text{ RESOURCE_EXHAUSTED}$): During testing, we hit the free-tier quota limits. This forced us to design a Graceful Degradation fallback. If the API returns a rate-limit error, the backend catches the exception and immediately serves a locally cached mock profile. This keeps the frontend UI active and stable rather than throwing a crash screen to a vulnerable patient.
  • OS-Level Hardware Locks: Running screen recording tools on our development machines caused hardware-exclusive microphone conflicts in the browser's getUserMedia layer. To prevent crashes, we adapted the frontend logic to selectively disable mic recording, allowing the video stream and speech synthesis output to execute reliably.
  • Windows File-System Conflicts: Our initial backend accumulated raw 16-bit PCM chunks and packaged them into WAV files. However, naming these temporary files with ISO-formatted timestamps created file-system errors on Windows because colons (:) are illegal in filenames. We solved this by cleaning up our temp-file generation parameters.

Accomplishments that we're proud of

  • Resilient Fallback Design: We developed an automated fallback state machine that gracefully catches API and network failures, displaying pre-cached local context instead of letting the application freeze or crash.
  • Zero-Interaction UI: We designed an interface that respects the cognitive boundaries of moderate-stage Alzheimer's patients—requiring only a single click to start, with high-contrast, large-format text overlays that adjust dynamically to device size.
  • Bidirectional WebSockets: We constructed a working asynchronous WebSocket pipeline in FastAPI capable of handling continuous frame transmission and returning text-to-speech triggers.

What we learned

  • Designing for Accessibility: We learned that when building software for users with cognitive decline, the best UI is no UI. Standard web patterns must be replaced by passive, automatic workflows.
  • Asynchronous State Handling: We gained experience coordinating real-time media streams, background debouncing loops, and external model calls concurrently in Python using asyncio.
  • Windows-Specific Portability: We learned how minor differences in OS file-naming conventions and hardware media loops can impact cross-device web applications, reinforcing the need for defensive, OS-agnostic coding practices.

What's next for AURA

  • Dedicated Hardware Integration: We hope to transition from a lanyard-bound phone to lightweight, wearable hardware, such as smart glasses or discreet clip-on cameras, to make the system completely hands-free.
  • Secure Caregiver Portal: We want to develop an administrative portal where families can securely upload labeled photos, edit memory notes, and review safe-wandering analytics.
  • On-Device edge Processing: We plan to integrate lightweight, on-device face recognition models to ensure the core identification features remain functional even in environments with absolutely no cellular connectivity.

Built With

Share this project:

Updates