R U OK?

A smart personal assistant designed for the elderly.

Comment

Inspiration

In an old apartment building, 78-year-old Grandma Wang was at a loss because she couldn't find her reading glasses. She couldn't remember where she had put them and kept overlooking details. She searched everywhere, stumbling over unnoticed electrical wires and water stains on the floor. When she finally found her glasses and was about to leave, her memory failing her, and she forgot to turn off the lights and lock the door.

As global population aging accelerates, the safety and memory problems of elderly people living alone has become increasingly critical. As global population ageing accelerates, the number of older adults is rapidly rising (e.g., the 60+ share is projected to increase from 12% to 22% between 2015 and 2050). Meanwhile, living alone is increasingly common among older adults (e.g., ~31% of people aged 65+ live alone across OECD countries). This makes safety and memory challenges especially critical: falls cause ~684,000 deaths annually, and dementia affects ~57 million people worldwide.

R U OK? was inspired by such simple question: How can we make technology truly care for the elderly, rather than just monitor them? We wanted to create a gentle, caring, and proactive intelligent assistant that can perceive hazards, make intelligent decisions, maintain memory, and engage in natural dialogue.

What it does

R U OK? is a real-time monitoring system based on a multi-agent architecture. The system protects seniors through:

Core Features

  1. Real-Time Visual Perception - Uses Gemini Vision API to identify 7 types of indoor hazards (trip hazards, slippery floors, clutter, etc.) and detect personal items/appliances,predicting danger in advance.

  2. Intelligent Decision System - Risk-based decision logic with smart cooldown. Only warns when hazards are within 1.0m and belong to critical types: $$ \text{Warning} = \begin{cases} \text{True} & \text{if } d < 1.0\text{m} \text{ and } h \in \text{critical_types} \ \text{False} & \text{otherwise} \end{cases} $$

  3. Memory System - Tracks item locations, risk history, and supports natural language queries ("Where is my phone?")

  4. Abnormal Behavior Alerts - Monitors daily activities and reminds users of forgotten tasks:

    • Lights left on: Alerts when user leaves a room with lights still on
    • Gas stove safety: Warns if stove is left on for >5 minutes after leaving kitchen, escalates to emergency if >10 minutes
    • Water faucet: Reminds to turn off faucet if left running for >1 minute after leaving room
    • Door lock reminder: Detects when user leaves home and reminds to lock the door (planned feature)

  5. Real-Time Voice Interaction - Gemini Live API integration for bidirectional voice conversations

  6. Fall Detection & Emergency Contacts ⭐ - Monitors for falls using pose estimation. If fall detected and no response after 30s, automatically calls emergency contacts with context (location, recent activity).

How we built it

Technology Stack

  • Frontend: React 19.2.4 + TypeScript + Vite
  • AI: Google Gemini API (Vision + Live)
  • Styling: Tailwind CSS
  • Storage: LocalStorage

Architecture

Multi-agent system coordinated by Orchestrator:

  • SentryAgent: Visual perception
  • StrategyAgent: Decision logic (includes fall detection)
  • ChroniclerAgent: Memory updates
  • MemoryAgent: Natural language queries
  • ActionAgent: Action execution (includes emergency calls)

Key Implementation

Fall Detection Algorithm:

  • Pose estimation from video frames
  • Monitors vertical displacement and velocity
  • Response sequence: Fall detected → Voice check → Wait 30s → Call emergency if no response

Distance-Based Risk Assessment:

  • Near (< 1.0m): Urgent warning
  • Mid (1.0-2.5m): Warning
  • Far (> 2.5m): Ignore

Abnormal Behavior Detection:

  • Tracks appliance states and room transitions
  • Detects when user leaves room with appliances still on
  • Time-based thresholds: lights (3 min), faucet (1 min), gas stove (5 min warning, 10 min emergency)
  • Context-aware reminders based on user location

Challenges we ran into

  1. Gemini API JSON Output Stability

  2. Real-Time Audio Synchronization

  3. Fall Detection False Positives

  4. Performance Optimization

Accomplishments that we're proud of

  1. Complete Multi-Agent Architecture - Clear, extensible, and maintainable system design

  2. Gentle Yet Effective Alerts - Only warns when necessary, uses conversational reminders, smart cooldown

  3. Powerful Memory System - Natural language queries, automatic item tracking, time-series events

  4. Seamless Real-Time Voice - Low-latency bidirectional audio with transcription

  5. Life-Saving Fall Detection ⭐ - Accurate detection with automated emergency response, privacy-first design

What we learned

Technical: Multi-agent design, real-time audio/video processing, AI API integration, computer vision for pose estimation

Product: Empathetic UX design, balancing safety with privacy, accessibility considerations

Humanistic: Technology as a guardian, not surveillance. Understanding the responsibility of building life-saving systems.

What's next for R U OK?

Short-Term (1-3 months)

Sensitivity and detail adjustments

Mobile App Optimization & Daily Logging

  • Enhanced mobile interface design: Redesigned UI optimized for elderly users with larger buttons, clearer typography, and simplified navigation
  • Daily activity log: Comprehensive daily journal tracking:
    • Items found/remembered each day
    • Fall detection incidents and false alarms
    • Safety reminders triggered (lights, appliances, etc.)
    • etc.
  • Reminder system: Input-based reminder functionality:
    • Users or caregivers can set custom reminders
    • Scheduled notifications for medications, appointments, tasks
    • etc.
  • Door lock detection: Visual recognition of door state and reminder system

Core Improvements

  • Improve fall detection accuracy with ML models
  • Enhanced emergency contact management (contact groups, scheduling)
  • Smart home integration with emergency automation

Medium-Term (3-6 months)

Smart Glasses Integration ⭐ Key Innovation

  • Partnership with smart glasses manufacturers: Integrate R U OK? directly into smart glasses
  • Hands-free operation: Continuous monitoring without requiring phone/camera setup
  • First-person perspective: More accurate hazard detection from user's viewpoint
  • etc.

Outdoor Navigation & Safety

  • Outdoor mode activation: Extend safety monitoring beyond indoor environments
  • Road condition recognition:
  • Route finding & navigation:
  • Outdoor fall detection: Enhanced pose estimation for outdoor scenarios
  • Traffic awareness: Crosswalk detection and traffic light recognition

Advanced Features

  • Multi-modal fusion (visual + audio + sensors)
  • Personalized learning and adaptive thresholds
  • Offline fall detection capabilities
  • Integration with medical alert services

Long-Term (6-12 months)

Complete Ecosystem

  • Health integration:

AI Companion Evolution

  • Emotional support: Proactive companionship and conversation
  • Cognitive training: Memory games and brain exercises
  • Social connection: Video calls with family, community features

Conclusion

R U OK? is our practice of "technology for good." Let's use technology to protect those we love. 💙

Built With

Share this project:

Updates