NARAE: A Voice-First Wearable and Real-Time Caregiver Platform

Inspiration

As populations age, cognitive decline and dementia present growing challenges for elderly individuals and their caregivers. Everyday activities, such as remembering medication schedules, locating personal belongings, and communicating urgent needs, can become increasingly difficult.

Meanwhile, nurses and family caregivers often manage multiple responsibilities, making continuous individual supervision challenging.

We wanted to address this gap without introducing another complex device that requires elderly users to navigate touchscreens. Instead, we envisioned a simpler approach built around natural speech and minimal physical interaction.

This inspired NARAE, a voice-first smart bracelet connected to a real-time caregiver dashboard. It aims to help elderly individuals access everyday assistance while giving caregivers better visibility into patient activities and urgent requests.

What It Does

NARAE connects two interfaces into a single care ecosystem.

Patient Interface: AI Smart Bracelet

The 3D bracelet simulation is designed around simple interaction, featuring two primary controls:

Schedule (Blue): Provides access to daily routines and scheduled activities.
Help (Red): Allows patients to request assistance when needed.

Its voice-based capabilities include:

Contextual Voice Assistance

Powered by Google Gemini Function Calling, the assistant retrieves information from the patient's registered inventory and responds through Indonesian text-to-speech. For example, a patient can ask, "Where are my reading glasses?" and receive an answer based on available inventory records.

Emergency Detection and Escalation

The system recognizes predefined distress expressions and forwards relevant alerts to the caregiver dashboard, including the patient's complaint transcript and room information. NARAE facilitates caregiver intervention rather than providing medical diagnoses or clinical recommendations.

Caregiver Interface: Mobile Dashboard

The caregiver dashboard provides visibility into patient routines, assistance requests, and emergency events.

Its main capabilities include:

Routine Management: Tracks activities through state transitions such as SCHEDULED, IN_PROGRESS, NEEDS_APPROVAL, and MISSED.
Real-Time Emergency Notifications: Delivers patient complaint transcripts and room information to help caregivers respond to urgent requests.
Flexible Patient Management: Supports both 1:1 homecare and 1:N nursing home scenarios, allowing caregivers to manage individual or multiple patients.
Isolated Demo Sessions: Generates independent testing sessions with real-time pairing between a laptop running the bracelet simulation and a smartphone displaying the caregiver dashboard.

How We Built It

NARAE follows a modular, layered architecture that separates the frontend, AI agent, backend services, and data persistence.

Technology Stack

Component	Technologies
Wearable Simulation	Three.js, Web Speech API
AI Agent	Google Gemini 2.0 Flash, Function Calling
Backend	Node.js, Express, TypeScript
Database	PostgreSQL 16
Frontend	React 18, Vite 5, Tailwind CSS v4
Infrastructure	Docker, Docker Compose

Implementation Highlights

3D Wearable and Voice Interaction

We built the bracelet procedurally using Three.js, with interactive orbit controls, component breakdowns, and LED indicators. The Web Speech API enables Indonesian speech-to-text (STT) and text-to-speech (TTS), allowing users to interact with the simulated device through voice commands.

AI Agent and Safety Guardrails

The @narae/agent package integrates Gemini through multi-round Function Calling. Explicitly defined tools allow the agent to retrieve contextual information and execute supported application operations instead of relying entirely on generated responses.

We also implemented restricted generation, closed-world contextual retrieval, and fallback handling to prevent unsupported medical advice and keep interactions within the system's capabilities.

Backend and Data Management

The backend uses Express and TypeScript with separate controller, service, repository, and domain layers. PostgreSQL 16 handles relational data using parameterized queries and referential integrity constraints.

Scheduling logic accounts for recurring activities, task durations, and day boundaries. Time-sensitive evaluations are normalized to Asia/Jakarta to ensure accurate status transitions and prevent future activities from being incorrectly marked as missed.

The application is containerized using Docker and Docker Compose to provide a consistent execution environment.

Challenges We Ran Into

Maintaining Safe AI Behavior

Preventing the language model from generating unsupported medical recommendations was one of our main challenges. We addressed this through constrained Function Calling, deterministic system instructions, contextual retrieval, and explicit fallback mechanisms. The agent is designed to handle supported requests and escalate relevant concerns rather than provide clinical advice.

Timezone-Aware Scheduling

Recurring tasks introduced edge cases involving expiration times and transitions across calendar days. Incorrect timezone handling could cause future activities to be marked as missed. We addressed this through SQL date evaluation, duration-aware status transitions, and consistent WIB (Asia/Jakarta) timezone normalization.

Cross-Device Demo Synchronization

Our demonstration required a laptop-based wearable simulation and a separate smartphone dashboard to operate within the same environment. We implemented isolated demo sessions with dynamic identifiers and real-time device pairing to prevent data collisions between simultaneous demonstrations.

Accomplishments

We successfully integrated the wearable simulation, AI agent, backend services, and caregiver dashboard into an end-to-end working application.

Our key accomplishments include:

Voice-Driven Operations: Voice commands can invoke application functions, trigger database mutations, and initiate caregiver notifications.
Integrated Emergency Workflow: Connected patient-side distress detection with real-time caregiver alerts containing relevant complaint information.
Multi-Device Demonstration: Enabled synchronized laptop and smartphone sessions through isolated demo environments.
Accessibility-Oriented Design: Created a voice-first interaction model with minimal physical controls to reduce interface complexity.

What We Learned

Building NARAE strengthened our understanding of integrating AI into applications where reliability and user safety are essential.

We learned how Function Calling can connect language models with deterministic backend operations, allowing AI to retrieve contextual information without relying entirely on free-form generation.

We also gained practical experience in accessibility-oriented interaction design, particularly the importance of simple controls, clear auditory feedback, and reduced interface complexity.

From an engineering perspective, the project deepened our experience with layered architecture, relational database management, timezone-aware scheduling, concurrency considerations, and isolated demo environments.

Most importantly, we learned that AI safety must be addressed at the architectural level, not solely through prompting.

What's Next for NARAE

Our next development phase focuses on extending NARAE beyond simulation:

Physical Hardware Prototyping: Transitioning the Three.js simulation into an ESP32-based wearable equipped with a microphone, vibration motor, and speaker.
Personalized Family Voice Integration: Exploring consent-based voice personalization for medication reminders and daily notifications.
Healthcare Interoperability: Investigating HL7 and FHIR integration to connect relevant information with existing hospital infrastructure, including nurse call systems.

Our long-term goal is to develop NARAE into a practical assistive technology that improves communication between elderly individuals and caregivers through accessible, reliable, and responsible AI-powered interactions.