HeldCare

Inspiration

We look into lots of elder's problems in hospice care centers, that many elders aren't able to comprehend in the future when the health problems become serious. We want to build an ER system that would prevent further risk in their health.

What it does

We designed HeldCare as a multi-agent pipeline, with each agent responsible for one part of the emergency-response process.

HeldCare supports the full emergency-response workflow, not just fall detection. When a potential fall is detected, it creates an incident and alerts the facility dashboard. Responders can accept the incident and access a bedside AI copilot from their phone.

The copilot guides the assessment using live camera and microphone input, while Presage and Gemini Live provides contactless pulse and respiration estimates. HeldCare records observations and generates a structured EMS handoff. It does not diagnose patients; it helps caregivers respond faster with better information.

How we built it

The application was primarily built with Next.js, TypeScript, MediaPipe Pose, Gemini Live, Presage/SmartSpectra integration architecture, Server-Sent Events, and browser camera/microphone APIs

Challenges we ran into

One of our biggest challenges was connecting several independent systems into a single real-time workflow. Fall detection, incident management, Gemini Live, camera and microphone access, vitals monitoring, and assessment state all operate differently. We had to design shared interfaces and typed handoffs so that one component could fail or change without breaking the entire system. Mobile browser permissions were another challenge. Camera and microphone access on iPhones requires a secure HTTPS context, so we had to create a Cloudflare tunnel that allowed the phone-based responder experience to communicate with the same development server running the facility dashboard.

Integrating contactless vitals was also challenging because we wanted the rest of HeldCare to work independently of the underlying Presage implementation. We solved this by creating a VitalsProvider abstraction, allowing the application to use simulated measurements during the hackathon while maintaining a path toward the live SmartSpectra integration. Finally, we had to carefully constrain our AI agents. Because HeldCare deals with emergency situations, Gemini is used to organize observations, guide workflows, and generate handoffs — not to diagnose a resident or invent medical findings.

Accomplishments that we're proud of

We're especially proud that HeldCare became more than an isolated fall-detection demo. We built an end-to-end emergency-response workflow where a detected fall becomes a live incident, the incident reaches responders in real time, a responder can open an AI-assisted bedside experience from their phone, contactless vitals can flow into the same assessment, and the collected information can ultimately become an EMS handoff. We're also proud of the modular architecture. Fall detection, incident orchestration, Gemini, vitals, assessment tools, and EMS reporting communicate through defined interfaces rather than being tightly coupled together. Most importantly, we designed HeldCare around human responders. AI supports the caregiver instead of replacing them, while the system deliberately avoids making medical diagnoses. What we learned

What we learned

HeldCare taught us that building an AI healthcare application is not just about choosing the best model. The difficult part is designing how AI, sensors, interfaces, real-time events, and people communicate with one another. We learned how valuable specialized agents can be compared with giving one AI model responsibility for everything. Separating context retrieval, protocol guidance, charting, problem recording, visual assistance, and report generation made the system easier to reason about and control. We also learned a lot about real-time browser applications: synchronizing state across devices, working with camera and microphone permissions, streaming events, handling mobile HTTPS requirements, and designing fall detection around pose data. Most importantly, we learned that AI in emergency response needs clear boundaries. A useful system doesn't have to diagnose someone; it can provide significant value simply by helping humans detect incidents earlier, organize information, and communicate more efficiently.

What's next for HeldCare

Our next step is moving HeldCare from a hackathon prototype toward a more realistic senior-care environment. First, we want to replace the simulated vitals pipeline with a fully validated live Presage SmartSpectra integration, allowing compatible cameras or phones to provide real-time contactless measurements directly to the responder workflow. We also want to improve fall detection across different camera positions, lighting conditions, body types, mobility aids, and real-world senior-living environments while reducing false alarms. Beyond falls, the same architecture could eventually support additional safety events and longer-term monitoring workflows. We would also like to add persistent resident and incident records, responder authentication and permissions, stronger privacy protections, facility analytics, and integrations with existing healthcare systems. Ultimately, our vision for HeldCare is an AI-assisted safety layer for senior living: technology that continuously watches for emergencies when enabled, quickly brings a human responder into the loop, gives them useful information at the bedside, and helps transfer that information accurately to the next level of care.