FrontLine

Our Solution

Inspiration

Several of us have had loved ones injured or in medical distress where help felt delayed—not because responders didn’t care, but because the path from a phone call to meaningful care requires assembling fragmented information, such as the injured person’s identity, location, and condition. In moments of extreme stress, callers often struggle to describe what’s happening, follow verbal instructions, or clearly advocate for the person in need, causing the most critical early moments to be lost. Those minutes matter, and this project grew from a desire to use every available second more effectively by helping initiate the right actions as early as possible, while professional care is still on the way.

What it does

Frontline is a video-first emergency response system that enables individuals in need to visually connect with 911 operators, allowing responders to assess situations beyond what can be conveyed through audio alone. The system integrates live video access via Zoom, which is analyzed in real time using computer vision through Claude to extract environmental and situational context. Caller audio is processed in parallel to capture spoken descriptions and responder instructions. Rather than independently analyzing raw audio, Perplexity serves as a synthesis layer, combining structured outputs from the computer vision pipeline with parsed audio input to generate a consolidated incident report.

How we built it

Mobile frontend (emergency caller app): The user-facing mobile application is built with React Native and TypeScript, creating a frictionless app experience for emergency callers.
Web frontend (operator dashboard): The operator interface is built with HTML, CSS, and JavaScript, providing real-time visibility into incoming emergency calls and active responses.
Real-time communication: FrontLine’s video streaming capabilities are planned using Zoom Video SDK to enable live video connection between emergency callers and operators during active responses.
AI processing: Claude (Anthropic) analyzes real-time images from emergency scenes to extract environmental context, while Perplexity AI synthesizes audio and visual findings into post-call incident reports.

Challenges we ran into

Zoom meeting SDK integration: Encountered difficulties embedding the Zoom Video SDK into the web application for live video streaming between callers and operators, requiring careful configuration of authentication and session management.
Poke integration: The main challenge was migrating Poke from a local environment to deployment on Render, requiring adjustments to environment configuration and asynchronous API handling to ensure reliable execution after the transition.
Computer vision analysis: Ensuring accurate analysis requires correctly accessing and isolating the appropriate video stream and frames within the Zoom session, specifically distinguishing and processing the caller’s video feed rather than the operator’s, while maintaining real-time performance.

Accomplishments that we're proud of

Built a fully functional emergency response app in 36 hours that could have a real positive impact on people's safety.
Successfully integrated Claude AI with Zoom to analyze video/images in real-time for emergency situations.
Created an intelligent system that processes visual information to help emergency responders make faster, better-informed decisions.

What we learned

How to integrate multiple third-party APIs and SDKs (Claude AI, Zoom SDK, etc.) into a mobile app, handling authentication, real-time data streaming, and making different services communicate seamlessly.
Tackled integration challenges in a niche problem space with limited documentation, learning to troubleshoot and problem-solve without established guides.
Built a product that addresses a real public safety need, designing with accessibility and ease-of-use in mind for high-stress emergency situations.

What's next for FrontLine

Scene-specific models: Extend computer vision beyond generic scene understanding to detect medically relevant cues (e.g., body posture, bleeding indicators, unconsciousness, fire/smoke presence).
Temporal reasoning: Track changes over time (e.g., patient movement, responsiveness) instead of frame-by-frame analysis to improve situational continuity.
Low-bandwidth robustness: Add adaptive frame sampling and degraded-vision fallbacks for poor network conditions common in emergencies.