VITAL

Dashboard Tab
Library Tab
Live Feed Tab

Inspiration

Overcrowded waiting rooms often mean that patients can deteriorate without staff noticing in time, especially after initial triage when attention is limited. Hospitals remain packed, overcrowding is getting worse, and healthcare workers are under more and more pressure. We were motivated to build a system that continuously monitors patients and ensures that critical changes in condition do not go unnoticed during these gaps in care, especially as one of our teammates has lost someone to these issues.

What it does

VITAL is an AI-powered camera system that passively monitors waiting rooms in real time, detecting signs of patient distress or deterioration and flagging high-risk individuals. It provides simple risk-level alerts and notifications to help medical staff prioritize attention more effectively. The platform consists of three main interfaces:

Waiting Room Monitoring Dashboard

A live monitoring dashboard displays multiple CCTV-style camera feeds simultaneously. Patients are detected and tracked in real time, each assigned an identifier (e.g., Patient 1, Patient 2).

A vision-language model analyzes behavior and posture to identify potential medical emergencies and ranks alerts based on urgency. These alerts appear in a prioritized alert panel so staff can immediately see which patient requires attention first.

Critical alerts can trigger notifications through optional voice alerts so urgent situations are not missed.

Video Analysis and Event Timeline

This interface allows staff to upload recorded footage for automated analysis.

Once a video is uploaded, the system processes the footage and identifies key behavioral events for each detected patient. These events are mapped onto an interactive timeline, allowing users to quickly jump to important moments without manually reviewing the entire recording.

Below the video player, a timeline bar highlights segments where notable events occurred. Each segment is paired with AI-generated logs describing what occurred during that moment, such as:

sudden collapse abnormal movement patterns prolonged inactivity possible seizure activity

This allows medical staff to quickly review incidents, audit events, or investigate patient deterioration.

Live Patient Monitoring

For patients requiring closer observation, the system supports dedicated live monitoring feeds.

Using a low-latency video analysis pipeline, the system continuously evaluates live footage and generates event logs in real time. Instead of a timeline (since the stream is live), the interface produces timestamped logs describing detected behaviors as they occur.

A summary panel continuously compiles the most significant events detected during the session, ensuring staff can quickly review critical moments without scrolling through every log entry.

When high-risk events are detected, the system can automatically notify caregivers through messaging, ensuring immediate awareness. The caregivers are then able to text back in case they need guidance with dealing with the patient, which responds back with suggestions from possessing the knowledge of patient symptoms/context

How we built it

We built VITAL by combining computer vision models for pose and motion detection with a rule-based risk scoring system. The system processes video input, extracts behavioral signals such as movement and posture, and converts them into real-time triage alerts. The system detects and tracks individuals within video feeds, analyzes behavioral signals, and identifies potential medical emergencies such as seizures, choking, bleeding, stroke symptoms, or loss of consciousness. Detected events are triaged by severity and surfaced to healthcare staff through visual alerts, logs, and notifications.

Computer vision models are used to detect and track individuals across frames. Behavioral signals such as posture, motion patterns, and inactivity are extracted from these detections and analyzed to identify potential medical emergencies.

Vision-language models interpret these signals and generate structured event logs describing patient behavior. These events are ranked based on severity to produce triage-style alerts.

The platform integrates real-time notification systems with Photon to surface urgent events directly to caregivers while maintaining a full event history through timelines and logs.

Challenges we ran into

Some of the largest challenges we found were getting accurate, low-latency pose detection. Additionally, we initially had trouble with the VLM identifying multiple people within one frame, and analyzing each person separately. We also needed to be careful with the rate limit, not exceeding/burning through tokens too quickly when testing. We also kept running out of tokens and hitting API rate limits.

Accomplishments that we're proud of

We are proud of building a complete end-to-end prototype that can detect critical behaviors and generate real-time alerts, integrating both alerts on the dashboard and through iMessage. We're proud that within one weekend, we have a polished, professional, useful, and potentially lifesaving product. The project demonstrates that passive monitoring can meaningfully support and augment existing triage systems.

What we learned

We learned that focusing on observable behavioral signals is more effective than attempting to directly infer medical conditions. We also recognized the importance of prioritizing sensitivity in a healthcare setting where missed detections can have serious consequences.

What's next for VITAL

Next, we plan to improve model accuracy, develop a simple interface for clinical use, and test the system in more realistic environments. Over time, we aim to integrate with hospital workflows and expand deployment across different care settings.