-
-
this was the brainstormed vision before starting a project
-
how the project ended up looking
Inspiration
worked as an Aquatic Supervisor for over 3 years, it was time to hire/create a new assistant. Drowning is deceptively silent. There's no screaming, no dramatic splashing — it happens in seconds, often in plain sight. We kept thinking about the statistics: drowning is one of the leading causes of accidental death worldwide, and a significant portion of those incidents happen in supervised pools. Lifeguards are highly trained, but they're human. They blink. They get distracted. They can only focus on one area at a time.
What it does
AquaGuard is a real-time AI swimmer distress detection system. It ingests a live camera feed or recorded pool footage, runs YOLOv8 pose estimation on every frame to track each swimmer's body position, and feeds those keypoints into a custom distress scoring engine. Each swimmer receives a score from 0–100 computed from eight concurrent signals: wrist velocity, arms raised above the head, head position relative to a dynamically detected waterline, absolute stillness duration, loss of nose keypoint confidence (indicating submersion), body vertical lock, arm asymmetry, and disappearance from frame. When a score crosses 40, a WARNING fires on the dashboard. At 70, the system escalates to DISTRESS — triggering a spoken voice alert via ElevenLabs, a Gemini Vision AI confirmation of the scene, and a full-screen overlay telling the lifeguard exactly which swimmer needs help and what to do. Gemini also independently analyzes the entire pool scene every 2 seconds, providing a live risk level and recommended action in the sidebar.
How we built it
The backend is a Python 3.11 FastAPI server. We used YOLOv8s-pose from Ultralytics for per-frame skeleton inference, running on Apple Silicon via MPS at 20–30 FPS. ByteTrack (built into Ultralytics) handles multi-swimmer tracking and persistent ID assignment across frames — including spatial re-identification if a swimmer briefly submerges and resurfaces. The distress scoring engine runs on the median keypoints of the last 5 frames (frame fusion) and applies an exponential moving average (α = 0.35) to smooth scores and suppress noise. Waterline detection uses OpenCV HSV masking and Hough line transforms to find the actual water surface each frame dynamically, so head-position scoring adapts to camera angle and lighting conditions. For AI scene analysis, we integrated Gemini 1.5 Flash at two tiers: a background loop running every 2 seconds for overall scene risk, and a per-event call triggered when the pose engine flags DISTRESS. ElevenLabs multilingual v2 synthesizes spoken alerts in a background thread so inference is never paused. The frontend is React 18 + Vite, connected via WebSocket with sub-100ms latency. The annotated MJPEG
Challenges we ran into
False positives was my biggest enemy. Kids splashing, swimmers doing butterfly stroke, and people pushing off the wall all generate signals that look like distress. I went through several iterations of the scoring weights and introduced frame fusion (median of the last 5 keypoint frames) and exponential smoothing before false alarm rates dropped to an acceptable level. Waterline detection proved harder than expected. Pool footage has reflections, foam, lane rope shadows, and varying lighting that confuse simple HSV thresholding. We iterated on the Hough line parameters and blending approach significantly. Swimmer re-identification after submersion was a subtle problem. If a swimmer ducks underwater and resurfaces, ByteTrack assigns them a new ID, losing all their score history and stillness timers. We solved this with a spatial overlap heuristic that matches re-emerging bounding boxes to recently lost tracks. Gemini API latency at the per-event distress confirmation step initially added noticeable delay to the overlay. We decoupled it into an async thread so the visual alert fires immediately while Gemini's response streams in separately.
Accomplishments that we're proud of
We're proud of building a genuinely multi-layered detection system rather than relying on a single model. The pose engine and Gemini Vision operate completely independently — if one misses a distress event, the other is still watching. The two-tier Gemini integration (scene-level every 2s + per-event confirmation) felt like a meaningful design choice, not just an API integration checkbox. The dynamic waterline detection is something we're especially proud of. Instead of hardcoding a pixel threshold, the system finds the actual water surface in each frame — which means it works across different pool configurations, camera heights, and lighting conditions without recalibration. Getting everything to run at 20–30 FPS on a MacBook with real-time WebSocket updates, voice synthesis, and a responsive React frontend — all within a hackathon timeframe — felt like a real achievement.
What we learned
We learned that the hard part of a detection system isn't the detection — it's the signal processing around it. Raw model outputs are noisy. The score smoothing, frame fusion, and per-swimmer state machines ended up being as important as the pose model itself. We also learned a lot about responsible AI design in safety-critical contexts. A system like this has real consequences if it cries wolf too often (alert fatigue leads lifeguards to ignore it) or misses events (the whole point breaks down). That tension shaped every design decision we made around thresholds, cooldowns, and confirmation layers. On the technical side: Gemini's Vision API is remarkably capable at natural-language scene understanding with minimal prompting, and ElevenLabs synthesis is fast enough for real-time safety alerting when handled asynchronously.
What's next for AquaGuard-AI
The most impactful next step is training a custom YOLOv8-pose model on actual labeled pool footage with annotated distress events. The current base model was trained on COCO person images, not swimmers — a fine-tuned model would dramatically improve keypoint accuracy in water environments. Beyond that: multi-camera support for large aquatic facilities, integration with existing lifeguard communication systems (pagers, radios), a mobile companion app for lifeguards on the deck, and a historical analytics dashboard for pool operators to review incidents and near-misses. We'd also want to conduct a formal evaluation study in a real pool environment with trained lifeguards to measure detection latency, false positive rate, and actual response time improvement — the metrics that matter for a safety system deployed in the real world.
Log in or sign up for Devpost to join the conversation.