Inspiration
We can’t always be there for the people we love. Whether it’s a quick trip to the grocery store or an entire ocean between us, there are gaps in the day when a grandparent or relative is on their own. Most of the time, that’s fine. But when a fall or a sudden collapse happens in one of those gaps, the difference between a scare and a tragedy often comes down to how quickly someone finds out.
That worry is something all of us on the team share, with grandparents of our own. We started by looking at the fall-detection cameras and wearables already used in homes, and asked a simple question: what if a system could notice an emergency the moment it happens, confirm it, and get the right person to respond, even when no one is in the room?
We wanted to build that bridge: not a replacement for family, but a safety net that keeps watch in the moments we can’t, so help arrives in minutes instead of hours. We also wanted it to respect the person being watched, which is why C-BOS keeps all video on-device and shares only an anonymous skeleton outline.
What it does
C-BOS (We See, We Save) is a low-cost home safety monitor for people who live alone. Built from a Raspberry Pi 4, a webcam, a microphone, and a speaker, it watches for falls and sudden collapses, then confirms the emergency and gets help moving, without anyone having to be in the room.
Day to day, it stays on the device. A pose model running on the Pi tracks a stick-figure skeleton of the resident, and an audio monitor listens for distress words like “help” and impact sounds. Nothing is streamed continuously to the cloud.
When something looks wrong, it checks before it alarms. If the resident stays down, C-BOS pulls a handful of keyframes and sends them for a second-opinion analysis, combined with what the microphone heard. If it confirms a likely emergency, C-BOS speaks up and asks, “Are you okay?” Saying “I’m fine” cancels the alert, which keeps false alarms from becoming a nuisance.
If the answer is distress or silence, it escalates. C-BOS calls first responders or family with the resident’s medical profile, and a dashboard shows what happened so the person responding can verify the situation before acting.
How we built it
C-BOS runs on a Raspberry Pi 4 with an asynchronous Python (asyncio) architecture. A LangChain Deep Agent orchestrates two sensing subagents through a five-state machine: MONITORING → EVALUATING → CHALLENGING & AWAITING_RESPONSE → ESCALATING.
Monitoring. On the Pi, an Ultralytics YOLO pose model tracks the resident’s skeleton at 5 FPS over a 20-second sliding window. If they’re down for most of it, the vision subagent pulls 5 keyframes and raises an alert. An audio subagent listens for distress words and impact sounds, ignoring negated phrases like “I don’t need help.”
Evaluating. The keyframes go in parallel to a custom-trained Roboflow YOLO fall-detection model. Gemini 3.5 Flash then weighs the frames, the Roboflow report, and the audio transcript and returns a structured Pydantic EmergencyDecision. If Gemini or Wi-Fi fails, the agent falls back to on-device and Roboflow evidence.
Challenging & Awaiting response. If an emergency is confirmed, the mic opens. “I’m fine” cancels the alert. Any other response, or silence, escalates.
Escalating. A second Deep Agent uses custom @tool functions to look up the resident’s medical notes and emergency contact, then places an automated Bland.ai phone call that explains what happened and answers the contact’s questions. A 120-second lock prevents duplicate calls. Meanwhile, aiohttp streams the evidence frames to our SentinelAPI dashboard so a human can verify the situation.
Challenges we ran into
Limited sensing hardware. We didn’t have an external camera, and our external microphone was unreliable, so we built and tested with a laptop webcam and its built-in mic. That meant tuning our pose tracking and distress-keyword detection around hardware that wasn’t what the final design called for.
A late start on the Raspberry Pi. We couldn’t get the Pi set up until about 12 hours into the hackathon, which left much less time to run and debug the on-device pipeline. We developed the software in parallel and moved it onto the Pi once it was ready.
Designing a system with many moving parts. We had to work out how the vision, audio, cloud reasoning, voice check-in, and dispatch components would fit together, and which tools could best interpret sensor data on the hardware side. Settling on a state machine coordinated by a Deep Agent gave us a clear structure to build around.
Training our own fall-detection model. Most of us were new to model training, and processing our Roboflow dataset took far longer than we expected. Every iteration cost us time we didn’t have. Therefore, we were only able to use one dataset.
Running on limited Gemini API credits. With a small credit budget, we couldn’t count on Gemini being available for every decision. So we paced our requests and built fallbacks throughout the pipeline: if Gemini is unavailable, the agent decides using on-device and Roboflow evidence, and the emergency call can still go out without the LLM. A safety system shouldn’t depend on a single paid service, so the constraint made C-BOS more resilient.
Changed messaging system. Originally, we had Twilio for our SMS messaging system. However, because of limitations with our free account, we weren’t able to implement an SMS feature to the project.
Accomplishments that we're proud of
Building a multi-agent system from scratch. None of us had worked with computer vision, Deep Agents, or subagents before this weekend. By the end, we had a working pipeline in which a Deep Agent coordinates vision and audio subagents through a five-state machine, from spotting a possible fall to escalating it.
Getting it running on a Raspberry Pi. Even though we couldn’t set up the Pi until about 12 hours in, we still got our control software running on it, with pose tracking and the full state machine working on a $55 board.
Combining several kinds of AI into one workflow. C-BOS brings together a computer vision interpreter that tracks pose and detects falls, a voice interpreter that listens for distress words and answers to the check-in, and an AI phone call that briefs the emergency contact and answers their questions. Each handles a different part of the job, and together they take the system from detecting a fall to getting help.
A system that keeps working when parts fail. Because we were short on Gemini credits, we built C-BOS to fall back to on-device and Roboflow evidence and still place the emergency call without the LLM. We’re proud that it doesn’t depend on a single service.
What we learned
How to build with Deep Agents and subagents. We learned to split a problem into specialized parts: vision and audio subagents that filter raw sensor data, and a Deep Agent that decides what to do with their reports. Structuring the flow as a state machine kept the system predictable and easy to debug.
How to train and use a computer vision model. Most of us were new to model training. We learned how much of the work is in the dataset, from collecting and labeling to waiting on processing, and how to combine a custom Roboflow model with on-device pose tracking.
How to design for failure. Limited Gemini credits, a late-arriving Pi, and a borrowed webcam and mic taught us to ask what happens when a piece goes missing. That led to fallbacks, request pacing, and a dispatch path that works without the LLM.
How to build fast with limited resources. We improvised around missing hardware and tight time, and came together as a team to turn a 2am idea into a working prototype.
What's next for C-BOS: We See, We Save
Hardware upgrades
- Portable webcam and microphone, built for the setup we originally designed
- 3D-CAD-printed enclosure for a clean, single-device build
- Cooling fan to keep the Pi stable during continuous inference
- Raspberry Pi 5 with a GPU accelerator, so more video processing runs on-device
Beyond the home
- Deploy in public spaces like transit stations, campuses, and parking garages, where someone could collapse with no one nearby
- Send first responders and police dispatch a verified alert with the camera’s location and keyframes
- Turn existing cameras from after-the-fact recordings into a faster path to help
Doing it responsibly
- Skeleton-only views by default, with footage released only after a confirmed emergency
- A human dispatcher in the loop
Built With
- base64
- deep-agent
- fastapi
- gemini-4-hydrogen
- geminiapi
- langchain
- python
- raspberry-pi
- roboflow
- ultralytics-yolo26
- ur-fall-detection-dataset-(kwolek-&-kepski