Inspiration
Nursing home staff currently have to balance daily care, scheduling appointments, and unexpected requests for assistance. When staffing is limited, it becomes harder to notice when a resident needs help. HearthWatch came from the idea that timely alerts could help staff respond sooner to possible falls, requests for assistance, and missed appointment check-ins. Our goal is to make those needs easier to notice and coordinate, giving staff more time to focus on residents.
What it does
HearthWatch connects local camera observations and care check-ins to a nurse response workflow:
- Possible Fall Detection: Our own posture model, trained by us (using timed labels from Hugging Face's OmniFall) and running on an NVIDIA Jetson Nano, labels everyone in view as upright, sitting, or on the floor. Prolonged floor tagging notifies a nurse to check in.
- Gesture Help Requests: Recognize a held fist as a request for assistance, with a manual help button as another option.
- Nurse Subscriptions: Let nurses select rooms and alert categories, receive updates, and accept responsibility for a response.
- Appointment Reminders: Notify the care team when a patient's arrival is not confirmed within 15 minutes of a scheduled appointment.
- Camera Health: Flag disconnected sources, stale observations, and missing visibility.
- Simulation and Replay: Explore resident activities and staged incidents in a 3D environment where residents and staff navigate with pathfinding. Real people seen by the Jetson appear as live avatars beside clearly labelled simulated residents.
How we built it
- Dashboard: Built with React, TypeScript, and Three.js for interactive room visualization and simulation controls, alongside a nurse dashboard that receives WebSocket alerts and supports room subscriptions, response acceptance, and resolution.
- Edge Perception: On the NVIDIA Jetson Nano, jetson-inference poseNet (TensorRT) finds every person's keypoints. When someone may be lying down, it adds rotated inference passes, because poseNet sees lying bodies far better when the frame is turned 90°. Our posture model then judges each person in 0.56 ms, alongside res10 face detection, all at about 13 FPS on a 2 GB board. Only structured observations leave the device, never video.
- Our Own Posture Model: Written and trained in PyTorch on an RTX 3060 Ti. Using timed labels from Hugging Face's OmniFall, we built about 100,000 training frames from five public fall datasets (including AI-generated elderly residents), extracting body keypoints with YOLO11-pose. The network is a small neural network (7,683 weights), exported to plain numpy for the Jetson.
- Monitoring: Implemented in Python and FastAPI, using Pydantic to validate observations and temporal state machines to generate alerts from sustained issues. SQLite preserves appointment schedules and overdue reminders, while WebSockets deliver updates to subscribed nurses.
- Simulation and Evaluation: Built a seeded simulation with autonomous resident ing, configurable rooms, and controllable incidents. Synthetic observations passthrough the real monitor and are compared against separate ground-truth timelines to measure detection delay, missed events, and false alerts.
Challenges we ran into
- Camera Placement: Height-based floor rules depended heavily on camera position. Our trained model judges body shape instead of position in the frame, so it only needs the whole body in view.
- Incomplete Body Detection: Lying bodies were difficult for poseNet to capture. Rotated inference passes improved keypoint coverage, and we extended them to everyone in view, not just the tracked person.
- Unstable Predictions: Brief gestures, low-confidence classifications, and missing frames could interrupt detection. We added hold times, confidence thresholds, release conditions, and cooldowns.
- Live and Simulated Clocks: Accelerated playback needed to preserve posture timing while camera health reflected actual arrival times. We separated the clocks and isolated replay runs with unique identifiers.
- Identity Without Overclaiming: Face detection establishes presence, not identity. Appointment check-in uses an explicit token. Recognition (YuNet + SFace) is a separate, opt-in demo with fictional profiles.
Accomplishments
- Training our own posture model that, on a locked test of 10,333 frames from pemacro F1 of 97.0% (weighted 97.5%):
- Floor: precision 97.9%, recall 96.1%, specificity 99.3% (a 0.67% false positive rate)
- Upright: precision 98.4%, recall 98.6%
- Sitting: precision 95.5%, recall 95.7%
- On our own camera it catches 96% of floor frames.
- Running it live on a 2 GB Jetson Nano for every person in view, with the panel warning when anyone is on the floor.
- Connecting observations to a complete alert -> nurse acceptance -> resolution workflow
- Bringing the live Jetson feed into the 3D simulation, so real and simulated residents share one room.
- Supporting fist help requests even when the posture model cannot detect a body
- Adding persistent appointment reminders and a clearly labelled 15-second demonstration mode.
- Evaluating the real monitor without giving it access to simulator ground truth
What we learned
- Temporal Reasoning: Useful alerts depends on evidence over time.
- Honest Metrics: Splitting by person and locking a test set matters. Validation that shares rooms and cameras with training will overstate accuracy.
- Shared Contracts: Pydantic models and generated TypeScript types helped indepeagree on their inputs and outputs.
- Uncertainty Handling: Camera silence, occlusion, and weak detections must remain visible rather than imply safety.
- Evaluation Design: Repeatable scenarios make failures easier to investigate, iat can still trigger a possible-fall alert.
What's next for HearthWatch
- Motion-Aware Model: A model that reads several frames at once, to tell falling to handle close-ups and partial views.
- Broader Testing: Evaluate more camera positions, lighting conditions, mobility aids, and real older adults rather than young actors staging falls.
- Appointment Interface: Add a calendar and accessible resident check-in control
- Resident Feedback: Make it clearer when a help request is received and a nurse is responding.
- Access and Retention: Add authentication, protected room access, and configuraing real resident records.
- Expanded Evaluation: Report detection delays, missed events, and false alerts across a larger set of scenarios.
Built With
- bash
- cuda
- fastapi
- huggingface
- javascript
- jetson-interface
- matplotlib
- numpy
- nvidia-jetson
- onnx
- onnxruntime
- opencv
- posenet
- pydantic
- python
- pytorch
- react
- sqlite
- tensorrt
- three.js
- typescript
- ultralytics
- vite
- websockets
- yolo
Log in or sign up for Devpost to join the conversation.