FallLoop

OpenCV 5 spots a person who may have fallen; an agent decides what to look at next; a human makes every call.

Comment

Inspiration

When someone falls at home and can't get up, simple camera alerts can miss it, or raise so many false alarms that nobody trusts them. Someone lying down to sleep looks a lot like someone who fell. I wanted a vision system that acts on its own uncertainty (look again, wait, or ask for a better view) and then hands every important call to a person instead of deciding alone.

Who it is for

The intended user is a caregiver or a care-home staff member who watches several rooms at once and cannot watch every camera. FallLoop does not replace that person. It shortens what they have to look at to a review queue with a face-blurred crop, a reason and a timeline. This is a research demonstration, not a medical device, and it has not been tested with care teams yet.

What it does

FallLoop watches a video and finds people with OpenCV 5. It estimates posture and motion, and flags anyone who appears to be lying still. An agent closes the loop:

  • when nobody is found, it re-thresholds the detector;
  • when a person is poorly seen, it re-detects on a zoomed region;
  • when evidence is borderline, it waits for more frames;
  • when it is still unsure, it asks for a new view;
  • when a possible fall persists, it escalates to the human review queue.

A dashboard shows the annotated video (faces blurred), the agent's timeline and the queue. Only a human can confirm ("needs help now"), downgrade ("check later") or dismiss ("false alert").

How I built it (OpenCV 5 at every step)

  • Decode: cv2.VideoCapture
  • Motion: cv2.calcOpticalFlowFarneback on a half-resolution grey frame, normalised to body-heights per second
  • Pose detection: cv2.dnn.readNet + cv2.dnn.blobFromImage (640×640 letterbox) running a YOLO26n-pose ONNX model; the output tensor (1, 56, N) carries box, score and 17×(x, y, visibility)
  • Filtering: confidence filter + cv2.dnn.NMSBoxes; an IoU tracker keeps identities
  • Zoomed re-detection: crop + cv2.resize + a second cv2.dnn forward pass, kept only if keypoint visibility improves
  • Privacy: head-region blur with cv2.GaussianBlur before anything is stored or shown
  • Output: cv2.VideoWriter (H.264 locally; WebM/VP8 on Lambda, which has no ffmpeg)

On AWS (SAM template, passes cfn-lint, deployed in us-east-2):

  • Uploading an .mp4 to a private S3 input bucket triggers the processing Lambda.
  • That Lambda is a python3.12 zip on arm64 (Graviton), 2048 MB and 300 s, bundling the OpenCV 5.0.0 aarch64 wheel and the ONNX model. It runs the exact same run_pipeline.run() as the laptop.
  • A dashboard Lambda behind a Function URL serves media through 10-minute presigned links, so the buckets stay private.
  • Least-privilege policies, object expiry, a USD 1 budget alert and aws/teardown.sh keep it safe and cheap.

Live dashboard for judges: https://shuwf5nbecmvl633bee4crfkie0nziur.lambda-url.us-east-2.on.aws/

How the agent decides (vision results drive the next action)

On every processed frame (about 10 fps), the OpenCV 5 pipeline hands the agent one observation per person:

  • posture (lying, sitting or standing, from the torso angle between the shoulder and hip keypoints);
  • keypoint visibility;
  • motion inside the box.

The agent answers with one of six tools, and each tool changes the next OpenCV call:

  • set_conf (triggered when nobody is detected): lowers the cv2.dnn confidence 0.35 → 0.30 → 0.25 → 0.20 (floor); restores it when the person is found again
  • roi_redetect (triggered when keypoint visibility < 0.5): crops an expanded box, upscales, re-runs cv2.dnn; kept only if visibility improves (max 2 tries)
  • observe_more (triggered when lying and still, not yet persistent): waits; persistence doubles (0.5 s → 1.0 s) when motion is borderline or keypoints are weak
  • request_new_view (triggered when still weak after ROI tries, or nobody at the floor threshold): asks the operator for a better angle
  • escalate_to_human (triggered when lying and still, persisted): creates a pending_human item ("possible fall - check now?") with a face-blurred crop
  • annotate_review_item (triggered when the person is later seen upright and moving): adds a note for the reviewer; the item stays open

A real trace from the demo run (demo/run/incident.json):

  • At t=4.1 s, keypoint visibility of 0.35 triggers roi_redetect, and the result raises visibility from 0.41 to 0.59.
  • When visibility stays weak, request_new_view follows.
  • At t=6.3–6.5 s nobody is detected, so the agent steps the threshold down to 0.20. It restores 0.35 at t=7.1 s when the person reappears.

Evaluation (pre-registered, honest)

Data. GMDCSA-24 (Alam et al., Data in Brief 2024, CC BY 4.0), a deterministic subset of every third clip: 57 clips, 28 falls and 29 activities of daily living. Five of the daily-activity clips include lying down to sleep.

Protocol. The split is by person. Subjects 1–2 were used only to pick the stillness threshold (0.15 body-heights/s). Every result below comes from subjects 3–4, which were never tuned on. In hindsight that threshold is not the best one on TEST: 0.02–0.05 or 0.20 would have scored higher. I report the pre-registered choice anyway.

Results on held-out actors

  • Clip level, "lying + still → human review": precision 0.79, recall 0.73, F1 0.76 (TP 11, FP 3, FN 4, TN 10)
  • Same alerts judged as fall detection only: precision 0.64, recall 0.69. Sleeping on a bed also gets flagged, which is exactly why a person reviews every alert
  • Frame level, lying: precision 0.94, recall 0.84
  • Frame level, lying + still: precision 0.95, recall 0.79
  • Time from labelled fall start to escalation: median 1.61 s (alerted falls, n=23, all subjects)
  • Recall in low light (all subjects): 7 of 8 positive clips
  • Recall with occlusion (all subjects): 2 of 3 positive clips

The low-light and occlusion groups are too small to conclude anything.

Agent behaviour (all 57 clips, 4,301 processed frames)

  • The agent acted on 15.8% of frames: 435 threshold changes, 197 observe-more holds, 22 new-view requests and 81 zoomed re-detections.
  • 39 of the 81 re-detections (48%) improved keypoint visibility.
  • The agent never produced a final label. Every escalation is stored as pending_human.

Speed

  • CPU benchmark (shared x86, 8 vCPU, 6 fixed clips, 1280×720): 87.9 ms per frame with 4 threads (about 11 fps), 103.9 ms with 2, 152.4 ms with 1
  • Biggest stages on that benchmark: cv2.dnn forward ~44.5 ms, Farneback ~35.7 ms
  • AWS Lambda arm64 (Graviton, 2048 MB, 2 threads), one run of the 23-second demo video: 2.64 processed fps end to end, cv2.dnn forward ~295 ms per frame, 337 MB peak memory, the same 3 review items as the local run

The Lambda is slower than the x86 box. Memory and thread sizing on Graviton is the next thing to tune. The full evaluation ran on x86 CPU.

Reproduce: python evaluate.py --run --bench --report && python tools/eval_to_md.py

Failure cases I show on purpose

  • The detector loses people once they are on the floor. 19.7% of fall-clip frames had no detected person, against 3.1% in daily-activity clips. In the 5 missed falls, the agent changed the threshold 7–21 times and asked for a new view in 3 of them, but each clip ended before stable evidence returned.
  • Push-ups, bending and picking things up off the floor get flagged as lying.
  • No bed-versus-floor context.
  • Threshold flapping when detection flickers. Hysteresis is the planned fix.
  • Only one person per clip was tested.

How FallLoop maps to the judging criteria

  • Technical execution (30%): OpenCV 5 end to end (list above); the same pipeline locally and on Graviton; pinned dependencies; held-out evaluation
  • Innovation (20%): vision output drives the agent's next action, and the agent cannot close a case
  • Real-world impact (20%): home-fall review for caregivers, measured on a home-fall dataset; failure cases documented
  • UX (10%): one-page dashboard: video, clickable timeline, queue, three reviewer buttons
  • Documentation (10%): README, TECH-REPORT, EVAL report, architecture and agent diagrams
  • Cloud, reproducibility, responsible use (10%): S3 + Graviton Lambda via SAM, budget alert, teardown, data expiry, face blur, "not a medical device"
  • Agentic: OpenCV 5 + agent integration (30%): six tools, each changing the next cv2.dnn call
  • Agentic: orchestration and autonomy (25%): threshold stepping and restore, zoomed retry, wait, new-view request, escalation
  • Agentic: task effectiveness and evaluation (20%): clip and frame metrics on held-out actors; 48% of zoomed retries helped
  • Agentic: failure handling, observability, security, human control (15%): full JSON trace, pending_human only, private buckets, presigned links
  • Agentic: UX, docs, demo (10%): live AWS endpoint plus video

Responsible use, observability and security

  • This is a research demonstration, not a medical device. It makes no clinical decision.
  • No face recognition: faces are blurred before anything is stored or shown. No real patient footage was used.
  • There is a full JSON trace of every decision and tool result, plus per-frame observations, for audit.
  • The system cannot write a final decision. Only the reviewer's buttons close an item, and those decisions are stored in reviews.json.
  • Cloud data sits in private buckets behind presigned links and expires automatically. A budget alert and a one-command teardown are in place.

Challenges

  • Detecting people lying at unusual angles.
  • Keeping the agent from flapping between thresholds.
  • Evaluating honestly on staged clips without tuning on the test actors.

What I learned

Visual evidence is most useful when the system can act on its own uncertainty, and say so, instead of hiding it.

What's next

  • Hysteresis for set_conf.
  • Bed-versus-floor scene context.
  • Rooms with several people.
  • Memory and thread sizing on Graviton.
  • An agent-on versus agent-off comparison on the same test split.
  • Testing with care teams.

Built With

Share this project:

Updates

Submission history