Care Ladder Vision

OpenCV spots the moment; a fail-closed care ladder confirms, waits, then escalates with privacy-first audit trail.

Comment

Inspiration

After several falls earlier this year, we needed a way to keep an eye on my mother without invading her privacy.

I built Care Ladder.

We put a camera in her home, but it didn't help much. She hated being watched, and neither of us could watch a feed 24/7 from opposite sides of the world. She also often forgets to charge her phone. More than once that ended with panicked calls to neighbors, asking them to check on her because she was out of camera view and not answering.

In the past six months she has fallen four times. Once she was hurt badly enough that she had to drag herself out of the bathroom to reach a phone.

The camera was never the problem. The problem was that nobody could act on what it saw. Care Ladder turns the camera from a feed someone has to watch into something that checks in first. OpenCV notices stillness, a sudden fall, or a covered lens. Before anyone is paged, a voice check-in in the room asks if she's okay, so it works even when her phone is dead. If she says she's fine, it stands down and nobody gets a frightening call. If she's silent, it pages the family chat, and any of us can acknowledge from anywhere. If nobody responds, it moves on to the next person. Every step is logged, so we can see exactly what happened and why. Clips are blurred before they leave the device, because she should never feel watched.

It is not a medical device and it will not call emergency services. It helps a family spread across three time zones know their mum is okay.

One in four adults over 65 falls each year. For people living alone, the dangerous fall is the one nobody sees. Families often face a false choice: do nothing, or leave a camera feed that feels like surveillance.

Care Ladder is a third path: a camera that checks in before it escalates, and proves every step on an audit timeline. Built for the OpenCV AI Competition 2026 (Agentic Vision path), team NOTATEAM (Anoop Pamu).

What it does

OpenCV 5 perception emits structured cues (no_movement, no_visibility, camera_occlusion, distress_heuristic). An orchestrator walks a configurable YAML escalation ladder:

  1. Re-perceive (confirm the cue; a covered lens holds as camera-health, not distress)
  2. Smart-speaker check-in ("Are you okay?" or "Could you clear the lens?")
  3. Wait / listen (Acknowledge can stand the ladder down mid-flight)
  4. Notify caregiver / family BotThread (Telegram inform card with 1|2|3, or FakeTelegram stub; console Acknowledge is a secondary ack; inform-only on occlusion)
  5. Dial primary caregiver (stub)
  6. Dial secondary (stub)
  7. Emergency rung (fail-closed by default; even enabled, code audits only and never places a real 911 call)

Agentic Vision evidence: OpenCV cue → ladder → Telegram inform → family ack. The same stillness cue resolves with no dial on a clear_ok reply (Path A, including soft OK like "don't worry"), escalates on silence (Path B) to a BotThread / Telegram family chat mirror, and stands down on family or console ack. It jumps to a human on needs_human (mixed hurt never invents OK). A covered camera is a different cue (camera_occlusion) and a different tool path: ask to clear the lens, inform the caretaker, never claim distress. Every jump is logged with from_index / to_index / reason on the caregiver timeline.

Family Telegram (confirm-before-escalate, not the vision claim): after speaker silence the family gets a BotThread inform card (inline 1|2|3 or reply 1/2/3). Any family or console ack stops dial. OpenCV cues still choose the next tools; Telegram is the family page after the speaker, and without TELEGRAM_BOT_TOKEN the demo stays on an honest stub / chat mirror.

Privacy is enforced at attach: blur or silhouette runs before any pre-event frame is stored or served. Raw identifiable video never leaves the device path. Fall detection frames are intentionally privacy-blurred in the product and in the demo video stills.

Adaptive schedule learning (honest scope): the ladder learns a household's usual day from closed-incident histograms (RoutineProfile: rapid, then settled after about ten confirmed-OK days, freezable) and adapts the stillness timeout with a timeline-visible explain line. This is schedule-baseline learning only. It is not a neural net, not a risk score, not clinical, and it never auto-enables the emergency rung or changes rung order.

How we built it

OpenCV 5 (substantive analysis)

  • Person detection: MediaPipe person-detector ONNX via cv.dnn (OpenCV Zoo), with OpenCV 5 portability fixes
  • Fall signature: BlazePose ONNX keypoints → torso angle + hip height; sudden vertical→horizontal = distress; gradual transitions stay on the stillness ladder (two-tier design)
  • Presence / zones: Otsu, contours, pointPolygonTest
  • Motion: MOG2 background subtractor + absdiff fallback
  • Privacy: Gaussian blur / silhouette before persist; imencode for caregiver clips

Agent + AWS

  • FastAPI caregiver console and incident API
  • The AWS path is live, not sketched: CloudFront serves the console over HTTPS in front of an ALB; ECS Fargate (X86_64) runs the OpenCV 5 DNN pair and FastAPI as a self-healing service from an ECR image (scan-on-push); DynamoDB is the durable incident store behind the audit timeline; S3 holds only privacy-transformed (silhouette) pre-event clips with public access blocked; EventBridge carries every cue on a dedicated bus with a CloudWatch archive rule; GitHub Actions runs tests plus real-footage evals, then rolls the service, so a push to main is a verified deploy
  • Demo honesty: stub dialer, scripted speaker, reserved NANP fiction numbers (NPA-555-01XX), no medical or accuracy claims beyond the labeled harness

Challenges we ran into

  • OpenCV 5 DNN graph differences vs 4.x wrappers (input API, output blob order, NMS index shape)
  • Separating sudden falls from pedestrian bend-overs without false distress cues
  • Gradual collapses: pose unreliable on a motionless subject, so stillness timeout must carry the second tier
  • Keeping emergency fail-closed in plan and code while still showing the full ladder to judges
  • Adaptive learning: badge number must equal the actual detector arming threshold (spec section 6)

Accomplishments that we're proud of

  • Eval honesty first: the labeled harness is n=11 cases (synthetic fixtures including covered-lens occlusion, plus three real-footage cases: one hard fall, one gradual collapse, one pedestrian negative). On that small set the cues behaved as designed (recall 1.00, no false escalations, mean time-to-confirm 18.0 s incl. real clips); that is directional evidence at demo scale, not a clinical validation, and we say so on the timeline and in docs/failure-modes.md
  • Real KU Leuven fall re-enactment fires distress_heuristic / sudden_vertical_to_horizontal; gradual bed collapse escalates via stillness; OpenCV vtest pedestrians do not fire distress
  • Live HTTPS demo on AWS with privacy-only clips and a full audit timeline
  • Confirm-before-escalate UX that families can understand in one click (Path A / Path B fixtures)

What we learned

  • Agentic Vision is not "LLM narrates a frame." It is: perception changes the next tool or human gate, and the change is observable.
  • Fail-closed emergency and reserved demo numbers are features judges can trust, not footnotes.
  • Schedule learning helps timeouts stay household-realistic only if it stays explainable and never touches safety gates.

What's next

  • Continuous on-device camera path (demo today is upload / fixture oriented)
  • Real telephony behind Secrets Manager with explicit live grant (out of demo scope)
  • Optional COOL / Graviton Arm path with measured baselines (not claimed in this entry)
  • Broader real-footage eval beyond the current demo-scale set

Built With

  • amazon-alb
  • amazon-cloudfront
  • amazon-dynamodb
  • amazon-ecr
  • amazon-ecs
  • amazon-eventbridge
  • amazon-fargate
  • amazon-web-services
  • audit-logging
  • computer-vision
  • docker
  • fastapi
  • github-actions
  • mediapipe
  • onnx
  • opencv-5
  • pose-estimation
  • privacy-preserving-vision
  • python
  • s3
  • yaml
Share this project:

Updates

Submission history