Ergo — Devpost submission

Tagline: Passive CCTV watch that only wakes up when backs bend: cheap OpenCV pose gating a Strands + Bedrock agent that confirms risk and stores proof.

Demo video: (link the rendered MP4 here once uploaded) Stock footage + Deepgram narration carry the story; every UI shot is the real app, captured live with Playwright — upload a clip, the agent flags a bent-back lift, and the proof lands in the Flagged clips tab.

Inspiration

Back and spinal injuries don't come from the lifts everyone watches. They come from the routine ones — the thousandth box of a shift, bent over at 35 degrees, with nobody looking. The cruel tradeoff in workplace safety: watching every frame with a vision-language model is prohibitively expensive, and watching none of them misses everything.

Ergo exists to break that tradeoff. Stay passive and cheap almost all the time. Wake up — and spend money on intelligence — only in the seconds a human body looks at risk. A back got hurt on a lift exactly like this one; we built the system that should have been watching.

The science backs the premise: Coenen et al. showed that cumulative mechanical low-back load at work is a determinant of low-back pain, with a clear exposure–response relationship — workers with the highest cumulative lumbar torques faced roughly 2.4–5.1× the risk (Coenen, Kingma, Boot, Bongers & van Dieën, 2014, Occupational and Environmental Medicine, 71:332–337; https://pubmed.ncbi.nlm.nih.gov/24676271/). It is the thousandth routine lift, not the single dramatic one, that wrecks backs over a career. That is why Ergo counts every bend instead of waiting for an incident report.

What it does

Ergo ingests CCTV footage (files or RTSP streams) and watches every frame with a lightweight pose model. It computes trunk, knee, and shoulder angles plus RNLE horizontal/vertical proxies, and scores each moment 0–1. The moment risk crosses 0.65, a Strands agent wakes up: it pulls the pre/post-roll clip, asks a Bedrock multimodal model to take a second look, and then makes a call — STORE the clip to S3 with full metadata, request HUMAN_REVIEW, or DISCARD with a miss log. Every single step lands in trace.jsonl, so any decision can be audited back to the exact frame that triggered it.

The demo path, end to end: upload a warehouse clip → the pipeline scores a forward bend (OpenCV 0.694, trunk 35.1°) → the VLM confirms bending_risk 0.72 → a HUMAN_REVIEW event fires → the captured clip appears in the Flagged clips tab with the trace attached.

Privacy is architectural, not a policy promise. Ergo wires directly into existing CCTV/RTSP feeds, and the system is passive by default: no human ever watches the streams, and the VLM — the only component that truly "sees" — is only ever shown the handful of seconds around a detected infraction. Clips are stored only on STORE or HUMAN_REVIEW; anything DISCARDED leaves behind nothing but a miss log. Anonymized or low-confidence views are downweighted and escalated rather than guessed at. The cameras are already on the walls — Ergo just makes sure nobody has to watch them until a back is actually at risk.

How we built it

• Perception (passive, cheap): YOLOv8-pose through OpenCV 5's cv2.dnn.readNetFromONNX with a letterbox + NMS decoder (~6 ms/frame). Skeletal angles and RNLE proxies feed the 0–1 risk score.
• Judgment (active, rare): Strands Agents @tools plus BedrockModel (Claude/Nova multimodal). The agent only ever sees the seconds that matter.
• Storage & UI: boto3 for S3/Bedrock, FastAPI judge UI with Run Inference, Flagged clips, and trace views.
• Evaluation: SAFELIFT (Zenodo 18243098, CC BY-NC-SA 4.0) — estimated H vs annotated H (MAE), flag vs LI-risk (precision/recall/F1), end-to-end latency.
• Demo realism: a mock backend keeps the whole loop runnable with zero AWS cost, and the demo video's UI footage is the actual app driven by Playwright — a real upload producing a real HUMAN_REVIEW event, not a mockup.

Challenges we ran into

• OpenCV 4.x can't parse YOLOv8's Split node. That single missing operator forced the move to OpenCV 5 — no workaround, just a version wall.
• Anonymized and oblique views degrade keypoints. Rather than pretend the data is better than it is, low-confidence detections are downweighted and escalated to a human. Uncertainty became a feature of the policy, not a bug report.
• 2D horizontal distance from one camera is approximate. The VLM second look exists precisely to resolve that ambiguity — the agentic design is the answer to the perception limit.
• The demo fought back. Dogfooding the video pipeline exposed three real bugs — an absolute review-clip path that broke outside the project root, a UI poller that dropped "queued" jobs, and a drawer crash on a missing canvas — all fixed in the actual codebase before a single frame rendered.

Accomplishments that we're proud of

• Real ONNX pose running on real lifting footage, at ~6 ms/frame.
• A three-way agent policy (STORE / REVIEW / DISCARD) verified end to end, with a full trace from OpenCV output to final action.
• An honest demo: the video's numbers (0.694, 35.1°, 0.72) are the pipeline's actual outputs on the demo clip, not scripted values.
• Reproducible by default — pinned dependencies, Dockerfile, infra stub, and a mock backend so anyone can run the loop with zero AWS spend.

What we learned

• Gate expensive intelligence behind cheap perception. The passive-then-active pattern cut VLM calls to only the moments that matter — the architecture insight everything else hangs on.
• Design for degraded input. Anonymized footage taught us that confidence handling and human escalation aren't edge cases; they're the product.
• Demo your system, don't mock it up. Capturing the real UI caught real bugs that unit tests missed. If the demo can't survive contact with your own pipeline, the pipeline isn't done.
• Traceability is a safety feature. In a system that judges worker behavior, trace.jsonl isn't logging — it's the audit trail that makes the system's decisions defensible.

What's next for Ergo

• ByteTrack multi-person tracking for busy lines with overlapping workers.
• On-camera RNLE lifting-index display, so supervisors see risk live.
• A COCO-style Graviton benchmark to prove the $1k-per-site deployment path.
• A pilot with an EHS (environment, health & safety) team on a real line.

Built with: OpenCV 5, AWS (Bedrock, S3, EC2), Strands Agents, FastAPI. Dataset: SAFELIFT, Zenodo 18243098 (cite IUI'26).

Built With

• amazon-ec2
• fastapi
• s3