Argus - Agentic Video Triage on OpenCV 5 + AWS

Inspiration

Security, retail, and compliance teams drown in video. A single 8MP camera produces more footage than any human can review, so almost everything is watched passively. The bottleneck is not storage or detection - it is the expensive, reasoning-bound step of deciding which frames deserve scrutiny at all.

Argus inverts that. Instead of running a heavy detector on every frame, it uses OpenCV 5 to cheaply rank frames by "how much is happening here", and spends the expensive model only on the top candidates. An agent then reasons over the selected evidence and verifies each candidate event against an explicit policy.

What it does

1. Frame selection (OpenCV 5). Cheap motion, edge, saliency and colour-anomaly signals rank candidate frames so the expensive stage sees the top-K instead of the whole stream.
2. Agentic verification. A tool-using agent reads each candidate, reads the policy, and decides: confirm, dismiss, or escalate.
3. OpenCV output drives the agent. The numeric signals from OpenCV 5 change what the agent does next: they decide which frames are fetched, which measurement tools are called, and which policy gates can pass. Remove the OpenCV stage and the agent has nothing to act on.
4. Human control. Only escalate cases reach a human, with the evidence bundle (frames, reasons, confidence) attached - the agent never guesses on low confidence.
5. Observability. Every decision is logged with its evidence, policy gates and tool trace, so each result can be reconstructed after the fact.

How we built it

OpenCV 5 handles ingestion, frame selection, and measurement on video. Inter-frame motion (difference plus Farneback dense optical flow), Canny edge density, saliency from OpenCV contrib, and an HSV histogram colour-anomaly score are combined into a weighted ranking with a temporal-diversity constraint, so a burst of near-identical frames cannot consume the whole budget.

The agent calls a small set of typed tools (clip metadata, fetch frame, measure motion/edges/saliency, read policy) rather than free-form browsing, so every action is auditable. Four gates - motion_present, salient_subject, illumination_usable, temporal_persistence - are combined into a confidence. High confidence confirms, low confidence dismisses, and the middle band escalates. The default reasoner is a deterministic heuristic so CI and evaluation are reproducible; an optional LLM reasoner drives the same tools through an OpenAI-compatible endpoint.

Deployment is reproducible on AWS: S3 for evidence, ECS Fargate for the API, Lambda for event-driven triage, CloudWatch for metrics and logs, provisioned with Terraform. A FastAPI endpoint exposes the pipeline and backs the demo UI.

Challenges we faced

• Getting frame selection to preserve recall while cutting cost - naive top-K by motion drops slow-onset events. We mitigated with a diversity constraint so selected frames are not all from one burst.
• Making the agent's autonomy bounded and legible. We kept tools narrow and made confidence thresholds explicit, and added a hard rule that a missing-motion gate can never auto-confirm.
• Comparing systems fairly. We report precision/recall/F1, selection recall, missed event rate, cost per stream-hour, and latency together, including failure cases.

Accomplishments we are proud of

The escalation-first design: the agent is useful precisely because it stops and asks. We can show measured evidence of reduction, recall and cost from the eval harness, and every decision ships with the trace that produced it.

What we learned

Spend compute where uncertainty is, not uniformly. Most of the engineering value was in the selection and evaluation layers, not the detector.

What's next

• Learned selection policies instead of hand-tuned ranking
• Broader benchmark set with slow-onset and occlusion cases
• Finer-grained escalation policies per deployment

Built With

• amazon-web-services
• computer-vision
• docker
• llm-agents
• observability
• opencv
• opencv-5
• python