VisionTwin AI

Agentic Visual Digital Twin for Active Inspection

Comment

Inspiration

Most "AI camera" products fall into one of two traps. Either they stream every frame through a detector and dump a wall of "person detected" / "box detected" alerts on an operator who eventually just tunes them out — or they hand the decision to a black-box model that silently auto-resolves things with no explanation and no human in the loop. A blocked emergency exit and a box nudged two inches on a shelf get the exact same treatment: a bounding box on a screen.

We wanted to build something closer to how a good human inspector actually works: glance at the whole floor, notice something's off in one corner, walk over, look closer, and only then decide whether it's worth raising an alarm — and be able to explain that decision afterward. That's the idea behind VisionTwin AI: an agentic visual digital twin that actively chooses where to look next, instead of treating every pixel as equally important.

What it does

VisionTwin AI watches a scene (warehouse-style, in this build) and maintains a live digital twin — objects, zones, risk state — that updates every cycle. On top of that twin runs a deterministic, fully-auditable agent loop:

  1. Perceive — classical OpenCV 5 vision (background subtraction, contour analysis, centroid tracking, zone mapping) builds a picture of what's in the scene and where.
  2. Compare — the live scene is diffed against the last persisted twin snapshot to find what changed.
  3. Decide where to look — a transparent, weighted priority score (magnitude, zone risk, detector confidence, object importance, incident history — no black box) ranks every candidate change.
  4. Inspect — only the top-priority region gets a deeper, more expensive OpenCV pass (Canny edge density + SSIM) to confirm or reject it.
  5. Act — confirmed events become an audited incident with a rendered evidence JPEG and an operator notification for high/critical severity, gated by a responsible-AI rule that never lets the agent auto-resolve a critical incident on its own.
  6. Update the twin — the baseline snapshot refreshes every cycle, whether or not anything happened.

Every state transition is persisted as a replayable event — never raw model chain-of-thought — so a full run can be inspected end to end through GET /api/v1/agent/traces.

How we built it

  • Vision layer — OpenCV 5, no shortcuts. Every stage is real, working classical CV: MOG2 background subtraction + contour analysis for detection, greedy nearest-centroid tracking, point-in-polygon zone mapping, absdiff + a from-scratch Gaussian-window SSIM (Wang et al. 2004) + dense Farneback optical flow for comparison, and Canny + SSIM for deep inspection. No trained deep-learning detector is used or claimed — that's a deliberate scope choice, documented rather than hidden.
  • Agent layer — a deterministic state machine, not a prompt loop: OBSERVE → ANALYZE → CHANGE_DETECTED → ASSESS → SELECT_REGION → INSPECT → REASSESS → DECIDE → ACTION → UPDATE_TWIN → COMPLETE, driving 8 explicit, named tools (compare_scene, inspect_region, calculate_inspection_priority, capture_evidence, create_incident, notify_operator, update_scene_state, request_human_confirmation).
  • Priority formula, out in the open: priority = 0.30·magnitude + 0.20·zone_risk + 0.20·confidence + 0.10·object_importance + 0.10·history_score, dampened for ambiguous change types — so the agent's next move is always traceable to concrete factors.
  • Cloud layer — AWS as an adapter, not a dependency. S3 stores evidence images (presigned URLs), DynamoDB mirrors incidents/events for durability (explicitly not the system of record — SQLite always is), SNS notifies operators on high/critical severity, CloudWatch carries metrics and logs. A single factory picks AWS or an honest local-mode fallback (files, JSONL, structlog) based on whether credentials are actually configured, so the whole system runs completely offline with zero AWS account required.
  • Backend: FastAPI, 20 REST endpoints across 9 route modules, plus a live SSE event stream.
  • Frontend: React + TypeScript + Vite, a typed API client covering the full backend contract, and 8 pages (Overview, Digital Twin, Agent Trace, Incidents, Evidence, Live Inspection, Scene History, Settings).
  • Demo data: since we didn't have a physical camera or warehouse, vision/samples/warehouse_scene_generator.py deterministically draws synthetic scripted scenes from pure OpenCV primitives. It only manufactures pixels — every detection, track, comparison, and inspection result downstream is computed by the real vision pipeline running against those pixels, nothing is faked.

Challenges we ran into

  • Keeping the "active perception" idea honest rather than cosmetic — it would have been easy to fake a priority score that didn't actually gate anything. Instead the priority formula genuinely determines which single region gets the expensive deep-inspection pass each cycle.
  • Building a responsible-AI gate that's a real code path, not a policy statement: request_human_confirmation hard-routes any critical-severity incident to investigating — there is no code path anywhere in the agent that lets a critical incident auto-confirm or auto-resolve.
  • Zone-localization wasn't perfect: our evaluation harness scores 1.0 precision/recall/F1 on synthetic ground truth, but zone-localization accuracy is 0.80 — the barrier_moved scenario fails to localize to any zone at all three tested resolutions. We chose to report that honestly rather than quietly drop the metric.
  • Time-boxing meant AWS mode is implemented and unit-tested (adapter tests against local/mocked behavior) but wasn't exercised against a live AWS account, and docker-compose.yml was reviewed but not run, in this build environment. We'd rather list those gaps plainly than overstate what's verified.

Accomplishments that we're proud of

  • 51 backend tests passing across agent, integration (API + AWS adapters), unit, and vision suites.
  • Perfect detection precision/recall/F1 (1.0) on our synthetic evaluation harness (18 labeled samples, 3 resolutions × 6 scripted change types) — reported alongside the 0.80 zone-localization gap, not instead of it.
  • Measured, reproducible performance numbers (not estimates): ~150–210 ms/frame end-to-end on a dev machine with no GPU, dominated by dense Farneback optical flow in the coarse diff stage.
  • A genuinely working end-to-end loop — synthetic scene → OpenCV detection → priority-scored inspection target → deep inspection → evidence JPEG → audited incident → notification — all traceable through a real, replayable agent trace.
  • A documentation suite (10 docs, all grounded in actual file paths, line numbers, and re-run numbers) that says what's not done as clearly as what is.

What we learned

That "agentic" is easy to claim and hard to make real. The difference between a system that looks like it's making judgment calls and one that actually is comes down to whether the intermediate decisions (the priority score, the inspection gate, the human-confirmation threshold) are visible, numeric, and replayable — or just narrative dressing on top of a fixed pipeline. We also relearned that classical OpenCV, used deliberately, still gets you a long way before you need a trained model — and that saying so explicitly is more useful to a judge (or a future user) than dressing it up as deep learning.

What's next for VisionTwin AI

  • Run AWS mode against a live account and a real Graviton/Compute Optimizer benchmark.
  • Replace greedy nearest-centroid tracking with a Kalman filter / Hungarian-algorithm assignment for more robust identity across occlusions.
  • Fix the barrier_moved zone-localization gap identified in evaluation.
  • Independently verify the frontend end-to-end against a running backend (it currently aligns with the API contract by construction but hasn't been UI-tested by us).
  • Validate the whole pipeline against real camera footage instead of the synthetic generator — lighting, noise, and ambiguous real-world scenes are the next real test.

Built With

Share this project:

Updates

Submission history