Drishyam

An OpenCV 5 inspection agent for large bridge photos: it chooses where to look, zooms where its segmenter is unsure, and hands every critical finding to a person to sign.

Comment

Try it in 40 seconds: open the live demo and press Watch it inspect a bridge. It runs live on AWS Lambda. Also: code, technical report, film (4:37).

Inspiration

A bridge inspection leaves an engineer with thousands of high-resolution photos to check by eye, and most of each photo is sound concrete. We wanted software that looks where damage is likely, explains every look, and never signs off on anything dangerous by itself. Drishyam is Sanskrit for "that which is seen".

What it does

Drishyam is an inspection agent that works like a pan-tilt-zoom camera inside one large photo.

  1. Perceive. It blurs faces, people and vehicles first. Then it builds an image pyramid and a saliency map with OpenCV 5 to find places worth a look.
  2. Decide. It takes the look with the best expected value per millisecond. It zooms in where its segmenter is unsure, takes a confirm look at a weak critical finding, and stops when the next look is worth less than it costs or its pixel budget runs out.
  3. Act. It segments each look (cracks, spalling, exposed rebar and more) with our dacl10k-trained model on OpenCV 5's new DNN engine, and drafts the findings.
  4. Hand off. Critical findings (exposed rebar, large spalls) wait for a person. Only a person can approve or reject them, and only a person can sign off an area the agent could not assess. The agent, the optional LLM planner and the MCP tools have no path to either. The work order is HMAC-signed.

Every step writes its reason into a live trace, and KILL stops a job before its next look. Claude can also drive the decisions over MCP (12 allow-listed tools), and it still cannot approve anything.

How we built it

  • OpenCV 5.0.0: dnn with ENGINE_NEW for the ONNX segmenter, CV_Bool masks for copyTo (new in 5.0), YuNet and NanoDet-Plus for the privacy blur, warpPerspective for the virtual camera, and morphology and contours to measure each defect.
  • AWS: Lambda arm64 (Graviton) runs the agent behind CloudFront and WAF and streams its trace. DynamoDB holds the kill switch and Secrets Manager the signing key. CloudWatch, X-Ray, SNS and a budget watch it. It is one CDK stack with a smoke test.
  • Evaluation: each rule was committed before its run, each configuration was scored once on the held-out sets (earlier reads are disclosed), and recall comes with paired bootstrap confidence intervals and a chance baseline.

Results (honest)

On 200 real CODEBRIM bridge photos:

  • It finds 0.49 of the labelled defects (95 % CI 0.44–0.54), 2.5× the classical engine. Boxes in random places already find 0.30, and a plain tiling grid finds 0.50.
  • It puts a box on 0 of 51 photos with no damage.
  • Under the live policy a person signs about 1.15 critical findings per photo instead of 2.17, with recall unchanged.
  • It rarely finds exposed rebar (0.14).

Agent control: every step carries a cause, and 40 of 40 planner drills and 10 of 10 governor drills recovered. Live, a step's p95 has been 0.52–1.14 s.

Challenges we ran into

  • OpenCV 5's classic DNN engine cannot import our model's dynamic shapes, so the live site runs ENGINE_NEW. This finding and a speed finding are written up for upstream in bench/repro/README.md.
  • Honest evaluation. Chance baselines showed how much recall large boxes get for free, and a plain tiling grid matched the agent's recall.
  • The COOL AMI could not be subscribed from our account, so COOL was not run.

Accomplishments that we're proud of

  • A live perception-decision-action loop in which every step says why.
  • A human gate the agent cannot bypass.
  • Published negative results: an LLM router, a backlog rule and a blur gate were measured and not adopted.

What we learned

Measure against chance and the simplest baseline before believing a gain, and commit the rule before the run.

What's next for Drishyam

Measure crack width at full resolution (today the width is the segmenter's mask), and run the faster classic engine once its import is fixed upstream.

Built With

Share this project:

Updates

Submission history