CrackScout

An inspection agent for concrete on site. OpenCV 5 finds and measures cracks, the agent decides clear, reshoot, file or escalate, an engineer approves. On AWS Graviton with COOL.

Comment

Inspiration

I work on tower cranes in Prague. Cracks in concrete are part of every site: the crane foundation, anchor blocks, walls that went up last week. The way they get checked is someone holds a crack comparator card to the wall, takes a phone photo and sends it to the engineer in a chat. Half the time nobody writes down the width. I wanted the boring part done the same way every time, with the engineer still making the call.

What it does

A worker photographs the surface on a phone and opens the page. An agent runs OpenCV 5 tools on the photo in a loop. First it checks whether the photo can be judged at all. Then it looks for crack-like structures and measures them: length, width (median, 95th percentile, maximum), orientation, how dark the crack is against the surface. If it is unsure about the best candidate it zooms in and looks again. Then it ends with one of four actions.

Clear: nothing holds up. Reshoot: the photo is blurred, badly lit, or a candidate could not be confirmed, so take a closer photo of this region. File: cracks exist but are below the action width, keep the record. Escalate: a crack is at or above the action width (0.3 mm by default when the photo has a scale), stop and wait for the engineer, who approves or rejects in the same page with a note.

Every tool call is in the trace with its inputs, outputs and milliseconds. The engineer sees the annotated image, the crack table and why the agent decided what it decided.

How we built it

The vision is plain OpenCV 5, no training. CLAHE, a Gaussian, then the maximum of morphological black-hat transforms at four kernel sizes, which picks out dark lines from hairlines to wide cracks while the shading of the surface cancels out. The threshold is robust (median plus four MADs of the response) with hysteresis so fading cracks stay connected. Components pass an elongation test, get thinned with Zhang-Suen to a skeleton for the length, and the width comes from a distance transform at half maximum along the skeleton. With a scale the widths are in millimetres and get a severity band after ACI 224R-01.

The agent has four tools (assess_quality, detect_cracks, zoom, finish) and two planners that use the same tools: a deterministic rule policy, and Amazon Nova Lite on Amazon Bedrock through the Converse API with tool use. The model only picks the next tool from the results so far. Step budget of 8, and if the model stalls or Bedrock is down the rules finish the run and the trace says so. A deterministic guard checks the final action against the evidence: a bad photo ends in reshoot, a crack at the action width always escalates, a confirmed crack is never cleared.

It runs on a Graviton4 c8g.medium from the Cloud Optimized OpenCV (COOL) AMI. The user-data script finds the COOL interpreter, installs our pinned dependencies into its venv without touching its OpenCV, builds a second venv with the pip OpenCV 5 wheel as the baseline, writes the build information of both, and starts the FastAPI service on port 80. /health shows which OpenCV is serving, /evidence lists every benchmark and evaluation file the instance produced.

Claude Code was the coding agent. I made the product calls (what the actions mean, what the engineer needs to see) and ran the thing.

Challenges we ran into

The zoom tool was quietly broken, and only the evaluation caught it. Inside a small crop the crack is no longer a rare outlier, so the robust threshold inflated past the crack response and the zoom "found nothing", which made the agent answer clear on two CrackForest photos that have cracks. Now the zoom keeps the decision threshold from the full photo, and a candidate that vanishes ends in reshoot, never in clear.

The COOL image took three launches: the root volume must be 60 GiB, and its OpenCV is dynamically linked, so import cv2 only works with the two variables from the venv's activate script. The headless OpenCV 5 wheel has no ximgproc, so thinning is our own Zhang-Suen in NumPy. Oriented line openings looked clever and killed F1 to 0.03 because real cracks are jagged. Bedrock in Stockholm serves Nova Lite only through the EU cross-region profile, and a new account's quota is small, so the client backs off adaptively and the evaluation is paced.

Accomplishments that we're proud of

F1 0.717 on CrackForest at the usual 2 px tolerance with zero training, 30 ms per photo on one Graviton4 core. At the agent level: none of the 46 cracked photos is answered clear, all 46 blurred ones end in reshoot, all 20 crack-free surfaces end in clear. A trace an engineer can actually read. And a deploy that is one command from CloudShell.

What we learned

Measure the agent, not only the detector. The pixel metrics looked fine while the loop was making a bad decision in a branch nobody had looked at.

Small models are bad at comparing numbers. In the first evaluation round on the instance, Nova Lite agreed with the rule planner on 35% of photos, mostly because it kept writing "8.4 px is below the action width of 6 px" and filing the crack. The guard caught that in 61% of runs. Then we moved the comparison into the tool result (a plain over_action_width flag) and told the model not to redo it: agreement went to 93% and the guard was not needed once in 46 runs. All three rounds are in the repo.

And the COOL build is not a free speed-up for every pipeline. On a single Graviton4 vCPU our whole pipeline runs at parity with the pip wheel (which also ships KleidiCV on arm64); CLAHE is 1.7 times faster on COOL, black-hat is the same. We measured it on the same instance and wrote down what we got.

What's next for CrackScout

A reference-card detector so the scale comes from the photo itself, multi-photo runs where the agent asks for the second shot and merges, evaluation on concrete walls (DeepCrack), and a small classifier on candidate crops to raise precision on coarse surfaces while the measurement stays classical.

Built With

Share this project:

Updates

Submission history