Foveated Inspector

An inspection agent for huge concrete photos: with OpenCV 5 on AWS Lambda (Graviton), it chooses which tiles deserve full-resolution analysis and asks a human when unsure.

Comment

Inspiration

Inspection photos of concrete are enormous, and almost all of the surface is fine. Running a full-resolution detector everywhere is slow and expensive, and on busy textures it produces so many false alarms that the person reviewing them stops trusting the tool. Human inspectors don't work that way: they glance, zoom where something looks off, follow a crack, and call a colleague when unsure. We wanted to measure how far an agent gets with the same habit under a strict pixel budget. The name comes from the fovea of the eye, which sees sharply only at the centre of gaze.

What it does

  • Takes a 4096 × 4096 surface photo and a pixel budget for full-resolution analysis.
  • Glances at the whole image at ¼ resolution with OpenCV 5 and builds a belief map (how likely each cell is to hold a defect) over a 16 × 16 grid of cells (256 × 256 pixels each).
  • Repeatedly picks the next look: a pending full-resolution confirmation first, otherwise the cell and zoom level with the highest weighted defect belief per pixel; the visual evidence decides the next step: zoom further, follow a crack into the neighbouring cell, report only what full resolution confirms, or send the cell to a human-review queue.
  • Stops at the budget and shows every step: the web app replays the trace, shows the pixel cost against exhaustive scanning, and lets a reviewer approve or reject queued cells.

How we built it

  • OpenCV 5: multi-scale morphology (thin dark or bright marks), Hessian ridge strength (line-like shapes), Gabor energy (oriented texture), Lab colour deviation (stains and colour change), connected components, minAreaRect elongation, PCA crack directions, distanceTransform widths, ximgproc.thinning lengths, and cv2.ml random forests per scale with hard-negative mining.
  • AWS: API Gateway HTTP API (throttled) → a 256 MB web Lambda that validates and rate-limits without loading OpenCV → an arm64 (Graviton) inspect Lambda with an OpenCV 5 layer; S3 (private), DynamoDB (reviews, limits), CloudWatch. Benchmark generation, training, tuning and evaluation all run sharded on Lambda.
  • Evaluation: 14 photographed concrete surfaces (Poly Haven, CC0) with injected defects and exact pixel ground truth; development and test surfaces are disjoint, and the detector was trained and the policy tuned on development data only (we did look at test results of two withdrawn versions, so the test set is not fully untouched; both are described under Challenges we ran into, below). Random tiles and raster order (cells taken left to right, top to bottom) get the same full-resolution budget as the agent; coarse-only (the ¼-resolution candidates reported directly) and exhaustive analysis (every cell at full resolution) are reference points at very different pixel costs. A report counts as finding a defect only if its box overlaps the defect's box with intersection-over-union ≥ 0.1, or covers ≥ 30 % of the defect while its own box is at most 9 × the area of the defect's box.

Results (held-out test surfaces with synthetic defects: 28 images, 155 defects)

Defect-cell recall is the share of defects that have at least 30 % of their pixels inside one flagged cell; flag precision is the share of flagged cells that actually contain a defect.

  • At a 40 % full-resolution budget the agent's defect-cell recall is about the same as random tiles' (30.3 % vs 29.7 %) while it flags one seventh as many cells (1.8 % vs 12.5 %), so its flags are right 42.9 % of the time versus 8.3 %. At a 5 % budget its flags are right 87.5 % of the time.
  • The cost is recall: with this conservative setting the agent's defect-cell recall is only 19.4 % (5 % budget) to 30.3 % (40 % budget).
  • Exhaustive full-resolution analysis has the highest defect-cell recall (70.3 %) but produces 116.8 false alarms per image.
  • A hand-crafted OpenCV detector under the same policy does about as well on these test surfaces (33.5 % defect-cell recall and 55.2 % flag precision at a 40 % budget, but more false alarms per image: 6.3 vs 4.4), so the learned detector does not clearly beat the rule-based one yet.
  • Graviton vs x86 Lambda on the same OpenCV 5 tool: about the same latency (median 39 ms vs 37 ms billed).

Challenges we ran into

  • A merge rule that chained boxes along cracks made box recall (measured on report boxes with the old matching rule) look like 94.8 %; the agent's report boxes covered a median 77 % of each image. We caught it on the live demo overlay, measured it, fixed the merge rule (no chaining, bounded box size) and the matching rule (the overlap test above), and withdrew those numbers.
  • A first selection threshold (at most 5 false alarms per image) picked a configuration with low recall; we changed the threshold using development data only. We had looked at the test results of both withdrawn versions, so the test set is not fully untouched.
  • The detector trained on synthetic defects does not transfer to real crack photos yet: on 2,000 real crack / non-crack patches (Özgenel 2019) its accuracy is 52.9 %, close to chance on this balanced set, because it also labels 92.2 % of the non-crack patches as defective.
  • Our new AWS account allowed only 10 concurrent Lambda executions and 3008 MB of memory per function; with that limit no concurrency could be reserved, so we used API Gateway throttling instead and kept every heavy job sharded.

Accomplishments that we're proud of

  • An agent whose every decision is inspectable, and whose flags are right five times as often as random scanning at the same 40 % budget.
  • A reproducible, serverless benchmark: the 84-image 4096 px benchmark (28 training, 28 development, 28 test images) is generated on Lambda in 43 seconds.
  • Reporting the withdrawn results and the real-crack failure next to the main numbers.

What we learned

A better look-selection policy makes flags more precise, but recall stays limited by how well the detector separates defects from busy textures. Checking box sizes and the flagged area exposed a merge rule that had inflated recall.

What's next

Real defect data (and a detector trained on it), a stronger full-resolution model through OpenCV 5's DNN engine, and a learned look-selection policy evaluated against the same budgets.

Responsible use

This is a triage aid, not a safety certification. The trace shows which parts of an image were never examined at full resolution, cells still uncertain after full-resolution inspection go to a human review queue, and the main limitation (no transfer to real crack photos yet) is stated above. The public demo is throttled and capped per day; S3 lifecycle rules expire a raw upload after 1 day, and the inspected copy and its results after 14 days.

Disclosure

Code and text written with AI coding assistants (OpenAI Codex and Anthropic Claude Code) under human direction. The in-product agent is a rule-based policy; it does not call a generative model. Surfaces: Poly Haven (CC0). Crack patches: Özgenel 2019 (CC BY 4.0).

Built With

Share this project:

Updates

Submission history