ForgeSight

An agentic steel inspector that refuses to guess: OpenCV 5, a measured gate, and a human in the loop.

Comment

ForgeSight is a solo build for the OpenCV AI Competition 2026: an agentic visual inspector for rolled steel, running on OpenCV 5.0.0.

Inspiration

ForgeSight started with a failure mode I kept seeing in inspection demos: a classifier returns “no defect” with high confidence on a frame that is blurred, blown out, or glared. The model is answering the question it was asked, but the measurement is bad. On a steel line, that confident wrong answer is expensive—scrap, a missed defect, or a stopped line. I wanted to build the part those demos skip: a system that says “I cannot judge this frame,” changes the capture, and only then decides. OpenCV 5 made that practical. Classical CV gives interpretable quality metrics and a Gabor descriptor that runs on a CPU; the “agent” can be a small, auditable state machine over those measurements, not an LLM planner. That constraint became the design: every step has a number behind it, and every decision can be replayed.

What it does

ForgeSight takes a 200×200 steel capture and returns a class, a confidence, a defect region when it finds one, and a decision: auto-pass, human review, or escalate. It covers seven NEU classes: crazing, inclusion, patches, pitted_surface, rolled-in_scale, scratches, and no_defect. POST /inspect is a single pass; POST /inspect/loop runs up to six steps. The loop starts with a quality gate: sharpness, glare, blown highlights, noise, contrast. If the capture is unusable, the system refuses to classify. The agent selects a bounded capture action—steady_and_refocus for soft focus, boost_contrast for a crushed frame—the simulator re-renders the scene, and perception runs again. If the classifier says clean but the localiser finds structure, the case goes to a human instead of auto-passing. Work orders are persisted, not just displayed. A reviewer approves or denies; the record stores decided_by: human, timestamps, and the audit trail, survives a restart, returns 404 for an unknown order and 409 for a second decision. The demo UI and trace show every step, including the capture change and the reason for the decision.

How we built it

Backend: Python/FastAPI on one AWS c8g.medium in us-east-1 with pip OpenCV 5.0.0. A small demo UI calls /health, /inspect, and /inspect/loop. Perception: a quality gate (sharpness, glare, blown highlights, noise, contrast); a Gabor filter-bank descriptor with a compact classifier; a multi-scale localiser (proposal generator + 18-feature linear scorer, top_k=2, NMS 0.3). The capture simulator applies focus blur, exposure, glare, noise, and contrast so the loop can re-render under a chosen setting. Agent: a small state machine, not an LLM. It reads the gate measurements and the classifier/localiser outputs, chooses from a bounded action set, and stops at six steps. Every step is written to the trace. Human loop: a persisted work-order state machine with an audit trail. The point was to make “a human decided this” a fact in the data, not a label in the UI. Data: NEU-CLS and NEU-DET are the same survey, so I deduplicated to 1800 unique source images and added 269 clean crops from outside every annotation box—2069 rows. The split is by source image: 1138 train / 310 validation / 621 test. The test set is 540 defect + 81 clean and was not used for tuning. Thresholds and calibration were fitted on validation and frozen before the test read. COOL: a three-arm benchmark on Graviton4 with the same feature code, swapping only the OpenCV build—COOL 5.1.0-dev vs pip OpenCV 5.0.0. The demo API runs pip OpenCV 5.0.0; COOL is a separate, library-only benchmark, not in the request path. The submission video uses the real app trace and the repository’s architecture PNG; the work-order card mirrors the persisted fields.

Challenges we ran into

The first accuracy number was too good. NEU-CLS and NEU-DET are the same underlying survey; a naive split leaks source images. I deduplicated and split by source image. The honest headline is 92.75%, about 0.3–0.7 points lower depending on the model. I kept the lower number. The first calibration number was optimistic. The temperature was fitted on the same held-out split used to report ECE. I moved to a three-way split: weights on train, temperature on validation, everything reported on test. ECE is 0.0361 after scaling on test (0.0886 before scaling). The first localisation number was tuned on the images it was reported on. I rebuilt the split with disjoint tune/eval halves. The honest macro F1 at IoU 0.5 is 0.2247, barely moved from 0.2200, even though the proposal oracle rose from 0.346 to 0.546. It is a real improvement on an honest baseline, not a good detector. Crazing (0.061) and patches (0.053) are barely above zero; pitted_surface regressed from 0.278 to 0.250 when I added variants. It is the weakest part of the system, and I say so. The conservative gate is safe, but not free. Only 269 clean crops exist, because most NEU frames are annotated almost end-to-end. no_defect recall is 0.790—the lowest of any class. The gate auto-passes only 7.1% of traffic and sends 41% to human review. I did not retrain on the test set to lift either number. The loop is not a general bad-capture detector. It beats single-pass in all six conditions in the 740-capture ablation, but the gate only reliably detects soft focus. For specular glare and sensor noise, the loop improves the decision but the gate does not fire; glare refusal is 0.0081. I could have claimed more; it would not be true. COOL was slower, not faster. On our workload it is 0.94x—6% slower—302.60 fps vs 321.79 fps on c8g.medium. Canny is 1.57x faster, but Canny is not our hot path; Gabor is, and COOL is at parity there. The demo API runs pip OpenCV 5.0.0. I do not claim COOL in the request path. An agentic feature that did not earn its place. An earlier “second opinion” branch applied CLAHE when confidence was below 0.55. It fired 359 times and changed 165 classes, but was net-negative, so I disabled it. Shipping the measurement, not the idea, was the lesson.

Accomplishments that we're proud of

On the leakage-free 621-row test set: 92.75% accuracy across seven classes, 0 of 540 defects missed, auto-pass precision 1.000, and ECE 0.0361 on test. The sensitivity lower bound is ≥99.44% at 95% (rule of three), which I quote instead of a perfect point estimate. The closed loop beats single-pass in all six capture conditions in the 740-capture ablation—+0.161 on specular glare and +0.173 on sensor noise. A complete deployed system: FastAPI, demo UI, persisted work orders, audit trail, 404/409 semantics, survives restart. The COOL benchmark is honest: 0.94x, 6% slower; Canny 1.57x, Gabor 1.00x; library-only; demo API on pip OpenCV 5.0.0. Total AWS spend was under $1 against a $25 cap on a single c8g.medium.

What we learned

Refusal is a feature. A model that always answers is not a measurement system; capture quality is part of the model. Agentic does not mean LLM. A small state machine over measured gates is debuggable, auditable, and cheap; every step can be replayed. Leakage is subtle. Split by source, freeze the test set, fit calibration on validation, and report the lower number—even when it costs you the headline. Measure the feature and remove it if it does not earn its place. The CLAHE second opinion fired 359 times and was net-negative; disabling it was the right call. Negative results build trust. The 0.2247 localisation and the 0.94x COOL result are in the report and in the video, not buried. Human-in-the-loop is a data model, not a button: who decided, when, and why.

What's next for ForgeSight

Localisation is the first target. The proposal oracle is 0.546, but the linear scorer captures only ~36% of it; a learned scorer on frozen splits is the next experiment, with crazing and patches as the priority. no_defect recall needs more clean crops and hard negatives—269 is too few. The test set stays frozen. The quality gate should detect specular glare and sensor noise, not just soft focus; today those conditions improve through the loop but do not trigger the gate. Deployment: real camera/PLC integration, an edge build, and optimizing the multi-scale localiser (62 ms per 200×200 crop vs 20.5 ms single-scale in the demo build). Human review: reviewer identity/auth, audit export, and a real queue. COOL stays library-only until it wins on the hot path; contributing the benchmark upstream is more useful than forcing it into the request path. Code, architecture, evaluation notes, and benchmark artifacts: https://github.com/neemlama/opencv-visionguard

Built With

Share this project:

Updates

Submission history