NextView

An OpenCV 5 agent that tells you where to put the calibration board next, rejects bad frames with a reason, asks before changing the lens model, and stops when the estimated error is on target.

Comment

Inspiration

Every measuring camera, whether on an inspection robot, a drone or a phone used to size cracks, inherits its calibration error. The usual routine is to wave a checkerboard, collect a few dozen frames and trust the in-sample RMS reprojection error. That number hides the problem: in my simulated benchmark (robot, run-level medians) the agent's own in-sample RMS after 10 frames had a median of 0.066 px while the true error at the 95th percentile over the whole image was 1.23 px. The parts of the image the board never reached are where measurements go wrong. Calibration is step 0 of every metric inspection, so I wanted a tool that knows where the board should go next and when to stop.

What it does

NextView is a perceive–decide–act agent for camera calibration:

  • Perceive. For each frame it detects the ChArUco board (CharucoDetector), measures blur on the board's own square edges with a slanted-edge method (in pixels, with a motion direction), checks glare and board size, and estimates the pose with IPPE (solvePnPGeneric).
  • Decide. A priority-ordered rule table (R0–R13) rejects bad frames with a measured reason, drops outlier views, tests the residuals for an under-fitted lens model, refits with calibrateCameraExtended, and builds a per-pixel uncertainty map from projectPoints Jacobians with a Schur complement. A probability calibrated in simulation says how likely the true full-image error is above target.
  • Act. It ranks 300 candidate board poses by the predicted drop in the 95th-percentile uncertainty and shows the best one as a ghost outline with a numeric instruction ("board to bottom-left, ≈ 45 % of frame width, tilt ~35°"), or sends the pose to a robot.
  • Human control. Switching the lens model, using the release-object fit for a warped board, restarting after focus drift and every export require an approval with a reason. Fisheye lenses are declined: no clean stop in 20/20; R10 flags 15, the other 5 end on a drift alarm.
  • Interfaces. A web app (simulator, webcam, and an evidence tab with the benchmark figures baked into the image) on a public AWS endpoint, and an MCP server with the same tools, so any MCP client or a local LLM can drive the agent with the same approval gates.

How I built it

  • OpenCV 5 (opencv-python-headless 5.0.0.93) for everything visual: objdetect (ChArUco), geometry (PnP, projection Jacobians, undistortion, homographies), calib (calibrateCameraExtended, calibrateCameraROExtended), imgproc (the simulator's homography renderer, distortion remap and blur PSFs), core (FileStorage YAML export in memory). No DNN and no training in the loop.
  • AWS: a stateless Lambda container image (x86_64, 512 MB) from ECR behind an API Gateway HTTP API with stage throttling; the session travels in an HMAC-signed token; approved exports go to a private S3 bucket with presigned links; one JSON trace line per request with CloudWatch embedded metrics. Deployment, smoke test, load test and teardown are idempotent PowerShell scripts with pinned image digests.
  • Evaluation: a simulator with exact ground truth, eleven policies (the agent, ablations, random poses, a textbook template, an emulation of OpenCV's interactive-calibration app, coverage-greedy and a dense grid), a pre-registration written before the test split with frozen hashes, paired Wilcoxon tests, bootstrap CIs and Holm correction, plus real third-party data (METRIC, CC BY 4.0).

Results

On the pre-registered synthetic TEST split (6,600 runs: 100 cameras × 3 seeds × robot and human actuators × 11 policies):

  • After 10 frames the median true full-image p95 error is 1.21 px vs 2.71 px for random poses (robot) and 1.22 vs 3.61 px (human). All eight pre-registered endpoints favour the agent after Holm correction (largest corrected p = 2.9 × 10⁻¹³).
  • Median frames until the true p95 is below 1 px: 8 vs 31 (robot; 31 = not reached in 30) and 10 vs 31 (human).
  • The stop rule stopped 319 of 600 runs with 0 false stops; the fisheye, focus-drift and warped-board failure sets ended in an abstention in 20/20, 20/20 and 20/20 runs.
  • Reported honestly: the perception gate adds no accuracy in simulation and costs a hand-held operator extra attempts; on real METRIC data the agent is not consistently better than random frame selection; the first-order uncertainty under-covers (95 % ellipses hold 0.79 of the true errors) and the stop rule never fires on telephoto lenses. A local qwen2.5:7b drove the MCP tools through every session with the same choices as the planner, which shows the interface works but adds no accuracy.

Challenges I ran into

  • The AWS account caps Lambda memory at 512 MB, a quarter of the planned 2048 MB and of the CPU share that comes with it. I moved camera-map building into session start, added a warm-up at init, and made the slowest checks run one fold per step so every step stays within its time budget.
  • Honest evaluation: the first-order uncertainty is optimistic, so the stop decision uses a probability calibrated on a separate development split, and the results that went against my hypotheses are published next to the ones that held.
  • OpenCV 5.0 reorganised modules (pose and projection now live in geometry), and calibrateCameraROExtended rejects an intrinsic guess, so the release-object path always starts from OpenCV's own initialisation.

Accomplishments that I'm proud of

  • A visible, auditable loop where every OpenCV measurement that changes the plan is written into the trace.
  • A pre-registered benchmark with exact ground truth, ablations, real third-party data and published null results.
  • A public endpoint that costs a fraction of a cent per session and cannot change a lens model or export without a person.

What I learned

Most of the accuracy gain comes from where the board goes, not from rejecting frames; uncertainty maps need calibrating before they can decide when to stop; and an agent that abstains on lenses it cannot model is more useful than one that always returns a number.

What's next

Testing with physical cameras and boards on the live site, a glare-aware planner, masking faces in webcam frames, a multi-camera rig calibration with calibrateMultiview, and fisheye support.

Links

Team

Umar Mohktar, Singapore. Solo entrant: design, implementation, evaluation.

Licences and data

Code MIT; benchmark outputs CC BY 4.0. METRIC: Allegro, Terreran & Ghidoni (2023), doi:10.5281/zenodo.7976757, CC BY 4.0. Full list in ATTRIBUTION.md.

Built With

Share this project:

Updates

Submission history