Spatial Maintenance Copilot

Shows where a hidden component sits behind a panel, measures with OpenCV 5 how far to trust that estimate, and asks for a better camera view when it can't be trusted.

Comment

Inspiration

Before this project I worked as an industrial automation engineer at LG. Much of the time a fault cost me went not into the repair but into the search. I would stand in front of an electrical cabinet knowing which component I needed, but not where it sat behind the doors and covers. Drawings say roughly where, a video shows someone else's cabinet, and a vision-language model describes only what is visible. None of them relates this camera to this asset to a part that cannot be seen yet.

Spatial Maintenance Copilot is the tool I wanted then: it shows where the component is before the cabinet is opened. That makes the work easier and shortens downtime. On a production line downtime costs run into the hundreds of thousands, so every minute saved on searching is real money.

The same need reaches well beyond the factory floor:

  • mechanics working on vehicles;
  • further ahead, crews maintaining equipment on space missions, far from any expert and with no time to lose.

The asset changes, but the question stays the same, and here a new asset needs only a new Component Atlas.

My background covers both halves of the system: an Eng. in Automation and Robotics, and an MSc in Computer Science with a specialisation in AI. An earlier OpenCV project of mine taught me the principle behind the design. The variance of the Laplacian, a common sharpness score, depends on image resolution, so a threshold tuned on small images silently misjudges large ones. A measurement is a fact; a threshold is a calibrated opinion. So in this system the geometry measures, a fixed policy judges, and the AI agent acts on that judgement, never the other way round.

What it does

You point a phone at the asset, and OpenCV 5 does the rest:

  • It finds printed ArUco markers, computes the camera pose and projects the hidden component into the live view: an X-ray box behind the panel, with a 95% uncertainty region.
  • A fixed policy checks every measurement against a limit.
  • If the view is not good enough, the agent (LangGraph with Amazon Nova 2 Lite on Bedrock) asks the operator to move, for example "move left: the side markers face away from the camera".
  • When the view is good enough, the agent presents the result and waits for the operator's approval.
  • Once the panel is open, the target's own tag confirms or refutes the prediction.

Why a cabinet, not a car

The proposal planned a car with a hidden washer-fluid pump. I built the demonstrator on a TV cabinet at home instead, with a box hidden behind its flap. It is the same problem in a smaller, controlled form, and close to the electrical cabinets that inspired the project. A rigid asset I could reach every day let me:

  • record repeatable tripod viewpoints;
  • film each one with the panel shut and then open, which gives a measured ground truth;
  • run many sessions.

Nothing in the pipeline is specific to the cabinet: a new asset, a car included, needs only a new Component Atlas.

How I built it

  • Calibration and the asset frame. An iPhone 11 at 1080 × 1920, calibrated with ChArUco (RMS 0.90 px). A printed ChArUco board on the floor fixes the asset's coordinate frame.
  • Component Atlas. Two walk-around videos, panel shut and open, give 252 posed frames:
    • every marker corner is triangulated, with its σ from Monte Carlo (0.7-1.0 mm for the front and top markers);
    • the target is located by its tag;
    • the panels are fitted as planes.
  • Localisation. solvePnPRansac (SQPnP) and solvePnPRefineLM, then projectPoints for the target and its 3D box.
  • Uncertainty, measured rather than narrated. Every observation and landmark is perturbed by its σ, the pose is solved again and the target reprojected. The 95% region is the ellipse

$$ (\mathbf{x}-\boldsymbol{\mu})^{\top}\Sigma^{-1}(\mathbf{x}-\boldsymbol{\mu}) \le 5.99 $$

where \( \boldsymbol{\mu} \) and \( \Sigma \) are the mean and covariance of the Monte Carlo projections, and 5.99 is the 95% quantile of χ² with two degrees of freedom.

  • Decision policy. A result is reliable only when all of these hold:
    • at least 6 inliers, with an inlier ratio of at least 0.75;
    • RMS reprojection of at most 3 px;
    • inliers covering at least 5% of the frame;
    • at most 5% failed samples;
    • the region's semi-major axis at most half the target's projected radius;
    • sharp markers.

Otherwise the policy derives one move from the geometry (MOVE_LEFT, MOVE_RIGHT, MOVE_UP, MOVE_DOWN, MOVE_CLOSER, MOVE_BACK or HOLD_STILL).

  • The agent. A LangGraph graph runs localise, then guide or present, then a human interrupt.
    • The branch is the policy's, never the model's. The model words the decision and calls typed tools (suggest_viewpoints, request_view, present_result).
    • Guards replace any output that changes the move, contains numbers, or names a viewpoint that does not exist.
    • If the model fails, the operator gets the policy's own words.
    • Every step is written to a trace.
  • AWS. All in eu-north-1, and a frame localises in 60 ms on Graviton.
    • CloudFront and S3 serve the web app and the judges' recordings.
    • API Gateway leads over a VPC link and an internal ALB to ECS Fargate on Graviton (ARM64).
    • DynamoDB keeps the agent's sessions, Bedrock runs Nova 2 Lite in the EU, and CloudWatch collects logs and metrics.
    • There are IAM roles only and no stored keys, plus throttling, a daily cap on model calls and a budget alert.
  • Evaluation. Frozen datasets with SHA-256 manifests, a simulated operator running the real agent graph, and injected faults. make eval-check reproduces the results within fixed tolerances.

Architecture: OpenCV 5 and the agent on AWS

Challenges

  • Motion blur fools the geometry. A blurred frame still fits with a low reprojection error, because every marker is smeared the same way, while the target jumps. On 846 frames the reprojection error did not grow with blur, but the jitter did. Without a check, the policy accepted 43-52% of the frames with 5.5-8 px of blur. Remembering the Laplacian lesson, I measured blur on the markers themselves, as the width of a printed black-to-white edge:

$$ w = \frac{I^{(95)} - I^{(5)}}{G^{(98)}} $$

This is the intensity rise divided by the steepest gradient, the widest of eight directions, with the median taken over the markers. A still camera reads 2.8 px, and above 5.5 px the policy says HOLD_STILL.

  • Ground truth is harder than localisation. My first marker test was hand-held. Between the "panel shut" and "panel open" shots the camera moved 94 mm, so one "error" of 79 px was the camera, not the method. I re-recorded every spot as one continuous tripod clip (shut 5 s, open 5 s, shut again), which is split automatically by the change in the picture. Even then, stepping up to open the flap flexed the floor: the tripod moved a median 0.57 px, and that is part of every measured error.
  • A white cabinet is hard to track. Markerless model-based tracking (cv.rapid) ended 220-530 mm off after a single step. The silhouette is hidden by a TV and books, and the cabinet is white against a white wall. Printed markers were the honest answer.
  • A browser camera is not the Camera app. Thresholds tuned on stabilised videos never fired in a phone's browser. The live page now tells motion from the picture itself and analyses a view only once the camera holds still.
  • Language models like numbers. With Nova in the loop, 17 of 35 turns contained digits. The guard replaced each of them with the policy's own sentence, and every session still succeeded.

What I learned

  • Keep the measurement and the decision as separate things you can inspect. The agent is useful precisely because it cannot move the branch.
  • A wide uncertainty region "contains" any truth, so confirming a result needs a region that is also tight.
  • Ground truth deserves as much engineering as the method.
  • Failure cases are results. The report lists the ones I found, such as foreshortened markers read as blur, and a second opinion asked too early that described the flap's handle.

Results

These are the results on 27 tripod viewpoints held out from the atlas, with the box hidden:

Measure Result
Target error, median 1.7 px (2% of the box's projected radius)
Truth inside the 95% region 22 / 22
Wrong overlays accepted 0 of 18, and 0 in 41,788 injected-fault trials
Agent task success 25 / 25 sessions (policy alone: 18 / 25)
Latency 60 ms per frame on Graviton

A second dataset uses hand-clicked landmarks, and there fewer views localise. On it the agent reached a correct, accepted result in 19 of 19 sessions, against 4 of 19 for the policy alone and 64% for a random walk.

Agent workflow: perception, decision, action, human approval

What's next

A new asset needs only a new atlas, with the same perception, uncertainty and agent loop. Next come markerless landmarks on textured assets, an atlas from CAD, and asking for the second opinion only once the target is in sight.

Further ahead, the same loop could run on AR glasses. In a factory, the technician would see the component highlighted inside the electrical cabinet before opening it, and be guided to it step by step.

Concept illustration: the same loop on AR glasses in a factory. Not part of this submission.

Concept illustration (AI-generated): where this could go. It is not built and not part of this submission.

Try it: live application · code · technical report

Built With

Share this project:

Updates

Submission history