StepSpotter

A home-repair assistant that won't take your word for it: OpenCV 5 checks every proof photo before the AI does, and decides what happens next.

Comment

Inspiration

I bought my first house in Orlando and realised I didn't know how to do the simplest things around it. What works for me is my phone: someone marks up my own photo, one step at a time, "not this one, that one". This kind of help is how, in September, I made sense of the low-voltage wiring panel in my house after three text explanations in a row hadn't helped, and how I swapped the dead battery in my UPS without shutting down the computers running on it: an AI assistant explained each job on my own photos, the manual's drawings and frames from how-to videos. StepSpotter does that, with one rule I insisted on — it can't take my word that a step is done; it has to see it. Plenty of homeowners work this way: in the 2021–2023 American Housing Survey, 50 million of 140 million home-improvement projects were do-it-yourself (U.S. Census Bureau and HUD).

The first version (built for AWS "Agents for Humans", see Disclosure) asked a vision model to judge one photo. It worked, and it had three blind spots no prompt fixes: a blurred or dark photo, a photo of the wrong place or the same "before" photo sent again, and a change in the zone the step said not to touch. Each of those is a question about pixels and geometry, not language. That is OpenCV's job.

On 50 test cases (33 bad photos, 17 good), the gate, with OpenCV first, refused 30 of the bad ones; the model alone refused 11 when shown one photo and 19 when shown both. On the 9 real bad photos the gate refused 8, the same as the model shown both photos (one photo: 4); its lead is on the 24 synthetic ones (22, against 11 and 7). The gate made 31 model calls for 54 checks (the 50 cases plus 4 from my own video) instead of 54, at $0.89 per 100 checks (list price) against $1.27 for the model on one photo and $1.80 on two. It wrongly refused 4 of the 17 good photos, where the model alone refused 3; two of the four are same-place pairs we now think were mislabelled, and without them it is 2 of 15, the same as the model on one photo (on two photos: 1 of 15). Before we changed the glare rule after a first run on these cases, the gate wrongly refused 8 of the 17. The full set of 548 cases is built from photos of my own house, 40 public Wikimedia Commons photos and 16 frames from a phone video of the leak under my kitchen sink. Small sample: 2 cases per synthetic class went to the model, one run each (docs/EVAL.md).

What it does

You name a small job and send a photo. StepSpotter plans it and draws step 1 on your photo: where to work, what not to touch, what the next photo must show. The next step stays locked — a Strands BeforeToolCall hook cancels the advance_step tool in code — until a photo proves this one.

Every evidence photo now goes through OpenCV 5 first, and its output changes what happens next:

  • blurred, dark, glare over the work zone, the part out of frame → refused without a model call, with one instruction ("hold still", "more light", "step back so the splitter is in frame");
  • the same photo sent again → refused, no model call;
  • nothing changed in the work zone → refused, no model call;
  • something changed in the do-not-touch zone → the run stops; in the web app it goes on only after you press "I am the person — I checked it, carry on";
  • OpenCV can't line the photo up with your first photo → the model gets both photos and must answer same_place; the gate won't open without true;
  • all clear → the model checks the photo with OpenCV's aligned crop of the work zone.

Faces that OpenCV's YuNet detector finds in an evidence photo are blurred in the saved file before the model sees it; the first photo, which the planner reads, is not blurred. A video works through the agent's submit_video tool (the web pages take photos only): OpenCV picks the sharpest frame that shows the right place.

In agent mode (/agent) the Strands Guide agent decides which tools to call — it can ask OpenCV about a photo before submitting it — and every answer shows its tool calls, OpenCV's decision and any refusal by the gate.

It is meant for low-voltage, low-consequence work only. The planner refuses mains wiring, gas, roofing and structural work, and that refusal is a model judgement, not a certified hazard classifier. The face detector was tested on one public-domain portrait, so we claim no detection rate. The live service has daily caps on jobs, photo checks and agent turns that bound model spending, and a $15 AWS Budget sends an email at 80 %.

How we built it

  • OpenCV 5.0.0 (opencv-python-headless==5.0.0.93): Laplacian sharpness per tile, histogram exposure, connected-component glare; SIFT on CLAHE luminance, BFMatcher.knnMatch and findHomography(USAC_MAGSAC) with a sanity check on the homography for "same place"; near-identity homography plus aligned difference for "already sent"; warpPerspective, CLAHE in Lab, absdiff and morphology against the noise floor outside all zones for "what changed where"; VideoCapture keyframes; FaceDetectorYN (YuNet) for faces. In 5.0.0 itself, creating FaceDetectorYN prints a warning that the DNN's new graph engine doesn't support targets yet, and the wheel ships type stubs; the YuNet model is not in the wheel, so the build fetches it, checks its size and SHA-256, and fails if they don't match (docs/PLATFORM-FINDINGS.md).
  • Strands Agents on Amazon Bedrock (Claude Sonnet 4.6): the planner, the zone marker and the verifier with typed outputs; the gate is a BeforeToolCall hook that sets cancel_tool or raises interrupt("human_review").
  • AWS: one CloudFormation stack — App Runner (one x86_64 instance), a tagged Bedrock application inference profile, least-privilege IAM, CloudWatch metric filters on the JSON perception trace (decisions, model calls saved, latency), three alarms, a dashboard, an SNS alert topic and a $15 Budget; CodeBuild builds the image from git archive and checks import cv2 inside it; ECR by digest. One command deploys, one removes.
  • An evaluation harness that runs the same gate code in four configurations — model on one photo, model on two photos, OpenCV only, OpenCV then model — with a hard cap on model calls.

Challenges we ran into

  • Real re-shots don't line up. On our own photos, a photo of the same panel from another angle kept 4–8 SIFT inliers, exactly the range of photos of different rooms. No threshold separates them, and our first policy refused 3 of 5 honest re-shots. We changed the policy: when OpenCV cannot confirm the place, it doesn't refuse — it sends both photos to the model and makes the gate demand same_place=true.
  • Parallax. Aligning a real re-shot of a cluttered panel left 37 % of the frame "changed". Change is now read against the noise floor outside all zones, and a noisy map decides nothing.
  • A false positive must never trap a person. After three OpenCV refusals in a row on one step, the next photo goes to the model with OpenCV's findings attached.
  • White things look like glare. The first evaluation run showed OpenCV refusing 8 of 56 ordinary real photos — white basins, a white wall, a flashlight hot spot. Glare now refuses a photo only when it newly covers the work zone, and overexposure is judged against the first photo; 0 of 56 are refused now. The price: some synthetic glare now goes on to the model.
  • App Runner runs x86_64 images only, so the Graviton plan changed; the build now checks the image architecture before pushing.

Accomplishments that we're proud of

  • OpenCV output decides the next action — no model call, a one-photo or two-photo model call, a cancelled tool, or a stop until you confirm you checked it — and every step of that is in the trace and covered by an offline test.
  • On the 50 test cases the gate refused 30 of 33 bad photos with 31 model calls for 54 checks; every case is in eval/results/, the summary in docs/EVAL.md. Shown both photos, the model alone matches us on look-alikes of another place; our edge is a resent photo, no change, a change in a do-not-touch zone, overexposure and a cut-off work zone.
  • A live recording of the real agent (data/demo/agentic-opencv/): a blurred photo is refused in 44 ms without a model call, a photo of another room goes to the two-photo check and fails, the right photo passes and the gate lets the agent move on, and a skip attempt is cancelled by the gate.
  • We published the failures next to the successes, including the policy that refused honest photos and why we dropped it.

What we learned

Most "is this step done?" failures are not about understanding the photo; they are about whether this is the right photo at all. Geometry answers that cheaply and the same way every time — and where geometry can't answer, the honest move is to say so and ask a better question, not to refuse.

What's next

  • A blind test on photos of a real repair that no threshold was tuned on.
  • The misses we know about: mirror-image look-alikes, a resent photo with a mark drawn on it, a smeared video frame that passes the blur check.
  • Segmentation instead of boxes for the work and do-not-touch zones.
  • Job state outside the instance, so a job survives a redeploy.

Disclosure

StepSpotter was built on 9–14 September 2026 for the AWS "Agents for Humans" hackathon on Devpost. It was also submitted to the AI Builders hackathon, whose results were announced on 25 October 2026. That code is public at github.com/retailbox-automation/stepspotter (commit 80979af) and is imported into this repository as its first commit (295e3f3), with the text copied from manufacturers' manuals removed. Everything OpenCV 5 — the perception package, its integration into the gate and the verifier, face blurring, the evaluation harness, the AWS stack, the diagrams and the report — was built on 21–24 October 2026, in this competition's window: commits 9cc6ca1 through 8831099 (the deployed code; the commits after it change only documentation and add constraints.txt). We found no clause in the Official Rules about previously submitted projects; the nearest is «From August 26 through October 26, teams will build, evaluate, document, and submit their final projects.» The code was written with AI coding assistants (docs/AI-USE.md); the product uses Claude Sonnet 4.6 on Amazon Bedrock at runtime.

Built With

Share this project:

Updates

Submission history