Twice

Take the same photo twice. See what really changed. Sign the proof. OpenCV 5 evidence for renters, with an agent that rules out reflections and shadows.

Comment

Inspiration

Rental companies now scan returned cars with AI. One Hertz renter got a $440 bill for a one-inch scuff ($250 damage, $125 processing, $65 admin), and a US House subcommittee chair asked Hertz about damage assessments issued without human review. Landlords keep careful move-out photos too. Most renters have a few blurry phone shots that nobody can compare. We wanted the renter to have evidence as good as the company's.

What it does

Twice is renter-side evidence for apartments and rental cars.

  1. At move-in, you photograph a few spots. A quality gate waits until each photo is sharp and textured enough to match later. The set is hashed and sealed with a signature.
  2. At move-out, Twice puts a ghost of each first photo over the camera and guides you back to the same angle.
  3. OpenCV 5 aligns the two photos, finds what physically changed, and rules out shadows, lamps and exposure.
  4. When a change could be a reflection, the agent asks for one more photo from a step to the side. A reflection moves with the camera; damage stays. If the camera did not really move, it asks again, and then hands the finding to a person.
  5. The result is a signed report with every finding, every ruled-out shadow, the hash of every photo and the agent's full trace. Anyone can check it offline with the evidence bundle and python verify.py.

How we built it

  • OpenCV 5: SIFT and FLANN matching, a USAC_MAGSAC homography, warpPerspective and DIS optical flow for registration; a structure and colour z-score change map with connectedComponentsWithStats; NanoDet-Plus on the new DNN engine (cv2.dnn.ENGINE_NEW) to tell objects from surface marks.
  • Agent: a perception-decision-action loop with tools relight_check, glare_check, parallax_check, size_check and object_check, and a budget of two follow-up photos per spot. Every decision is logged with the measurement that caused it and signed into the report.
  • MCP: the same OpenCV 5 tools are exposed as an MCP server, so any agent (Claude Desktop, Claude Code, or our own runner) can run a check through them.
  • AWS: one container on Lambda (arm64, Graviton) behind a Function URL; S3 with Object Lock and SSE-KMS for evidence; an asymmetric KMS key (ECDSA P-256) for signatures; Claude on Bedrock for plain-language labels (it can name a change or flag it for a human, never change a verdict); CloudWatch metrics for latency and every agent decision. All of it is one SAM template (deploy/template.yaml). A COOL kit runs the same check on Graviton4 against pip OpenCV on Arm and x86.
  • Web: React and Vite, phone and desktop layouts.

Evaluation

  • Agent level (170 synthetic pairs × 5 RANSAC seeds = 850 runs, follow-up photos included): defect recall 0.88, precision 1.00; 754 of 850 runs fully correct. The agent asked for a side-step photo in 70 of 75 reflection runs, and all 70 ended correctly.
  • Real photos of unchanged scenes (Oxford affine sequences): 0 false alarms on 22 judged pairs; views too far away get a retake request instead.
  • Hybrid (drawn defects on real photos): recall 0.24, precision 0.96. The synthetic numbers are an upper bound, and we say so.
  • Failure gallery, ablations and methods are in docs/evaluation.md.

Agentic Vision evidence

1. Workflow diagram (perception → decision → action): agent-loop.png, also in the gallery above.

Stage What runs
Perception OpenCV 5: SIFT + FLANN, USAC_MAGSAC homography, warpPerspective, DIS optical flow, structure/colour z-score change map, NanoDet on cv2.dnn.ENGINE_NEW
Decision Rule planner (agent.py) with a budget of two follow-up photos per spot; each tool result selects the next tool call
Action Ask for a retake, a side-step photo or a close-up; rule out, report, or mark "can't judge"; hand a finding to a person; sign the report

2. Trace where OpenCV 5 output changes the next action (agent-trace.md, generated through the real API). Rental car spot, finding f1:

Run OpenCV 5 result Agent's next action
Renter steps aside glare_check: specular change +12% → side-step photo → compare_followup: mark vanished after the camera moved 2.5% Rule out as glare
Renter does not move (counterfactual) parallax_check: camera moved 0.0% of the frame, under 1%, twice → budget: no follow-ups left Mark "can't judge" and hand to a person

Only the photo differs between the two runs. The same loop is exposed as an MCP tool server (mcp_server.py); an external agent driving it makes request_human_review(f1) only in the counterfactual (moved, not moved), checked by test_mcp.py.

3. Evaluation (agentic-vision.md, section 3):

  • Task success: 754 of 850 full-loop runs correct (170 pairs × 5 RANSAC seeds); recall 0.88, precision 1.00. The engine alone flags all reflection pairs as damage; with the loop, 70 of 70 runs that asked for a side-step photo ended correctly.
  • Failure handling: no registration → retake; follow-up does not register → "can't judge"; camera did not move → ask again, then "can't judge"; budget spent → "can't judge"; person in frame → "can't judge"; Bedrock unavailable → rule labels.
  • Observability: every tool call is logged with the measurement that caused it, shown in the app and signed into the report; CloudWatch metrics AgentDecision (per tool), CompareLatency; error alarm.
  • Human control: the renter takes every photo; the agent only asks and explains why; the language model cannot add, delete or change a verdict; every "Changed" and "Can't judge" item goes to a person; anyone can verify the signed result offline.

Challenges we ran into

  • Reflections look exactly like scratches in a single comparison. The side-step photo and the check that the camera really moved were the fix.
  • RANSAC uses OpenCV's global RNG, so a result could depend on what ran earlier. Every benchmark now seeds it and reports five seeds.
  • Getting real damage pairs is hard. We built three benchmarks and report the weakest number alongside the best one.

Accomplishments that we're proud of

  • A counterfactual trace where only the photo differs and the agent's next action changes.
  • Evidence that can be checked without trusting our server.
  • A sample check a judge can finish in two minutes.

What we learned

Most of the work in change detection is ruling things out: light, exposure, small shifts, reflections. Saying "can't judge" honestly is better than a confident wrong answer.

What's next

Real move-in and move-out pairs with real marks, a study with renters, and a COOL run on Graviton4 to measure the speed-up.

Responsible use

Twice gives evidence, not verdicts. The report asks a person to review every "Changed" and "Can't judge" item. The language model cannot add, delete or change a verdict. Photos expire after 30 days, are encrypted with KMS, and are only stored for the check that made them.

Links

  • Code: https://github.com/mukeshkbj/opencv2026
  • Technical report: docs/report.md; evaluation: docs/evaluation.md; Agentic Vision evidence: docs/agentic-vision.md; COOL kit: docs/cool.md
  • Architecture diagrams: docs/diagrams/architecture.png, docs/diagrams/agent-loop.png

Built With

Share this project:

Updates

Submission history