LookAgain: Agentic Visual Troubleshooting

Phone photo in, OpenCV triage out. LookAgain uses vision metrics to pick the next tool—retake, compare, or bounded diagnose - across LEDs, brakes, and leaf spots. Bedrock + SIFT on AWS COOL.

Comment

Inspiration

Most “AI vision” demos classify a photo and stop. Real troubleshooting is messier: the first shot is glared, blurry, or too far, and a wrong claim (“safe to ride”, a disease name) is worse than no answer. I wanted an agent that lets OpenCV measurements choose the next action—retake, remediate, compare, inspect, or escalate—not a single forward pass.

What it does

LookAgain is agentic visual troubleshooting for three allowlisted packs: appliance_leds, bike_brake_pad, and leaf_spot.

  1. Phone capture (or no-prop fixture pairs) → triage_frame (sharpness, glare, exposure, ROI)
  2. On failure → remediate_frame, else request_capture with a reason (less_glare, blur, move_closer, …)
  3. On retake → compare_shots (SIFT + USAC_MAGSAC, reason-aware scale/overlap) gates whether the new shot is good enough
  4. inspect_pack → evidence_gate → finalize_report with bounded claims only (or escalate_human)

Amazon Bedrock Converse tool-use picks the next tool from vision JSON; LOOKAGAIN_AGENT=heuristic is a deterministic fallback. Packs outside the allowlist never finalize.

How I built it

  • OpenCV 5 locally (opencv-python==5.0.0.93) for triage, remediate, SIFT matching, pack inspectors
  • FastAPI session API + phone UI; one-click demo pairs via /api/demo/run_pair
  • Bedrock (Claude Haiku 4.5 inference profile) for the agentic loop
  • AWS COOL on Graviton (c8g.large): honest KleidiCV / OpenCV 4.x path on AMI—SIFT locked (no ALIKED); benches report parity, not inflated speedup

Challenges

  • COOL AMI is not OpenCV 5 and lacks ALIKED → design for SIFT + honest dual-build storytelling
  • Bedrock document types reject Python tuples in tool results → json_safe coercion
  • Software remediate can “fix” glare too well for demos → fixtures that still force a guided retake
  • Claim safety: forbid “safe to ride” / disease names so finalize stays competition-safe

What’s next

Face-on demo video with live Bedrock tool trace + short COOL /health proof; optional stock Arm demo box through judging.

Built With

Share this project:

Updates

Submission history