Second Look

• 

  What OpenCV 5 measured and the call it chose: glare gets a retake, a clipped retake trips a different rule, a non-document goes to a person.

• 

  Input: a synthetic receipt photo with a glare patch over the TOTAL line (data/synthetic/glare_text.jpg). This is what OpenCV measures.

• 

  OpenCV's verdict: 92% of the detected glare sits on text lines (threshold 0.15), so retake, with the glare's box and an instruction.

• 

  The retake photo: the glare is gone, but the receipt runs off the bottom of the frame (data/synthetic/crop_bottom.jpg).

• 

  Stored trace, linked by caused_by: the retake fixes glare (0.9211 to 0.0), then a crop defect fires a different rule.

• 

  An image that is not a receipt: no document outline found (quad_found False), so the agent escalates to a person instead of guessing.

• 

  The human-only step: the escalated capture waits in the review queue until a reviewer clicks Approve. The note is optional.

• 

  That capture's trace after approval: three agent entries, then entry 4 by the reviewer, who moved it to accepted.

• 

  Evaluation, 17 synthetic samples (15 labelled): 15/15 verdicts correct, 0 bad captures silently accepted. Full table in docs/evaluation.md

• 

  Architecture: one invoke() chokepoint behind every route, live on AWS EC2 at 32.236.165.113; the Lambda path is built but blocked by policy.

Code: github.com/guptachetan1995/opencv (MIT). Run it: make setup, then .venv/bin/python tools/run_demo.py for the whole loop printed as its trace, or make run for the HTTP server and review page.

Live demo: http://32.236.165.113/ — the same server on one AWS EC2 instance in Sydney (plain HTTP), live since 2026-09-28; the curl checks in the README's Deployment section run against it from the root of a clone (curl and python3 only), and the reviewer routes need a token the entrant holds. A static replay of a recorded run is at https://guptachetan1995.github.io/opencv/. The Lambda path is blocked by the AWS account's Free Plan policy (deploy.md). A live screen-share of the review page is also available to judges on request via a Devpost message.

Inspiration

Picture a nine-person surveying firm — an illustrative case, not a customer we have worked with — whose staff photograph their receipts, and whose finance team discovers the bad ones at month end, four to thirty days later, when the surveyor is on another site and the paper is in a bin. The expensive quantity is the latency between a bad capture and anyone knowing it was bad, because that latency is what makes the defect permanent. Two seconds after the shutter the photo can simply be taken again.

Capture-time checks already exist, and we started from them. Scanbot's Document Quality Analyzer grades a scan by how readable its text is and hands the score, a histogram and a heatmap to the integrating app. Veryfi Lens flags a capture with 20% or more blur and alerts the user, has optional glare detection, and crops the document during capture. Expensify's SmartScan reports a failed scan and offers Retake. What we added, none of which we found in their documentation:

• A different, located instruction per failing metric, from a committed rule table that records every clause it evaluated.
• Glare judged by where it falls: glare_over_text_frac is the share of the glare on the text lines, so glare on blank paper is accepted and glare across the total is not.
• Retake memory: compare_captures checks that the defect the retake was asked to fix is actually fixed, and after two retakes the agent stops asking and hands the capture to a person.
• A duplicate check only a person can resolve.
• A caused_by trace behind an actor- and state-guarded chokepoint, so which measurement caused which call, and which person made which decision, is a stored record.

The second thing that shaped the design: the two possible errors do not cost the same. Nagging someone for a retake they did not need costs seconds. Silently accepting an unreadable receipt costs the whole claim, and the person who pays is not the person who made the error.

What it does

Second Look is active perception for autonomous inspection, applied to receipt capture. It cannot move the camera, so it directs the person holding it. A phone posts a photo; OpenCV 5 measures it — document quad and rectification, per-tile focus, exposure and contrast, specular glare, text presence and coverage, frame-edge crop, QR or barcode, and a perceptual hash for duplicates — and a deterministic policy turns those measurements into exactly one of three verdicts: accept, retake, or escalate.

The measurement chooses the next call. A shot with glare across the total is asked to be retaken with the light moved, with a hint box on the region. The retake that removes the glare but clips the bottom edge gets a different instruction, from a different rule, fired by a different metric — and the trace shows compare_captures reporting the glare fixed first. The agent may auto-accept only when every metric is comfortably inside its band; the uncertain band, a defect that survived two retakes, a suspected duplicate and anything that does not look like a document go to a person, who sees the photo with every region OpenCV found drawn on it, the clause that fired, and Approve, Reject and "Is / Is not a duplicate".

How we built it

• OpenCV 5 (opencv-python-headless==5.0.0.93, pinned exactly so a loose constraint cannot resolve backwards into the 4.x line where warping numerics differ). Eight measurements over the rectified page, so defects are reported by location on the receipt; an evidence overlay drawn back onto the image with OpenCV for the reviewer.
• A committed threshold table (policy.toml): an 11-rule cascade, first match wins, no model and no randomness. Every clause evaluated is written into the verdict's firings.
• One chokepoint, two guards. agent_loop.invoke(tool, args, actor) is the only way to measure, decide or move a capture; a photo arriving only fills a slot. It refuses an agent reaching for a human verb (PermissionError), and every agent tool refuses a capture that is escalated, awaiting a retake, or decided (StateError). An agent may restate what the verdict says (the failing metric, the escalation reason) but never replace it. The review page's buttons call the same function with actor="reviewer", and over HTTP those routes need the reviewer token.
• Three thin adapters over the same routes: the standard library's HTTP server (local and EC2), a Lambda Function URL handler, and a stdio MCP server that offers an LLM agent the agent tools — never the human verbs.
• AWS: the same routes run live on one EC2 t3.micro in Sydney, built on first boot from the public repository by a user-data script and launched through the console. A Lambda container image on arm64 behind a Function URL is built and tested locally, but the account's policy blocks creating it.
• Python 3.13, standard library only for serving; one static HTML page with vanilla JavaScript; 161 tests.

Challenges we ran into

• Glare that matters versus glare that does not. Area alone is the wrong metric: a bright patch on white space must be accepted, and one across the total must not. glare_over_text_frac is what separates them, and both cases are held by fixtures.
• Duplicate detection colliding with itself. dHash runs on the rectified page, so defect variants of the same printed layout hash close to an accepted capture: through one shared batch, 6 of 15 samples escalate as suspected duplicates and task success drops from 15/15 to 9/15. We report both numbers and never loosen the threshold to hide it — escalation is the safe direction, and the silent-accept rate stays 0/15 either way.
• Making "waits for human approval" true by construction. A review of our submitted version found that an agent could re-decide an escalated capture and request a retake of it, walking it out of the queue. We added a state guard to every agent tool and a test that replays that exact sequence. A second review found that escalate recorded whatever reason it was given, so a suspected duplicate could be relabelled and approved without the duplicate question; escalate now keeps the verdict's own reason, and approve reads the duplicate flag from the verdict too.
• Documenting what was not built. Both diagrams are drawn as built with a legend — solid = built and tested (or, for the EC2 node, deployed and verified live), dashed = planned, not built or not deployed — because misrepresenting capability is an explicit rejection ground for this competition.

What we learned

Choosing the headline metric first changed the design more than any threshold did. Once the silent-accept rate is the number you report first, escalating rules must sit above retake rules in the cascade, an uncertain band has to exist, and a suspected duplicate can never be auto-rejected. The evaluation reports 0/15 silent accepts on the synthetic set — and reports alongside it that escalation precision of 100% is trivially high on fixtures that are bad by construction.

What's next

A real-photograph evaluation set, collected under the protocol already written in real-photo-set.md, so agreement and silent accepts are measured against hand labels; S3-backed capture storage and CloudWatch structured logs; override, discard and close_batch as reviewer verbs — two_documents has no correct resolution without them; and TLS in front of the EC2 endpoint, which serves plain HTTP today.

Agentic Vision Award

The award requires that "image or video results must influence a subsequent plan, tool call, action, or request for human approval… the visual evidence must change what the system does next." The qualifying beat, asserted end to end by tools/run_demo.py and measured in the retake-convergence set:

glare_text.jpg → 92% of the glare lies on the text lines, over the 0.15 threshold → rule 6 glare_over_total → retake, with a hint box. The retake fixes the glare (compare_captures: 0.92 → 0.00, fixed) but crops the page → edge_touch_sides contains bottom and the bottom band has text → rule 5 bottom_edge_clipped → retake again, from a different rule chosen by a different metric on a different image. Then clean_a.jpg is accepted. Nothing scheduled the second instruction; the second image's own measurements selected it. The same chain runs over HTTP (POST /inspect, then POST /retake/<successor_id>) and over MCP.

Why the decision layer is deliberately deterministic, not an LLM. The same photo always gets the same verdict, and every verdict carries the clause that produced it, so a threshold can be tested, audited and argued with, and the evaluation comes out the same on every run. Nothing leaves the machine to decide a capture — no per-photo model cost, no receipt image sent to a third party. The judgment calls that genuinely need intelligence (is this the same receipt, is this even a receipt) are routed to a person rather than to a model. An LLM agent can still drive the loop through the MCP server, and the same guards bind it.

The agent declines to keep guessing: attempt >= 3 fires persistent_defect and escalates rather than requesting a third retake. Human control is enforced at the chokepoint: approve, reject and resolve_duplicate raise PermissionError for any actor that is not "reviewer", approve refuses a suspected duplicate until resolve_duplicate has run, and every agent tool refuses a capture that is waiting for a person. Observability: every call writes a TraceEntry(seq, actor, action, inputs, outputs, caused_by), and GET /trace/:capture_id returns the whole retake chain as one numbered sequence.

Judging criteria — where to look

Built With

• amazon-ec2
• amazon-web-services
• docker
• html5
• javascript
• mcp
• numpy
• opencv
• python