Imali AI receipt vision

An OpenCV 5 agent on AWS Graviton that decides whether a receipt photo can be trusted, then records the expense, asks the owner, or asks for a better photo.

Comment

Inspiration

Imali AI runs a small business from one WhatsApp chat, and shop owners send its agent photos of their receipts to record expenses. Those photos are taken on a counter with one hand: at an angle, under a hard shadow, in a dim shop, crumpled, blurred or with the bottom of the slip out of frame. A reader that always returns its best guess is the wrong tool for a set of books, because a wrong total recorded quietly is worse than no total at all.

What it does

An OpenCV 5 agent stands in front of the books. It finds the slip in the photo, flattens it, scores how readable it is, cleans it and reads it with OpenCV's DNN text models. Then it acts: it records the expense only when a second line of the same receipt (cash minus change, or the items and the subtotal) confirms the total; it asks the owner to confirm when a total was read but nothing confirms it; and it asks for a new photo, saying exactly what to fix, when no total can be trusted. Nothing reaches the ledger without either arithmetic agreement or the owner's yes.

How we built it

  • Perception tools (OpenCV 5.0.0): three ways to find the paper (Canny on the L channel with morphological closing, Otsu on the blurred L channel, CLAHE plus adaptive threshold); the PP-OCRv3 DB detector (cv.dnn.TextDetectionModel_DB) to check that the outline holds all the text and to rebuild it from the text when a shadow cuts it short; getPerspectiveTransform and warpPerspective to flatten; Laplacian variance, Sobel stroke density, a morphological background estimate and clipped-patch analysis to score sharpness, shading and glare; three cleaning modes (divide by paper colour, non-local-means plus CLAHE, unsharp mask); and the model zoo's CRNN (cv.dnn.readNet) to read each line.
  • Agent loop: each tool's result picks the next call (another paper-finding method, a rebuilt outline, another cleaning mode, another read, up to three reads) and the final action. Every step goes into a trace with its arguments, result, time and reason, shown beside each answer in the demo.
  • AWS: one Graviton2 t4g.small (arm64) in Seoul with an Elastic IP; Caddy for TLS and security headers; FastAPI under uvicorn as an unprivileged, sandboxed systemd service; pinned dependencies; model files checked against SHA-256 at install; one JSON log line per request (no image data); live /api/metrics and /api/health.
  • Evaluation: a generator of made-up receipt photos with exact answers across eight capture conditions; thresholds set on calibration seeds only; the agent compared with plain OCR and with the agent's own rules on a single read.

Results

On 240 fresh made-up receipt photos in three sets (seeds 20 to 22, thresholds set on separate calibration photos), across clean, angled, shadow, low-light, crumpled, blurred, cut-off and glare conditions, on the Graviton instance:

Measure Plain OCR Agent
Readable photos recorded right, without asking 78.2% ± 3.9 88.3% ± 2.5
Right total recorded or proposed 78.2% ± 3.9 88.8% ± 1.9
Wrong total recorded without asking 9 of 240 0 of 240
Mean time per photo 2.4 s 3.2 s

The weak spot is hard shadow: 12 of 43 shadow photos were sent back for a new photo, 6 of them with the wrong stated reason. The full tables, the earlier version's results and the failure cases are in the report.

Challenges we ran into

Hard shadows made every contour method stop at the shadow line, so the agent now checks the outline against where the text detector finds text and rebuilds it when text lies outside. Glare could wipe the same leading digit off the total, the cash and the change, so cash minus change still agreed with a wrong total. That showed up on a test set, so we added a rule (under measured glare the item column must agree too) and scored the final version on three fresh sets that nothing was tuned on.

What's next

The step goes in front of Imali AI's vision model in production, so each receipt photo is checked and flattened before it is read, and its total is cross-checked against the model's.

Built With

Share this project:

Updates

Submission history