Menu photo to website

Inspiration

Small restaurants in Italy change prices on paper first: the printed menu, the price list by the till. The website follows weeks later, if ever. A customer who saw a dish at 9 euro online and pays 11 at the table does not come back happy. Owners rarely have the time or the login to fix the site, but they all have a phone.

What it does

The owner takes a photo of the printed menu. OpenCV 5 finds the page, flattens it and reads every line, accents and "€" included. An agent compares what it read with the restaurant's website config and sorts every difference: a price read with high confidence becomes a proposed change, and anything uncertain becomes a question for the owner. A dish missing from the photo and new opening hours are always questions, because a photo can miss a line and does not say which days changed. The owner ticks the changes to keep, and the site is rebuilt from those alone.

What the camera measured picks the next step at four points. When no page outline is found, or the outline cuts through the text, the agent tries again with the outline joined from its pieces, and asks for a new photo only if that fails too. Weak lines get a second read after sharpening. Each change becomes a proposal or a question. Every step goes into a trace with the step it chose next, and the demo page shows that trace.

How I built it

• The page: Canny edges, convex hulls and a four-corner outline, then a perspective warp. A sheet that runs out of the frame breaks its outline into pieces, so the second attempt takes the hull of all the long edge pieces together.
• The reading: cv.dnn.TextDetectionModel_DB (PP-OCRv3) finds the phrases, and PaddlePaddle's PP-OCRv5 Latin recogniser, run through cv.dnn, reads each phrase whole. A greedy CTC decoder gives each phrase a confidence, and a line counts as sure as its weakest phrase.
• Keeping each price on its dish's row: the page is levelled by the slope of its text boxes, and boxes that blur cut short are grown along their ink until the phrase ends.
• A line under 0.93 confidence is read again after an unsharp mask. A change still under 0.93 becomes a question.
• The site is a config-driven static site; the builder escapes every string it renders.
• It runs as one AWS Lambda container (image in ECR) behind a Function URL. Every run writes one JSON line to CloudWatch with the trace, counts and timing. The function's role can write its own logs and nothing else, and photos are never stored.

I set the goal, the constraints (owner approval for every edit, no stored uploads, zero spend) and the acceptance checks. The code, the evaluation scripts and the report were drafted with an AI coding assistant (Claude Code), which also ran the measurements. I reviewed the result before submitting it.

Challenges I ran into

The first version read word by word with the OpenCV Zoo CRNN. Its alphabet has no accents and no "€", and on steep, blurred photos it read as few as 3% of the dishes exactly. A phrase recogniser through cv.dnn fixed the alphabet and brought a new failure: under blur the detector's box can stop mid-phrase, and the recogniser reads "Involtini alla" with full confidence. Boxes now grow along the ink until the phrase ends. The steepest photos had a different problem. The sheet leaves the frame, its outline breaks in two, and the agent asked for a new photo on 8 of the 60 photos in the two steepest sets. Joining the pieces brought that to 0.

Accomplishments that I'm proud of

On 180 synthetic menu photos in six conditions, from a steady shot to a steep phone photo with a hand's shadow, glare and JPEG compression, the agent proposed every changed price in four sets and 57 of 60 in each of the two hardest. It proposed no wrong change on any photo; the CRNN version proposed 11. Every change it did not propose became a question to the owner, with the price read correctly. None of these photos was used for tuning.

What I learned

Asking is cheaper than guessing. The confidence threshold was set so that no wrong price became a proposal on the tuning photos, and the cost is a few extra questions: 1.0 to 1.1 per photo, where a perfect run asks 1.0. A second try at a failed step also beats sending the owner back to take the photo again.

What's next for Menu photo to website

Real menus: handwriting, chalkboards, two columns and laminated card, measured on photos from owners who agree to share them. Then a sign-in in front of each owner's page, which the public demo does not have.

Built With

• amazon-cloudwatch
• amazon-ecr
• aws-lambda
• docker
• numpy
• onnx
• opencv
• pp-ocr
• python