FoodFlow

Discover, decide and pay for campus food in one conversation — and onboard a hawker stall by photographing its menu board.

Comment

Inspiration

The challenge asks how merchants of any size can deploy a commerce agent. Every answer we found assumed the merchant already has a website to drop a <script> tag into. Almost no hawker stall in Singapore has one.

That assumption quietly excludes the merchants who need this most. So we built for the hardest case first — a stall owner with a phone, a menu board, and no website — and let everything else fall out of that.

What it does

For the customer. One conversation: "something warm and soupy, no pork, under $6, near Frontier." The agent understands the constraint, filters six real NUS canteens, explains each pick in the customer's own words, builds a cart that can span stalls, and takes payment in the same window — no redirect, no checkout page. Voice works too: press and hold, speak, and the agent replies out loud.

For the merchant. Send a Telegram bot whatever you already have — a photo of the menu board, a PDF, a spreadsheet, a voice note reading it out, or a link to your website — and it comes back as a menu tree you check and publish. Websites that show their menu as a gallery of pictures work too: it reads the pictures.

Corrections are typed however people actually type. "teriyaki salmon - 8.9, classis caesar - 7.9" fills in two prices despite the typo. "Take off the striploin, we stopped selling it" removes a dish. "It's called Eighteen Chefs, and make the no-price ones 4.90" does both at once.

Four doors in, all resolving to the same published agent:

Merchant has How they adopt it
Nothing but a stall and a phone Photograph the menu board into a Telegram bot — implemented
A spreadsheet Paste any messy menu into the web console
A Wix / Shopify site One-click marketplace app
A GrabFood listing Import the menu they already keep current
A POS system API + webhooks — live prices and stock

Adding a door is a distribution decision, not an architecture change.

How we built it

Two engines, and it matters which one is running.

  1. A deterministic rules engine (api/_engine.js) parses the message into structured needs — soupy, no pork, under $4, near Frontier — and applies every hard filter: dietary, allergen, budget.
  2. A language model (api/agent.js) is then shown only the dishes that survived those filters, and picks among them and writes the sentence.

That ordering is the safety design. The model cannot recommend a dish containing something the customer said they can't eat, because it never sees one. Its output is validated against the surviving candidate-ID set before it reaches the browser. An agent that confidently says "no shellfish" and is wrong is a real-world harm, not a bug — so allergens are a hard filter, never a soft preference the model weighs.

customer message
  → /api/agent
  → needs parsed, merged with what we already knew
  → hard filters (diet · allergen · budget)   ← deterministic, never delegated
  → shortlist
  → model picks and phrases                    ← sees only survivors
  → client renders, each pick explained
  → authorisation → settlement split per stall

One catalog. The client holds no dish data; it fetches /api/catalog on boot, so anything a merchant publishes is visible to the agent immediately. Every dish carries the attributes a conversation actually turns on — flavour, texture, temperature, heaviness, form, cuisine, dietary flags, contained ingredients. Keyword search cannot answer "warm and soupy, not too heavy, no pork, under $10". Scoring over those fields can.

Merchant onboarding is itself conversational. The merchant doesn't fill in a form to get an agent — they talk to one, in an app they already have open. Photo of the board → the agent reads it → a canteen → stall → category → dish tree → correct anything by tapping or typing → one tap to Publish → live in the storefront, and the food agent recommends from it seconds later.

Reading a menu means guessing, so we split the guesses by what happens when they're wrong:

Inferred generously Never inferred
Contains — chicken, pork, gluten, dairy, nuts. A false positive only hides the dish from someone avoiding that ingredient — the safe direction to be wrong in. Halal — a certification, not something readable off a photo.
Attributes — noodles vs rice, soupy vs fried, spicy, prep time. Wrong just means a slightly worse recommendation. Vegetarian — calling a dish vegetarian when it isn't puts food in front of someone who didn't want it.
Calories and protein — estimated, always shown with a ~. Prices — flagged as missing, never invented.

Dietary flags stay empty until the merchant confirms them, and the bot says so when it publishes. Nothing reaches customers until Publish is pressed.

Trust is one deliberate act, not three. No consent checkbox, no separate OTP screen, no separate pay button — those are theatre people click through without reading. Instead one screen shows everything being agreed to: items grouped by stall, where to collect, the linked card, and the spend mandate — this agent may spend up to S$40 per order at campus canteens; this order uses S$6.90 of it; single charge, never recurring. Then a press-and-hold plus a biometric check. That one gesture is simultaneously the consent, the identity verification and the authorisation. A cart spanning two canteens stays a single charge, split at capture, with the receipt showing exactly where the money went.

This maps onto Visa's Trusted Agent Protocol: an agent proving it is a recognised entity, acting for an authenticated user, carrying valid instructions for that transaction.

Stack: vanilla JS front end, no framework and no build step; Node serverless functions on Vercel; a key-value store for published menus. It runs with no API key at all — on the rules engine alone the full demo completes. A judging hall is a terrible place to find out you needed working wifi.

What is real and what is simulated — we'd rather a judge learn this from us than find it themselves:

Real Simulated
Stall names, dishes and prices from six NUS canteens Payment — no charge is made, no live Visa integration
Attribute extraction, filtering and ranking Face ID — a scripted step-up, not a WebAuthn passkey
Menu reading from photos, PDFs, voice and websites Courier quotes — plausible, not live
The model call, when a key is set Order status — advances on a timer
Server-side pricing and the settlement split Merchant payouts — computed and shown, not paid

Challenges we ran into

  • The catalog was duplicated in three places in the prototype we inherited — client array, server array, merchant upload — so publishing a menu changed nothing the agent could see. Collapsing it to one /api/catalog was the single highest-value fix.
  • Serverless kills you quietly. Our Telegram webhook acknowledged Telegram first and did the work after, which is correct on a long-lived server and fatally wrong on Vercel — the invocation ends the moment the response is sent, so menu-photo onboarding silently did nothing in production while every log looked healthy.
  • A model that narrates. Asked to rename a published canteen, it replied "I've noted your new storefront name" — and nothing had changed, because the draft had been cleared at publish. Now code writes every confirmation, after the write, from what actually changed. A model may decide what was meant; it may never report what was done.
  • Flows that trap the person using them. Asked "what's this canteen called?", a merchant typed "waht abt my drinks selections?" — and we stored that as the canteen's name. It shipped to the storefront that way. Questions are now detected and answered, and the pending question asked again.
  • Deciding what the model isn't allowed to infer took longer than building the inference. "Halal" was the argument that settled it.

Accomplishments that we're proud of

  • A hawker stall can onboard with a photograph. No website, no app, no spreadsheet, no forms — the thing the problem statement asks for, for the merchants usually left out of it.
  • The bot takes whatever you send. Photo, PDF, CSV, voice note, video, or a link — including restaurant sites whose menu is a gallery of images, which it reads with the same vision model.
  • The safety split holds on both sides. Reasoning proposes, deterministic code disposes — for allergens on the customer side, and for prices and dietary flags on the merchant side.
  • The PDF reader has no dependency. Text pulled straight out of FlateDecode streams with Node's zlib, about forty lines. A scanned PDF is detected and reported rather than silently failing.
  • It degrades honestly. No API key, no KV store, a site it can't read — each has a real fallback and a message that says what happened, instead of a dead end.

What we learned

The hard part of agentic commerce for small merchants isn't the agent. It's that the merchant has no structured catalog and no way to make one — and that every field the AI fills in is a liability someone eventually eats. Getting the split right between what a model may guess and what a human must confirm turned out to be the actual product.

We also learned how much of "adaptable" is refusing to hard-code. Every time we wrote a rule for one phrasing, a real person typed a different one. The fix wasn't more rules; it was letting the model decide what was meant and letting code decide what happens.

What's next for FoodFlow

  • Real settlement: reuse the acquirer's KYC and pay into the account the stall already has.
  • A POS feed for live prices and sold-out flags — the real fix for menu drift.
  • WhatsApp alongside Telegram, since that's what hawkers in Singapore actually use.
  • Take rate on completed transactions, not a subscription. The merchant pays only once they've been paid.

Built With

Share this project:

Updates