JOLO

Inspiration

Small cafés change their menus all the time: a new seasonal latte, a price bump, an item sold out.
Updating Google, Instagram, delivery apps and the website by hand is a chore most owners skip, so
customers see stale menus and owners lose sales. Meanwhile, customers just want to text someone "where can I get a good cortado near campus?" and get a real answer.

We wanted a tool with no new app and no dashboard to learn. Café owners already text. So we built one that works over iMessage.

##What it does

Owners text a photo of their menu. JOLO reads it, asks whether it's the full menu or a partial
update, shows what changed, takes corrections in plain words ("the oat latte is $5.50, not $5"), and saves only after the owner says yes. Items missing from a partial photo are never deleted.
Customers text questions: what's on a café's menu, who has a matcha, what's open now, what's near
Palmer Square. Answers are checked against the stored menus, so JOLO can't name an item or a price
that isn't there.
Owners also get weekly insights from what customers ask for, and ideas for drinks to add or feature.
Ownership comes from the sender's phone handle, never from what the message says.

How we built it

Eyes (photo to menu data): Qwen3.5-9B with vision, served with vLLM on a GPU VM, reads the photo. A validation and confidence step flags items to double-check, and PaddleOCR gives a second read. The
output is a structured menu: sections, items, sizes, prices.
Brain: a LangGraph agent that calls tools (find café, get menu, save menu version) in a loop. Search combines BM25 keywords, vector embeddings (Qwen3-Embedding) and a reranker, built on LanceDB. A grounding check compares every answer to the store.
Voice and face: an iMessage bridge built on Spectrum, plus the JOLO app (Expo / React Native) where owners and customers see the same data.
Data: Firebase Firestore is the source of truth. Menus are stored as immutable versions, so every
change is kept as history. Real Princeton cafés were imported from OpenStreetMap.
Shipping: the whole backend runs in Docker, with FastAPI behind POST /v1/turn.

Challenges we ran into

A photo is not a menu. Glare, curved chalkboards and dense handwriting wrecked early reads. We tuned upscaling, added validation, and made the system show uncertain items to the owner instead of
guessing.
Never invent a price. If a menu doesn't show a price, we store none. Scraped or inferred text never becomes a price.
Tool calling. The model's native tool calling scored 19/30 on our scripted conversations, while
having it write one JSON action as plain text scored 26/30, so we shipped that.
Real data is messier than demo data. Connecting to the live database exposed a bug where our store
expected a map for café locations and the app stores a Firestore GeoPoint.
Cold start. A trending feature has nothing to say with no users. We designed it to start from
outside signals and shift toward our own usage data as it grows.

Accomplishments that we're proud of

Hybrid search with a reranker reached nDCG@5 of 0.941 on 50 hand-labelled queries over 22 cafés, up from 0.809 with keywords alone.
On 30 scripted conversations, the agent completed 26 (87%) of them, with 0.955 groundedness: answers stayed tied to real stored data.
Menu reader results: [fill in: item accuracy / price accuracy from menu_reader/README.md].
Owners must say yes before anything is saved, and every version is kept.
Over 300 automated tests, and one command to run the whole backend in Docker.
Real Princeton cafés in the same database as the app.

What we learned

Reliability beats cleverness for small models. Constraining output to one JSON action beat the
fancier tool-calling mode.
Asking the owner to confirm is part of the product, not a limitation.
Simulations can show that a feedback loop works. They can't show it works on real customers, so we
report them as simulations.

What's next for JOLO

Pilot with real Princeton cafés and let owners onboard themselves.
A trends engine that starts from public signals (Reddit, Google Trends) and learns from real
searches, with counts only and hashed phone numbers.
Turn on the personalization bandit once we have real interactions.
Push updates out to Google Maps and delivery listings.
Run the brain next to the model on one VM, with no tunnel.