Saffron & Smoke — Voice-First Restaurant Ordering

Inspiration

Ordering food in India still runs on busy phone lines and a dozen apps nobody wants to download. Watching a small restaurant lose orders during the dinner rush—staff scribbling on notepads, customers giving up on hold—I wanted to see if the interface could just disappear. What if you could order dinner the way you'd talk to a waiter: out loud, in plain words, with no app and no sign-up?

Saffron & Smoke is that idea, end to end—a real, operable modern-Indian restaurant where a customer talks to Arjun, an AI voice agent, and the whole back office (kitchen, billing, inventory, staff) runs on the same data foundation.

What it does

Order by talking. Tap once and speak. Arjun takes the order over a live voice connection—understanding the menu, accommodating "make it extra spicy" or "no peanuts", and reading back the total before placing it.
Onboards and remembers you. New callers are registered by voice and given a memorable 4-digit code; returning callers are identified by that code or their phone number, with their address, preferences, and allergies recalled.
Never delivers where it can't. Every delivery address is checked against a geofenced service area—out-of-zone orders are politely refused, not silently accepted.
Live kitchen display. The instant an order is confirmed, it streams to the kitchen screen in real time—no refresh, no polling.
A real back office. Role-based staff access, order edit/void/refund with an audit trail, one-tap inventory "86-ing", GST-correct billing (CGST/SGST split), and Razorpay payments—plus a dine-in mode driven by a QR code at the table.

How I built it

The stack, by layer:

Front end: Next.js 16 / React 19, deployed on Vercel. A voice-first UI where every agent action mirrors onto visual cards (menu, cart, delivery check, confirmation), plus a full admin/kitchen dashboard.
Back end: FastAPI on Google Cloud Run. The voice pipeline uses Pipecat + Gemini Live for native speech-to-speech, with a set of tools (search menu, register customer, add to order, confirm, etc.) the model calls during the call.
Database: Amazon Aurora PostgreSQL (ap-south-1)—the single source of truth for customers, orders, table sessions, menu, and payments.
Real-time + coordination: Upstash Redis for pub/sub order events (bridged to the browser over SSE), short-lived WebSocket tickets, and a global concurrency cap on live voice sessions.

Why Aurora PostgreSQL—a deliberate choice.

The whole app is really just different views of one Aurora database, so I needed an engine that does two very different jobs well:

Transactional integrity for orders and money—relational tables, foreign keys, and exact-decimal billing.
Semantic search for the menu—so "something creamy but not too spicy" actually finds the right dish.

Rather than bolt on a separate vector store, I used the pgvector extension inside Aurora. Menu items are embedded, and Arjun's search runs as a vector similarity query against the same database that holds the orders—using cosine similarity:

$$ \text{sim}(q, d) = \frac{q \cdot d}{\lVert q\rVert \,\lVert d\rVert} $$

That means there's no second datastore to provision, secure, or keep in sync—relational integrity and vector search live in one managed, production-grade engine, and Aurora scales the whole thing from one demo restaurant to thousands.

Challenges I ran into

Keeping the voice loop real-time. Publishing order events to Redis synchronously on the agent's async event loop stalled the Gemini Live connection—tool calls were being cancelled before they committed, so a customer would be told they were registered when they weren't. The fix was to offload event publishing to a thread pool with hard socket timeouts so nothing blocks the audio loop.
A leaked-concurrency bug under the free-tier session cap. Stopping the mic mid-call closed the browser socket, but the pipeline didn't tear down—so each restart held one of the limited live-session slots until a 15-minute watchdog fired. After a few taps, every line read "busy." I traced it to the transport firing a disconnect event without ending the pipeline, and fixed it by cancelling the pipeline task on client disconnect so the slot frees instantly.
Making an LLM trustworthy with money and IDs. Voice models drift on numbers and invent identifiers. I constrained Arjun to never do arithmetic—every total is read verbatim from the database response—and to never guess a menu item ID, only use IDs returned by a prior tool call.
Designing voice + screen as one. Every spoken action had to be reflected on screen with no hidden state, which shaped the entire tool/UI-event contract.

What I learned

Postgres is a bigger tent than I gave it credit for. Pairing relational guarantees with pgvector in Aurora removed an entire moving part from the architecture—and made the system simpler and more capable.
Voice UX is a latency and trust problem, not just an NLP one. The wins came from teardown correctness, real-time event plumbing, and strict guardrails—not from a fancier prompt.
"Demo-credible" means the boring parts work—RBAC, audits, refunds, GST, geofencing. That's where a voice toy becomes something a restaurant could actually run.