CheckMate

Inspiration

Splitting a check after dinner with friends shouldn’t turn into guesswork, smudged receipts, and “what did I order again?” moments make it hard to be fair. We wanted something that feels as simple as taking a photo, but still trustworthy enough that everyone agrees on the line items before anyone pays.

What it does

CheckMate is a receipt splitter for groups. You upload a photo of the bill, our backend uses Gemini API to read the restaurant name, line items, prices, tax, and tip. When a line looks unreliable, we run an automated menu verification step (Browser Use) so fuzzy or garbled items get clearer names before you assign dishes to people. From there you split items (including an even “split the whole bill” option), and see totals per person, so nobody’s confused about what they’re paying for.

How we built it

Frontend: React + TypeScript + Vite, Zustand for state, Tailwind and shadcn-style UI. Deployed on Vercel
Backend: FastAPI + Uvicorn, with /scan for receipt parsing and /heal / /heal-batch for verification. We batch low-confidence lines into one browser session where possible to save time and cost.
AI stack: Gemini API for vision/OCR-style parsing, Browser Use + Playwright for web menu lookup when items need healing.
Hosting: API on Railway, UI on Vercel; GitHub for source control and deploys.

Challenges we ran into

Hosting: The app is really two products, a static React app and a long-running Python API, so we couldn’t “deploy GitHub to one place and call it done.” Wiring Vercel to the Railway API meant getting VITE_API_URL right, serving the API over HTTPS, and remembering that Vite bakes env vars in at build time (so missing or stale config silently broke scans until we redeployed).
Browser-use: Menu verification needs a real browser stack (Playwright), enough time and memory per run, and sane timeouts. Running that in the cloud is nothing like calling a single REST endpoint. We had to tune steps, handle failures, and batch multiple uncertain lines so we weren’t opening a full browser session for every item.
Gemini / OCR: Receipt photos are messy due to glare, folds, fonts, and abbreviations, so parsed line items aren’t always trustworthy out of the box. We had to combine confidence heuristics, healing for bad lines, and honest fallbacks when the model or image quality couldn’t produce a reliable read.

Accomplishments that we're proud of

End-to-end flow from photo → parsed receipt → optional menu-backed healing → assign → settle.
A batched heal path so multiple uncertain lines don’t each spin up a full separate browser run.
Actually shipping: frontend + API deployed, not just a demo on localhost.

What we learned

Environment-specific behavior (proxy vs absolute URL) is easy to underestimate; production debugging belongs in the Network tab and deploy logs.
Separating concerns (static UI vs Python API) matches how real teams ship, but it demands discipline around env vars and CORS/HTTPS.
AI + automation is powerful for messy real-world inputs, but needs guardrails, timeouts, and honest fallbacks when something can’t be verified.

What's next for CheckMate

Deeper payment handoff (e.g. Venmo / payment links) and optional shared session links so the whole group can review on their phones.
Stronger offline / retry UX and clearer copy when healing isn’t available.
Cost/latency controls for browser-use (caching, smarter batching, provider options).
Optional accounts or history so repeat restaurants get faster, more accurate behavior over time.