FreshLedger

Turn one grocery receipt into an editable ledger, freshness-aware pantry, rescue meals, exact waste costs, and smarter shopping advice.

Comment

Inspiration

A grocery receipt is usually discarded moments after checkout, even though it contains the information needed to answer three practical questions: What did I buy? What will spoil first? How much money am I losing to waste? FreshLedger turns that forgotten paper trail into a living food inventory.

We built it during OpenAI Build Week to explore how far a single resumable Codex session could take a safety-sensitive product from specification to a tested mobile demo.

What it does

FreshLedger closes five connected loops:

  • Receipt to ledger: photograph or choose a grocery receipt and convert it into editable line items, quantities, prices, and reconciled totals.
  • Grounded storage guidance: match recognized foods to a curated 100+ row USDA/FoodKeeper-derived reference, then show fridge, freezer, or pantry guidance and conservative eat-by windows.
  • Freshness-aware meals: rank the virtual fridge by urgency and suggest deterministic rescue meals that use food before it expires.
  • Exact waste-cost tracking: mark an item as eaten or spoiled and calculate the exact cost lost.
  • Purchase insights: use confirmed SQLite history to surface category spending, waste rate, and buy-less/buy-more guidance.

The app has three transparent modes. Demo runs original synthetic receipts through the real safety and persistence pipeline. Local OCR reads a real receipt on the FastAPI host with RapidOCR and ONNX Runtime, with zero cloud calls. A strict GPT-5.6 vision adapter is retained as an explicit future integration, but the submitted UI keeps it disabled so a user cannot accidentally spend API credit. Every scan reports its provider, model, mode, and whether AI was called.

How we built it

The client is Expo SDK 57, React Native, TypeScript, and React Native Paper. The server is Python/FastAPI with SQLite transactions and integer-cent accounting.

The zero-cloud recognition path uses PP-OCRv6-small through RapidOCR/ONNX Runtime. A deterministic spatial parser reconstructs receipt lines, separates totals and payment text from products, handles quantities and weighted items, and reconciles parsed prices. Product grounding is intentionally strict: only normalized equality with an explicit alias can unlock a canonical food and reference shelf life. Fuzzy matches are review-only.

Food safety is handled after recognition, not trusted to OCR output. Unknown or contradictory foods abstain to conservative defaults; prepared or moist foods cannot inherit dry pantry durations; eat-by windows cannot exceed storage duration; and identity or category edits immediately hide stale advice until the server re-grounds them.

GPT-5.6 Sol in OpenAI Codex served as FreshLedger's primary implementation engineer throughout one resumable session: architecture, Expo and FastAPI code, SQLite persistence, tests, fixture creation, debugging, adversarial review, documentation, and Git history. The optional OpenAI Responses API adapter uses a strict Structured Outputs schema for GPT-5.6 vision when explicitly enabled.

Challenges we ran into

Receipts are visually inconsistent and full of abbreviations, discounts, weighted produce, and lines that look like products but are actually totals or payment details. Reconstructing the page from OCR coordinates was much harder than extracting plain text.

Food-safety mistakes have asymmetric cost: abstaining is inconvenient, but confidently assigning a dry-rice shelf life to rice pudding is dangerous. Independent adversarial reviews uncovered compound-food, stale-UI, cache, and concurrency edge cases. We fixed them with exact grounding, conservative fallbacks, immediate advice invalidation, versioned draft caching, and serialized confirmation transactions.

We also chose a zero-cost submitted experience. That required a useful real-photo path without pretending that deterministic Demo data or Codex development work was a paid runtime GPT call.

Accomplishments that we're proud of

  • A complete photo → review → ledger → pantry → meals → waste → insights loop.
  • Real receipt-photo processing with no API key and no cloud token usage.
  • Three committed receipt fixtures passing exact 5/7/8-item identity, category, quantity, price, reconciliation, and storage-safety gates.
  • 139 passing server tests, 6 app tests, TypeScript type-checking, Expo Doctor 20/20, and a production Web export.
  • Merchant-scoped correction memory that learns only from explicit, safely grounded confirmation.
  • Two independent final code reviews with no remaining blockers.

What we learned

For safety-sensitive AI products, provenance and abstention are product features. Recognition, identity grounding, and advice generation should be separate layers with independent validation. We also learned that deterministic local tools and an optional model adapter can share one clean contract without hiding which path actually ran.

Codex was most valuable not just for generating code, but for sustaining a long implementation thread: reading the handoff, preserving constraints, running tests, coordinating independent reviews, and turning failures into targeted regressions.

What's next for FreshLedger

Next we would expand the receipt benchmark across more stores and languages, add barcode/UPC fallback, move more inference on-device, improve accessibility and household sharing, and deploy a hosted judge build. With funded API access, we would enable the existing GPT-5.6 Structured Outputs adapter as an explicit opt-in mode and compare it against Local OCR using the same safety and reconciliation gates.

Built With

Share this project:

Updates