Pantry Guard

Inspiration

In 2024, a Boar's Head listeria outbreak killed 9 people and hospitalized dozens more. Most victims had no idea there was a recall — they just kept eating the same deli meat they'd bought every week. That same year, an avian flu outbreak wiped out egg supplies nationwide, leaving store shelves bare overnight. These weren't fringe events. They were systemic failures in how recall information reaches ordinary people. The FDA publishes every recall. The data is all there. But almost nobody looks at it — and that's the real problem. There's no push, no personalization, no moment where the information meets you where you already are: standing in your kitchen, putting groceries away.

What it does

Pantry Guard turns your grocery receipt into a personal food safety monitor. Snap a photo of your receipt, and within seconds we've extracted every line item, decoded the abbreviated SKUs into real product names, and cross-referenced them against live FDA enforcement data using semantic similarity matching. Anything flagged shows up directly in your pantry — with the recall classification, the reason, and exactly what to do. No accounts. No manual entry. Just point, scan, and know.

How we built it

The pipeline runs in three stages. First, a vision model (Llama 4 Scout) OCRs the receipt and extracts raw line item codes. Second, Llama 3.3 70B decodes those abbreviated SKUs into clean food names with brand, shelf life, and category. Third, we embed both the pantry items and the full FDA enforcement database using sentence-transformers/all-MiniLM-L6-v2 via Hugging Face, then run cosine similarity matching to surface semantic matches — catching cases where the product name and the recall description don't share a single word. Groq powers the LLM calls for low-latency inference. The frontend is a native-feeling mobile web app with swipe-to-delete, a live recall feed, and a liquid glass UI.

Challenges we ran into

Grocery receipts are not designed to be human-readable, let alone machine-readable. "KS SALMN FLTD" is Kirkland Signature salmon fillets. Getting a model to reliably decode those abbreviations — and only return food items, not laundry detergent — required significant prompt iteration. The other hard problem was matching: a recalled product might be described as "ready-to-eat poultry products" in FDA data and "Kirkland Sliced Turkey" on a receipt. Exact string matching fails completely. Semantic embeddings were the only approach that worked.

Accomplishments that we're proud of

Getting the full pipeline — OCR → product lookup → FDA semantic match — to run in under 10 seconds on a real receipt photo. And the recall matching actually works: we tested it against products involved in real Class I recalls and the cosine similarity scores surfaced the right matches without a single keyword in common.

What we learned

Prompt framing matters more than prompt length. A 9-rule system prompt produced worse outputs than a 4-bullet one because the model spent cognitive overhead acknowledging exclusions instead of just applying them. We also learned that semantic search on short strings is surprisingly sensitive to how you construct the embedding input — concatenating brand, product name, reason for recall, and lot code info into a single string dramatically outperformed embedding names alone.

What's next for Pantry Guard

Push notifications when a new FDA recall matches something already in your pantry. Barcode scanning as an alternative to receipt OCR. Expansion beyond Costco to any grocery store. And a deeper focus on the communities that need this most — elderly shoppers, immunocompromised individuals, and families shopping at discount stores where recalled products sometimes stay on shelves longer. These are the people who check FDA.gov the least, and who get the sickest when recalls go unnoticed.

Built With

• css
• html
• javascript
• node.js