SNAPback

Turn every grocery receipt into real cashback for SNAP/WIC families, when they actually shop healthier.

Comment

💡 Inspiration

42 million Americans use SNAP every month. The cheapest, easiest thing on the shelf is almost always the worst thing for them.

We started this project after reading the FAO's problem statement for this hackathon. It nailed something we've all seen in our own families, generic "eat more vegetables" advice doesn't help a Caribbean grandmother shopping for callaloo, or a South Asian household buying atta and dal on a $480 monthly budget. The system tells low-income families what they should eat, but never meets them where they actually shop.

We wanted to build something that wouldn't just inform people. We wanted to pay them to shop healthier, and only pay them when the change was real, not just a one-time photo-op.

🛒 What it does

SNAPback is a Flutter mobile app that turns every grocery receipt into personalized, culturally-aware nutrition guidance, and rewards SNAP/WIC families with real cashback when they sustain healthier shopping habits.

Here's the user flow:

Sign up, tell us your monthly SNAP budget, family size, and which cuisines you cook at home (we support 24, Caribbean, South Asian, West African, Latin American, East Asian, Middle Eastern, and more). After every grocery trip, snap a photo of your receipt. Google Gemini 2.5 Flash reads the receipt multimodally, every item, quantity, and price, and scores each edible item 0–5 against a USDA-derived healthy-foods rubric (the FAO problem statement actually pointed us to USDA's resources for this).

You earn cashback per item:

5% for 5★ items,

3% for 4★,

1% for 3★,

+1pp bonus for cultural matches.

Your cashback accrues toward a monthly cap of min(10% × your SNAP amount, $25 × family size). Here's the part nobody else does, that cashback only unlocks if your monthly average health score stays at or above 4 out of 5. We don't pay you for one healthy haul. We pay you for sustained behavior change. We also generate nutritious recipe suggestions based on what you just bought, in your declared cuisines. The dashboard shows your monthly health score, cashback earned, days until reset, and whether your cashback is currently redeemable or still locked behind the gate. Every screen updates in real time as new trips process.

🛠 How we built it

Mobile (Flutter / Dart): MVVM architecture with Provider for state, go_router plus an IndexedStack for persistent bottom-tab navigation, fl_chart for the score ring, google_nav_bar for the smooth pill nav, and a custom animated SnapbackLoader widget. Light + dark themes, bouncing scroll physics everywhere, accessibility labels on every key surface (screen readers announce the score ring, redemption badge, and trip rows).

Auth + data: Firebase Auth (email/password), Cloud Firestore with live streams powering auto-refresh on Dashboard, History, and Trip Result, Firebase Storage for receipt images.

Server logic (Firebase Cloud Functions, Node 20, 2nd gen): Two functions doing all the work.

processReceipt triggers on Storage upload → calls Gemini 2.5 Flash with a strict JSON Schema and a long, carefully-tuned system prompt (USDA criteria + an explicit blocklist for non-edible items like toilet paper, laundry, household cleaners) → writes structured items back to Firestore. scoreTrip triggers on the processReceipt write → applies tiered cashback, computes the item-weighted monthly health average, evaluates the redemption gate, generates recipe suggestions via a second Gemini call, and finalizes the trip. The Gemini API key lives in Firebase Secrets (defineSecret). Schemas are enforced server-side with responseJsonSchema.

Backend reliability: We learned the hard way to create the Firestore trip document before uploading the image to Storage, that fixed a nasty race condition where processReceipt would fire before the doc existed.

😅 Challenges we ran into

The first OCR pipeline was a disaster. We started with traditional OCR + a separate scoring step. Receipts came back garbled, items merged, prices wrong. One test scored a $40 grocery trip as 0 points and detected zero items. That night we ripped it out and rebuilt the whole pipeline around Gemini 2.5 Flash multimodal, one model, structured JSON output, and a rubric grounded in USDA criteria. Night-and-day difference.

Race conditions in Firestore. processReceipt was firing before the trip document existed (5 NOT_FOUND). We restructured the upload flow to write the doc first, then the image.

The reward system was way too generous. Our first version was paying out $5–$10 per receipt. Sustainable for a hackathon demo, catastrophic for a real public-benefits program. We rewrote it to use per-item tiered cashback, a monthly cap proportional to SNAP allocation and family size, and the monthly health-score redemption gate that turned out to be our most original idea.

Non-edible items were inflating receipts. Toilet paper and laundry pods were getting scored. We added an explicit blocklist in the Gemini system prompt and a server-side keyword filter as defense in depth.

Onboarding had a bug where selecting a cultural preference auto-navigated to the dashboard before the user could hit "Continue." We tracked it down to an implicit onboardingComplete getter and replaced it with an explicit flag set only on the final step.

🏆 Accomplishments that we're proud of:

The redemption gate. Most cashback apps pay you for any purchase. We only release the money when your monthly average health score is 4/5 or higher. That's not punishment, that's the policy lever that turns nutrition science into action. We genuinely haven't seen another product do this. Cultural personalization that actually means something. 24 cuisines mapped to a real bonus tier and to recipe suggestions. A Caribbean family doesn't get told to eat quinoa; they get rewarded for plantains and callaloo and recipes they'd actually cook tonight. Gemini 2.5 Flash multimodal end-to-end. No separate OCR, no preprocessing, no fragile pipeline. One model reads the receipt and scores it against USDA criteria in a single structured-JSON call. Judges can scan a receipt and see the breakdown in 3 seconds. Live everything. Dashboard, History, and Trip Result all update in real time off Firestore streams. No pull-to-refresh anywhere. A redeemable cashback system that's actually safe to ship. The cap math (min(10% × SNAP amount, $25 × family size)) keeps it grounded in a realistic public-benefits budget.

📚 What we learned

Multimodal LLMs replace entire pipelines. What we built in two days with Gemini would have taken a small team a quarter to build with classical OCR + ML scoring. Behavioral economics > more reward. The first version of SNAPback paid more cashback. The current version, with the gate, is a better product and a better policy. Constraint creates meaning. Public-health UX is different. SNAP recipients skew older, more diverse, and more stressed than typical app users. Bouncing scroll, large bold typography, persistent bottom nav, and screen-reader labels weren't nice-to-haves, they were the whole game. Race conditions hide in the order of operations. "Write the document before you upload the file" is the kind of thing you only learn after the third NOT_FOUND log.

🚀 What's next for SNAPback

Real EBT pilot integration. Today the cashback is a tracked virtual balance. The architecture is ready to load unlocked cashback directly onto an EBT card on the 1st of each month via state pilot programs. Full localization. ARB files for Spanish and Mandarin (the language picker is currently English-only with the other locales flagged "coming soon").

WIC-specific item recognition. Formula, eligible categories, store-redemption codes. Push notifications for "Cashback unlocked" on the 1st and progress nudges through the month. Open community-vetted cuisine database. The cultural-bonus list should grow from real households, not from us.

Built With

  • cloud-firestore
  • dart
  • firebase
  • firebase-auth
  • firebase-cloud-functions
  • firebase-storage
  • fl-chart
  • flutter
  • gemini-2.5-flash
  • go-router
  • google-gemini
  • google-nav-bar
  • node.js
  • provider
  • usda
Share this project:

Updates