Shelfy

• 

  logo

Inspiration

We started with a scene we couldn't shake. Picture a volunteer at a small food pantry. There is a line out the door, and a caregiver walks up holding a donation box and asks, "My grandmother has a peanut allergy. Is anything here safe for her?" The volunteer looks at the shelf and sees tiny ingredient labels, faded dates, and cans that could be under a recall nobody in the building has heard about. So they do their best, and they hope. Then it clicked. The challenge asked us to translate confusing documents into plain language, and we realized a pantry shelf is a document too. It is scattered, technical, English only information that nobody designed for a person under stress. A grocery shopper gets recall alerts and clear labels for free. The family at the pantry gets a shrug. That gap is what we set out to close.

What it does

A volunteer waves each donation past a phone camera. In a couple of seconds ShelfSight reads the label, checks the date, tags allergens and dietary categories, checks the item against live FDA and USDA recall feeds, and adds it to a searchable inventory. It routes every item one of three ways: shelf it, flag it, or send it to a human. On the family side, a caregiver can filter for needs like "no peanuts," and ShelfSight explains in plain language, in their own language, what each item is and why it was flagged. A volunteer clears every item. It never marks anything as safe to eat. How we built it

How we built it

The intake station is just a phone on a stand running our web prototype. Computer vision and OCR read the label and date, a classifier tags allergens and dietary categories, and an agentic workflow checks each item against the openFDA and USDA FSIS recall feeds. A generative large language model then does the plain language reasoning. We grounded it on only the scanned label and the retrieved recall data, so it cannot invent an allergen or a recall it never saw. We tuned recall matching to flag rather than miss, because a missed recall is far worse than a double checked safe can.

Challenges we ran into

Reading labels off dented cans and faded dates was harder than it looked. Recalls are often issued by lot number, which OCR cannot reliably catch off a real donation, so instead of overclaiming we match on brand and product and send close matches to a human. Keeping the language model honest took real work, and the fix was grounding it so it can only describe what was actually scanned and retrieved. The hardest call was the threshold itself, since tuning the system to flag too much rather than too little means more volunteer checks but no silent misses.

Accomplishments that we're proud of

We're proud that we built something a real pantry could actually use, not a demo that only works in a slide. It runs on a phone propped on a stand, which is hardware these places already own, so there is no scanner to buy and no system to install before someone can get value out of it. The accomplishment we care about most is the discipline we put into what the AI refuses to do. It would have been easy to let the model say safe to eat and call it impressive, and instead we drew a hard line that a human clears every item, and that one decision is what turns ShelfSight from a clever prototype into something a pantry could trust. We also got the hardest technical piece honest rather than flashy. Recalls are usually issued by lot number, which OCR cannot reliably read off a dented can, so rather than fake exact matching we built it to match on brand and product and route the close calls to a person, and we grounded the language model on only what it actually scanned and retrieved so it cannot invent a recall or an allergen that was never there. And we're proud that we tuned the system the way the stakes actually demand, flagging rather than missing, because we would rather a volunteer spend ten seconds double checking a good can than let one recalled jar reach a senior or a child who cannot afford for it to be wrong.

What we learned

The hardest part was not the model, it was deciding what the AI should refuse to do. The moment we committed to never saying "safe to eat," the whole system became something a pantry could actually trust. We also learned why high recall beats high precision when a single miss can put a senior or a child in the ER. What it does

What's next for Shelfy

More output languages, a map that shows food banks which neighborhoods are going without, and a clean handoff into the manual safety checks pantries already run.

Built With

• claude
• expo-blur
• expo-camera
• expo-image-manipulator
• expo-image-picker
• expo-sqlite
• expo.io
• gemini-2.5-flash
• google-gemini
• google-genai
• matplotlib
• openfda
• python
• react-native
• react-native-reanimated
• react-navigation
• sqlite
• typescript