FridgeBridge

Upload a picture of your fridge
Gemini analyzes your fridge and determines recipes based on availability and your preferences (memorized through Backboard)
Call pantries near you, with notes about their stock from previous calls
Twilio completes the call for you, taking notes about their stock
Get a breakdown of your plan to receive your ingredients if they have what you're looking for!
Project architecture

Inspiration

Food insecurity affects 44 million Americans, but the barrier isn't always availability — it's information. Nearly 42% of UC students experience stress and anxiety due to food insecurity — and for many, the hardest part isn't finding a pantry, it's making the call. Not knowing if it's open, if it has what you need, or if the trip is even worth it. Asking for help is already vulnerable. We wanted to eliminate that friction entirely so that getting help feels as easy as checking your fridge. Food is out there. Help is out there. The bridge is just broken — and we built it.

What it does

FridgeBridge is a full-stack AI pipeline that turns a fridge photo into a personalized meal plan and automatically calls local food pantries to find your missing ingredients. Snap a photo, and Gemini identifies what you have and suggests recipes tailored to your preferences using Backboard's persistent memory. Pick a meal and FridgeBridge finds nearby pantries via MongoDB geospatial search and OpenStreetMap. Then an AI voice agent calls up to five pantries in parallel over Twilio, holds real-time conversations about availability, and streams live transcripts to your dashboard. When the calls finish, you get a Mapbox route to pick everything up. The entire pipeline — vision, memory, geospatial lookup, parallel voice calls, and routing — runs in under two minutes.

How we built it

We split the work across four engineers over 20 hours. The frontend is Next.js 16 with React 19, Tailwind CSS, and react-map-gl for the interactive pantry map. The backend is a FastAPI Python service that handles all AI workloads and call orchestration. Gemini 2.0 Flash powers fridge vision, recipe generation, and call analysis. ElevenLabs provides voice synthesis and Twilio makes the outbound calls, fanned out in parallel via asyncio. MongoDB Atlas stores pantry data with geospatial indexes, call history, and user-to-assistant mappings. Backboard persists user memory across sessions — dietary preferences, cooking style, ingredient habits — and injects them into every recipe suggestion. Real-time call updates stream to the browser over WebSockets.

Challenges we ran into

The hardest part was orchestrating parallel voice calls with real-time transcript streaming. Coordinating Twilio gather webhooks, async call state, and WebSocket pushes into one coherent live dashboard required careful async design — one race condition would silently drop results. We also had to wire Backboard's memory layer correctly: raw events stored verbatim aren't useful, so we distill each recipe interaction into short preference statements before storing them, which actually influence future suggestions in a meaningful way.

Accomplishments that we're proud of

A single photo triggers a fully autonomous pipeline — vision, personalized recipe generation, geospatial pantry lookup, parallel AI phone calls, and a routed pickup plan — in under 2 minutes. The voice swarm actually works: multiple pantries picked up, the AI held natural conversations, and the structured summaries came back correctly. Watching live transcripts stream in while calls were still in progress was a genuinely exciting moment for the team. Seeing Backboard memory visibly shape recipe recommendations across sessions showed that the personalization layer is real, not cosmetic.

What we learned

How to chain multiple AI modalities — vision, text, and voice — into a single seamless user-facing flow without the seams showing. We also learned that Backboard is a surprisingly powerful primitive for giving AI apps persistent, structured memory without building your own storage layer — but that what you store matters as much as the fact that you store it. Distilling user interactions into concise preference statements rather than logging raw events is what makes memory actually useful.

What's next for Fridge Bridge

Give food banks a simple dashboard to update their own inventory in real time, so the AI has accurate data before calling
Build an SMS-based version so families without a smartphone can access the same pipeline via text message
Expand beyond food to other community resources — clothing banks, shelters, free medical clinics