Traycer: Food-waste Intelligence

Turn leftovers into insights. Traycer helps kitchens track food waste and spot opportunities to waste less.
A live phone camera identifies hot dog, pizza, and salad with detection boxes and confidence scores.
Behind the scenes of the live phone camera streaming.
Review AI-identified leftovers and estimated weights, then accept or reject each capture.
Track demo waste totals, estimated cost and emissions, and Gemini's suggestions for reducing leftovers.
Explore estimated carbon and water impacts of food waste, translated into everyday comparisons.
See which dishes leave the most leftovers and how waste changes over time to guide kitchen decisions.
Live video calls with Traycer’s AI food-waste consultant and Gordon chef character, powered by Relay.
Simulated diners choose meals and return leftovers, revealing per-dish waste trends before a real service.

Inspiration

It started at our college dining halls. We kept watching trays come back to the dish return with food still on them, meal after meal, and nobody could say what was being thrown away or why.

The scale. The world wasted about 1.05 billion tonnes of food in 2022, 19% of all the food available. We throw away more than a billion meals' worth every day (UNEP Food Waste Index 2024).

The climate. Food loss and waste generate 8-10% of global greenhouse gas emissions, and are a particularly large source of methane (UNEP). Every uneaten portion carries the emissions and water it took to grow, move and cook it.

Hunger and health. About 645 million people may have faced hunger in 2025, and 2.69 billion could not afford a healthy diet (FAO, State of Food Security and Nutrition 2026).

The cost. UNEP estimates food loss and waste costs upwards of $940 billion a year. For any kitchen, an uneaten portion is food it bought, prepared and paid someone to cook.

What it does

Traycer turns the trash bin into data. It shows a kitchen which foods people leave uneaten, how much, and what that waste costs the planet. Then it lets the kitchen test a smaller portion or a different menu in a simulated service before cooking anything, so it can cook and serve less of what ends up in the trash.

1. A camera reads every returned tray. A phone camera spots each tray and plate and recognizes the food on it using computer vision. A separate vision model works out how much of each food is left, tray by tray, as it happens.

2. Every gram becomes carbon, water and cost. A real-time dashboard totals leftovers by food and converts them into estimated carbon, water and cost, with everyday comparisons like miles driven and showers.

3. Change the menu before you cook it. Choose a change, like a smaller portion of the food that keeps coming back, and the simulator runs a full service with the same simulated diners based on past data, showing leftovers and CO₂e. Every diner is its own AI agent, deciding what to take, who to sit with and how much to eat.

4. Two AI characters you can call. A chef and a food waste consultant on Relay answer texts and take voice and video calls. The consultant explains which foods are being wasted most, while the chef suggests recipes and portion changes to address them. Both can look up tray data and saved simulation results, so kitchen staff can ask questions, talk through recommendations and understand how a proposed menu performed in the simulation.

How we built it

We built Traycer around a connected workflow: capture food waste, understand the patterns, propose menu changes, and test them in a simulated cafeteria.

We turned a phone into a live camera that streams frames over WebSockets to a Python FastAPI backend. YOLO-World and OpenCV detect food and containers, while ByteTrack and our tracking logic help prevent the same tray from being counted twice. OpenAI vision refines food identification and estimates leftovers, with a review interface for staff to accept or reject captures.

We store tray observations and photos in SpacetimeDB and display them in a dashboard showing waste by food, estimated cost, and environmental impact. Gemini summarizes the capture data into readable insights. We also built a 3D cafeteria using React, Three.js, and React Three Fiber. A Node.js worker uses OpenAI to drive individual simulated students’ food choices, eating behavior, and conversations, while shared TypeScript rules and SpacetimeDB manage the world and waste accounting.

To make the findings actionable, we created two fictional AI assistants on Relay: LeBron, a food-waste consultant, and Gordon, a chef. Shared tools let them access camera reports, save recommendations, and retrieve simulation results. Pipecat and ElevenLabs support voice conversations, while SQLite preserves the evidence and links recommendations to their tests.

Finally, we built a workflow to compare baseline and proposed menus using the same seeded student cohort, with repeated baseline runs to measure variability. The simulation helps explore potential changes; the camera provides the real-world evidence to verify them at the next service.

=== SPONSORS ===

Best Use of SpacetimeDB

SpacetimeDB is the backend powering each and every component of Traycer. Three systems interact with two Spacetime modules across 25 tables:

Computer vision writes what the camera sees. Our Python vision service sends every tray observation to the traycer-camera module. One reducer replaces that tray's food rows and recomputes the per-food and per-service totals for grams, carbon, water and cost in a single transaction, so the totals always match the trays. The tray photos live in the database too, as JPEG bytes in their own table, and the dashboard reads both straight from Maincloud.
The agent simulation runs inside the database. A scheduled reducer ticks the world every 200 ms, handling movement, queues, seats, conversations and waste. When a student needs to decide something, the module writes a job row.
Relay agents read both. Our video agents, LeBron and Gordon Ramsey, query the camera database and use that information to run informed simulations themselves. When LeBron says bagels are the biggest leftover, he is quoting the same rows the dashboard shows.

The value for us was having one source of truth without building a backend for each system. Reducers gave us transactions, subscriptions gave us real-time sync for the simulation, and the worker and the 3D view can disconnect and reconnect without losing the world.

SpacetimeDB architecture Traycer architecture showing SpacetimeDB camera storage, authoritative cafeteria simulation, AI decision workers, live subscriptions, and the evidence bridge.

Relay Interactive Agents

We used Relay's agent audio and video calling capabilities to implement both an agent Gordon Ramsay, the world famous chef, and a commercial food waste agent consultant named Lebron into our cafeteria simulation. Lebron is able to read the food wastage CV data and the agent people's pattern data to figure out the Pareto-optimized simulation that has the least food waste, the least carbon emissions, and the most cost-savings via an agentic research loop. Gordon is able to read the agent people's opinion data on the menus to figure out the best recipe to serve to the agent people which would lead to the least food wastage.

Talk to Gordon and get your recipes figured out.

Best Use of Notability

We used Notability as Traycer’s working notebook throughout the entire project, connecting handwritten product sketches and architecture diagrams with build checklists, debugging notes, and real detector outputs and dashboard screenshots that we annotated directly on the page. During a late-night team meeting, we also used recording + note sync to keep our discussion connected to our notes, bringing the proposed design, missed food detections, stale dashboard totals, and next tests into one place where someone else could follow our thinking. We absolutely loved collaborating on Notability!

Our handwritten engineering plan in Notability: the proposed camera, vision, review, store and live UI pipeline, with the event contract, CO2e math, build gates and open decisions.

A real photo test of the detector, annotated by hand in Notability: the boxes pick up the paper beside the plate and miss the pizza label, so we noted trying a tighter crop.

Notability's audio recording and live transcript beside our handwritten test notes: 14 of 17 vision-model tests passed, and the same 3 failed until we forced the provider in a scratch run.

Best Use of ElevenLabs

We used ElevenLabs to animate our Relay agent chef Gordon Ramsay and commercial food waste consultant Lebron and also used it to give them a voice when they talk. For Gordon Ramsay, that means his signature angry British voice using the British preset voice "George" from ElevenLabs and for the consultant, we decided to have it speak in a professional, calm but also authoritative manner using the American preset voice "Eric" from ElevenLabs. For the agents speaking, we used the eleven_v4_turbo model and for the agents hearing, we used the scribe_v2_realtime model.

[MLH] Best Use of Gemini API

Gemini turns Traycer's live waste counts into a plain-language next step. As trays are counted, Gemini reads the real totals (how many trays, how much food is left, and which foods) and writes a two-sentence read of the service: what's being left behind most, and one small thing to try. It updates while the service is still running, so a kitchen sees the answer while there's time to act, not in a report afterward. The numbers beside it, including estimated CO₂e and water, come from our own math. Gemini does the part that needs language: telling a busy operator what those numbers mean for the next service. We call gemini-2.5-flash through the Gemini API with structured JSON output.

Live run of Traycer's Insights card: Gemini reads the real tray totals and names the main leftover (bagels) with one small experiment to try, next to the dashboard's own At a glance figures.

Figma Best Design

Navigate our landing page here! It's really cool, we promise :)

Traycer's design goal was to make a hard topic, food waste and its climate cost, feel calm and easy to read. One visual system runs through a scroll-choreographed 3D landing page, the live camera, the dashboard and the simulator: a pastel-green field, navy ink, warm paper, Manrope type with tabular numerals for data, capsule controls, and a family of faceless capsule characters. The landing's four chapters (Serve, Observe, Understand, Reimagine) glide a camera through an animated cafeteria, and a glass droplet follows the cursor over the hero. On the dashboard, estimated carbon is the largest figure, with water and meals beside it and everyday comparisons like miles driven and showers. Numbers ease in over 600 ms, charts keep text labels, motion follows the reduced-motion setting, and every page has a skip link.

Traycer desktop wireframes: landing page, live camera, dashboard, simulation, demo notice, and camera connection Traycer’s desktop wireframes in Figma connect the story to the workflow: introduce the problem, capture leftover food, understand its environmental impact, and explore changes to portions. A consistent layout and clear hierarchy tie the experience together.

[MLH] Best .Tech Domain Name

We named our domain trayitforward.tech. "Pay it forward" means doing something now that helps whoever comes next, and that is the whole idea behind Traycer. A camera reads what comes back on a returned tray, the dashboard turns it into an estimate of the food, carbon and water left behind, and that picture helps a kitchen decide what the next service looks like. What one tray teaches goes forward into the next meal. The domain opens Traycer's live landing page. We chose .tech because the whole project is a tech build: computer vision, a live database and an agent-driven simulation.

Challenges we ran into

Across the team, we tackled three very different kinds of real-time systems. The camera needed to recognize leftovers without counting the same tray repeatedly. The simulation needed to coordinate independent AI diners while keeping movement, queues, conversations, and waste totals consistent. Our voice and video agents brought another set of challenges: calls could connect without carrying audio or video, and interruption detection sometimes cut off replies before they finished. We worked through tracking rules, agent validation, network routing, and turn-taking to address those problems. Bringing everything together also meant giving the dashboard, conversational agents, and simulation shared records so they could refer to the same findings, recommendations, and test results.

Accomplishments that we're proud of

We're proud of how much we built together: a live camera and waste dashboard, a 3D dining simulation, and two AI characters people can text or call with voice and video. In the simulation, diners have distinct personalities, choose their food, talk to one another, and leave different amounts behind. The dashboard turns tray captures into estimated cost, carbon, and water impact, with Gemini summarizing the findings. We also connected our food-waste consultant and chef to shared evidence, saved recommendations, and simulation results, creating a workflow from identifying waste to proposing and testing a menu change. Seeing each teammate's work contribute to that shared experience was one of the most rewarding parts of the hackathon.

What we learned

We learned how to bring computer vision, real-time cloud data, and AI agents together into a working product. Building Traycer taught us how to track objects across video frames, give vision models clearer instructions, and keep a dashboard synchronized with incoming captures. Creating the dining simulation also gave us experience coordinating individual AI agents within a shared world. Beyond the technical work, we learned how much product design matters: translating food waste into carbon, water, and cost makes the data easier to understand, while pairing it with a suggested action gives kitchens a clear next step.

What's next for Traycer: Food-waste intelligence

Next, we want to give Traycer new ways to prevent waste. It will forecast how much of each dish a kitchen should prepare, nudge cooks mid-service when a pan is running too full, and route surplus that was never served to food banks and compost instead of the trash. It will track every kitchen's carbon, water and savings over time, so a restaurant group, hotel chain or caterer can compare sites and report real progress, and it will steer menus toward lower-carbon swaps that people actually eat. Our goal is a kitchen that knows what it would waste before it cooks, and cooks less of it.

P.S. Spot the Easter egg: our simulated diners are rocking sponsor merch and U-M gear. Go Blue!