PlatePilot

• 

Inspiration

Most nutrition apps are reactive—they track what you’ve already eaten. But at a wedding, party, or buffet, the damage is done before you log it. We wanted to build a proactive advisor that acts as a "Biological System Engineer." By using Gemini 3 Pro’s vision, we help users navigate food spreads based on their physical stomach capacity and health constraints (like lactose intolerance) before they even pick up a plate.

What it does

PlatePilot is a smart buffet assistant. The user inputs their current hunger level and health profile. Using a single photo of a buffet spread, the app:

Identifies Dishes: Uses Gemini 3 Pro to recognize food and detect hidden ingredients (like cream-based sauces for the lactose-sensitive).

Capacity Mapping: Ranks food by "Satiety-to-Volume" ratio to prevent early bloating.

Tactical Advice: Provides a "Plate Strategy" (e.g., "Protein Pivot" or "Salad Primer") to maximize the experience without the physical crash.

How we built it

Model: Gemini 3 Pro (via Google AI Studio) for advanced multimodal reasoning and vision.

Frontend: React with TypeScript and Tailwind CSS, allowing for a real-time "Verdict" dashboard.

Backend Logic: A specialized System Instruction set that converts visual data into a "Stomach Capacity Cost" score.

Context Management: A JSON-based user profiling system that ensures the AI "remembers" sensitivities across the session.

Challenges we ran into

The biggest challenge was "Hidden Ingredient Recognition." It’s easy for an AI to see "Pasta," but difficult to determine if it’s "Alfredo" (Dairy) or "Agliolio" (Safe). We leveraged Gemini 3 Pro's high-reasoning capabilities to look for visual cues like sauce sheen and viscosity to make high-probability guesses about allergens.

Building an AI that understands the chaotic environment of a buffet presented several "tasty" technical hurdles:

The "Messy Plate" Problem (Object Overlap): Unlike staged food photography, buffet plates are crowded. Foods overlap, sauces bleed into one another, and items like rice can act as a "bed" for other proteins. We overcame this by using Gemini’s multimodal reasoning to perform "semantic segmentation" through prompting. Instead of just asking "what is this?", we instructed the model to "deconstruct the plate layer-by-layer," allowing it to identify hidden ingredients like dressing under a salad or a meatball buried in pasta.

The Scale & Perspective Paradox: A single slider can look like a full-sized burger without a reference point. To solve the volume estimation issue without specialized hardware (like LiDAR), we implemented Contextual Anchoring. We trained the prompt to use standard buffet "anchors"—the known diameter of a standard dinner plate or the size of a lemon wedge—to calibrate its sense of scale and provide more accurate portion-size estimates.

Hallucination vs. Nutritional Accuracy: Early iterations of the model sometimes "guessed" ingredients that weren't there (e.g., assuming all white sauces were Alfredo). We mitigated this by implementing a Confidence-Score Gatekeeper. If the AI is unsure, it’s programmed to ask the user a clarifying question (e.g., "Is this creamy sauce dairy-based or a vinaigrette?") rather than providing potentially dangerous nutritional data for users with allergies.

Latency in the "Lunch Rush": Waiting 10 seconds for an AI response doesn't work in a fast-moving buffet line. We optimized our pipeline by implementing asynchronous API calls and image compression. By pre-processing images on the edge before sending them to Gemini, we reduced the "snap-to-insight" time by 40%, ensuring the app remains usable in real-time social settings.

Accomplishments that we're proud of

Successfully creating a "Pre-emptive Strike" UI that advises the user before they eat.

Implementing a "Stomach Capacity" slider that dynamically changes the AI's recommendations based on how much room the user wants to save for dessert.

What we learned

We learned that multimodal LLMs are not just for "labeling" images; they are incredible at spatial and physical reasoning—understanding that a heavy starch occupies more "stomach real estate" than a leafy green.

What's next for PlatePilot

While the current MVP focuses on immediate visual feedback, we envision [Project Name] evolving into a comprehensive biological management suite:

Real-Time Hydration & Alcohol Impact: Integrating Gemini's vision to identify beverage types (cocktails vs. water) and calculating the "Hydration Tax." This would provide alerts like: "You've had two high-sodium appetizers; we recommend one full glass of water before your next plate."

Wearable Bio-Sync (Apple Health/Garmin): Connecting to real-time blood glucose and heart rate data. If the app detects a "Sugar Spike" via a wearable after the first course, it will automatically adjust the "Plate Strategy" to favor high-fiber or protein-heavy options to stabilize energy levels.

Crowd-Sourced "Buffet Intelligence": In a social setting, multiple users could contribute photos of the same buffet. The AI would aggregate these views to create a 3D "Nutritional Heatmap" of the room, guiding users toward the freshest or most diet-compliant stations.

Microbiome Personalization: Allowing users to upload gut health data or specific enzymatic deficiencies (beyond lactose) so the AI can flag "High-Inflammation" foods specific to their unique biology.

"Re-create This" Grocery Integration: A feature that takes a photo of a delicious buffet dish and instantly generates a healthy, home-cookable recipe and a shopping list via the Instacart or Walmart API.