SmartPlate AI

Inspiration

Food choices affect both our health and the environment, but it can be difficult to understand both at the same time. We wanted to build something that makes this information simple, visual, and easy to act on.

That led us to create SmartPlate AI, an AI-powered food assistant that helps users make healthier and more sustainable food choices before or after cooking.

What it does

SmartPlate AI lets users choose between Before Cooking and After Cooking analysis.

Before cooking, users can enter ingredients manually or upload a photo of their ingredients. After cooking, users can upload a photo of their finished meal and optionally provide ingredients or dish weight for more context.

Google Gemini analyzes the food and provides:

Health Score
Sustainability Score
Detected foods or ingredients
SmartPlate Tips
Smart Swaps
Confidence indicators
Improved meal suggestions
Leftover Rescue ideas

Users can accept sustainable swaps to earn Eco Points.

SmartPlate also includes Hi SmartPlate, a voice assistant. Users can say “Hi SmartPlate,” ask questions about their meal, and receive a spoken Gemini-powered response.

How we built it

The frontend was built with React, TypeScript, and Vite.

The backend uses Node.js and Express, with the Gemini API kept securely on the server.

We use Google Gemini for multimodal food and ingredient image understanding, health and sustainability explanations, Smart Tips, Smart Swaps, improved meal suggestions, leftover ideas, and conversational responses.

Browser speech recognition and text-to-speech are used for the Hi SmartPlate voice experience.

LocalStorage is used for lightweight data such as the user profile, Eco Points, accepted swaps, and recent meals.

Challenges we ran into

One of the biggest challenges was that a food photo cannot always show everything inside a meal. Wraps, sandwiches, soups, and other mixed foods can hide ingredients.

Instead of pretending the AI is always certain, we added Estimated and Confirmed confidence indicators and allow users to provide ingredients for additional context.

Voice interaction was another challenge because browser speech recognition can behave differently across browsers. We kept push-to-talk and typed questions available as fallbacks.

We also had to make sure Gemini responses were handled safely so invalid or unavailable responses would not crash the application.

Accomplishments that we're proud of

We are proud that SmartPlate combines several features into one working experience:

Gemini-powered food image analysis
Before and After Cooking modes
Health and Sustainability scoring
Responsible AI confidence indicators
Smart Swaps and Eco Points
Improve My Meal comparisons
Leftover Rescue for reducing food waste
Hands-free “Hi SmartPlate” voice interaction
A responsive and animated user interface

Most importantly, Gemini is not just an added chatbot. It powers the main SmartPlate experience.

What we learned

We learned how to integrate multimodal AI into a full-stack application and how important it is to design around the limitations of AI instead of hiding them.

We also learned more about structured Gemini responses, image handling, voice interfaces, frontend/backend communication, responsive design, and responsible AI.

What's next for SmartPlate AI

With more time, SmartPlate could include user accounts, longer-term meal history, personalized sustainability goals, streaks and achievements, verified nutrition datasets, and location-aware food sustainability information.

The goal would be to turn SmartPlate from a hackathon prototype into a tool that helps people build healthier and more sustainable habits over time.