SafePlate

For millions with food allergies, eating out is stressful. SafePlate turns a menu photo into safe, personalized dish recommendations & explanations, helping people dine out confidently and safely.

Comment

SafePlate - AI-Powered Food Safety Assistant

Inspiration

Food allergies affect millions of people worldwide, yet dining out remains a stressful experience for them. Reading menu cards filled with complex dish names and hidden ingredients makes it difficult to determine what is safe to eat.

We were inspired to build SafePlate after realizing how technology — especially AI and computer vision — could simplify this decision-making process. Our goal was to create an application that empowers users to scan a menu and instantly understand which dishes are safe based on their allergen preferences.

We wanted to bridge the gap between food safety and accessible AI.

What it does

SafePlate is an AI-powered food safety assistant that analyzes restaurant menus and identifies allergen risks.

Key functionalities include:

  • Upload or scan a menu image
  • Detect all dishes using Vision AI
  • Extract ingredients for each dish
  • Allow users to input allergens (e.g., milk, almonds)
  • Classify dishes into:
    • Safe
    • Unsafe
    • Modifiable (can be customized)

Mathematically, the filtering logic can be represented as:

$$\text{Safe Dish} = \text{Dish} \cap \text{Allergens}_{\text{avoid}} = \emptyset$$

Where a dish is safe if it has no intersection with the allergen set.

How we built it

We designed SafePlate as a cloud-based, AI-driven pipeline:

1. Image Processing

Menu images are uploaded in Base64 format and sent to Vision AI for dish detection.

2. Dish Extraction

Claude Vision model extracts menu items.

3. Ingredient Analysis

Each dish is analyzed via LLM to generate ingredients.

4. Allergen Matching

User allergen input is normalized and compared.

5. Safety Classification

AI classifies dishes into safe, unsafe, or modifiable.

Tech Stack:

Python — Backend logic and AWS Lambda functions AWS Lambda — Serverless compute for processing menu images and handling API logic Amazon API Gateway — REST API endpoints for receiving image uploads and allergen preferences Amazon Bedrock — AI processing using Claude models: Claude 3.5 Sonnet (Vision) for menu image understanding and dish extraction Claude 3 Haiku for ingredient generation and allergen safety analysis Computer Vision (AI Vision Models) — Extracting structured dish data directly from menu images Boto3 (AWS SDK for Python) — Communication with AWS services JSON APIs — Structured data exchange between frontend and backend Base64 Encoding — Image transmission through HTTP POST requests Serverless Architecture — Scalable event-driven backend design REST API Integration — Communication between client app and backend services Flutter / Mobile Frontend — User interface for uploading menu images and selecting allergens

System flow:

$$\text{Menu Image} \to \text{Dish Detection} \to \text{Ingredients} \to \text{Allergen Filter} \to \text{Safety Results}$$

Challenges we ran into

Menu text complexity - Different fonts, layouts, and image quality affected dish detection.

Ingredient ambiguity - Some dishes don't explicitly list allergens (e.g., baked goods containing milk).

Latency & timeouts - Multiple AI calls per dish caused performance bottlenecks.

Prompt engineering - Ensuring models returned strict JSON required iterative tuning.

Error handling - Parsing model outputs with markdown wrappers added complexity.

Accomplishments that we're proud of

  • Built an end-to-end AI pipeline from image to safety classification
  • Successfully integrated Vision + Text AI models
  • Enabled real-time allergen filtering
  • Designed modifiable dish suggestions
  • Optimized calls by combining ingredient + safety analysis

Most importantly, we created a solution with real social impact for people with dietary restrictions.

What we learned

Through SafePlate, we gained hands-on experience in:

  • Vision AI integration
  • Prompt engineering techniques
  • Serverless architecture (AWS Lambda)
  • API design & testing
  • JSON parsing from LLM outputs
  • Performance optimization for AI workflows

We also learned that combining multiple AI capabilities creates exponentially more useful applications.

What's next for SafePlate

We plan to expand SafePlate with:

  • Dietary groups (Vegan, Gluten-Free, Halal)
  • Cross-contamination risk detection
  • Nutrition analysis per dish
  • Mobile app integration
  • Real-time restaurant menu APIs
  • Multilingual menu support

Future optimization goal:

Latency_total = Σ(Call_i) → Minimized (for i=1 to n)

By batching AI calls and caching ingredient data.

Built With

Share this project:

Updates