Swaad: AI Powered Food Recommendation System

Flowchart 1
Flowchart 2
Flowchart 3
Flowchart 4

Inspiration

Diners often face a dilemma: how to choose from vast menus while ensuring a delightful experience aligned with their personal taste. Traditional recommendations fall short, ignoring individual flavor nuances.

We've all been there, walking into a new restaurant with absolutely no idea what to order. More often than not, ordering something random leads to a disappointing experience.

This inspired our team to create Swaad, an AI-powered recommendation system that recommends what you want to eat based on what you like.

What It Does

Swaad introduces a dual-axis recommendation engine combining:

Semantic Search: Understanding the meaning behind user queries, not just keywords
6-Dimensional Taste Profiling: Mapping user preferences across sweet, salty, sour, bitter, umami, and spicy dimensions
AI-Powered Personalization: Learning from favorite dishes to predict perfect matches
Intelligent Filtering: Diet-aware, allergen-safe, location-smart recommendations

How We Built It

Query Reception & Parsing: The React frontend captures user input through a conversational chat interface and sends it to the FastAPI backend (/api/chat endpoint). The system extracts key components including location, dietary preferences, cuisine type, and intent from natural language queries using regex patterns and Groq AI.
Intent Detection & Classification: Using Groq's Large Language Model (llama-3.3-70b-versatile), the system classifies the query intent including greetings, dish-specific queries ("where can I get paneer pizza?"), restaurant menu queries ("show me menu for XYZ"), or general recommendations. The system also identifies dietary restrictions (veg/non-veg/mix) and allergens.
Semantic Embedding Generation: The user query is converted into a 384-dimensional semantic vector using the SentenceTransformer model (all-MiniLM-L6-v2). This vector representation captures the semantic meaning of the query beyond simple keyword matching.
User Profile Loading: The system retrieves the user's stored preferences from Pinecone database, including their favorite dishes, dietary restrictions, allergen list, and previously calculated taste profile vector.
Restaurant Data Retrieval: The system retrieves live restaurant and menu data using the Yelp AI API. Retrieved data is processed and stored in Pinecone vector database with pre-computed menu embeddings and taste vectors for efficient semantic search.
Pinecone DB Population: For each candidate restaurant or dish, the system calculates a 6-dimensional taste vector [sweet, salty, sour, bitter, umami, spicy] using our curated database which has taste vectors for ~231K dishes. This curated database was created using raw recipes from Food.com and ~1,000 ingredient flavor vectors generated by Claude Sonnet 4.5. The system compares these taste vectors against the user's learned taste profile using cosine similarity.
Vector Similarity Search: The similarity score of the generated query embedding and dish embedding is computed using cosine similarity. This semantic search (top_k=10-20) identifies restaurants and dishes that match the user's query meaning, not just keywords.
Personalization Layer: The system applies hard filters (dietary restrictions, allergens using Groq AI detection, location matching) and soft ranking adjustments based on the user's favorite dishes and taste preferences.
Top Dish Recommendations: For each recommended restaurant, the system ranks all menu items by taste similarity to the user's profile and returns the top 5 dishes most compatible with their flavor preferences.
Response Generation: Using Groq LLM, the system generates natural language responses that explain the recommendations in context of the user's dietary preferences, highlighting key features and taste compatibility.
Result Presentation: The frontend displays personalized recommendations as rich restaurant cards with ratings, cuisine types, price range, top 5 dish suggestions ranked by taste compatibility, enabling users to make informed dining decisions.

Challenges We Ran Into

Limited flavor profile data: Data was not easily available, especially for flavor profiles of ingredients. More research needs to be done in this area.
Image recognition complexity: Implementing the image reading functionality was particularly challenging, as standard methods like OCR were not very effective. This led us to adopt multimodal LLMs for better accuracy.
Design and architecture decisions: Determining how the whole project would work required extensive brainstorming and trial-and-error experimentation.
Recipe name inconsistencies: Matching user input to recipes was challenging due to naming variations like "Chicken Tikka" vs "tikka chicken" vs "Chicken Tikka Masala". We implemented fuzzy matching to maximize coverage.
Performance bottlenecks: We encountered significant latency issues during development. Similarity matching and filtering operations were taking too much time, which led us to adopt a vector database solution like Pinecone for faster queries.

Accomplishments That We're Proud Of

Functional and beautiful: The product works accurately and looks visually appealing, a win for both functionality and user experience.
Vision realized: Midway through development, we weren't sure if our team's vision would materialize, but we pushed through and brought it to life!
Innovative taste profiling: We successfully implemented a 6-dimensional taste profiling system that captures the nuances of individual flavor preferences, something that doesn't exist in traditional recommendation systems.
Seamless AI integration: We combined multiple cutting-edge technologies: semantic search, multimodal LLMs, Classic LLM's, vector databases, and fuzzy matching: into a cohesive, performant system.
Allergy-aware recommendations: People with dietary restrictions shouldn't have to compromise on taste or risk their health, our system ensures they can explore menus with confidence.
Personal impact: Now we can explore all the food we want to eat with confidence, knowing we'll love what we order.

What We Learned

Building novel datasets from scratch: When flavor profile data didn't exist, we learned to create our own by leveraging Claude Sonnet 4.5 to generate ingredient flavor vectors and combining them with Food.com's recipe database. This taught us that sometimes innovation means building the foundational data infrastructure yourself.
The power of multimodal AI over traditional approaches: Traditional OCR completely failed for menu image recognition, but multimodal LLMs succeeded beautifully. We learned to identify when cutting-edge AI solutions are necessary versus when simpler methods suffice.
Performance optimization is non-negotiable: Our initial similarity matching took far too long to be usable. Migrating to Pinecone vector database taught us that user experience depends on architectural choices made early, scalability can't be an afterthought.
Real-world data is messy: Recipe names vary wildly ("Chicken Tikka" vs "tikka chicken" vs "Chicken Tikka Masala"). Implementing fuzzy matching taught us to design systems that embrace human inconsistency rather than fighting it.
Interdisciplinary thinking matters: Combining food science (6D taste profiling), AI/ML (semantic search, embeddings), and UX design required us to think beyond pure coding. The best solutions came from understanding the domain deeply.

What's Next for Swaad: AI Powered Food Recommendation System

Expand the flavor database: Collaborate with food scientists and nutritionists to increase our ingredient flavor vectors from 1,000 to 10,000+, covering more cuisines and regional variations for even more accurate taste profiling.
Mobile-first experience: Develop native iOS and Android apps with camera-based menu scanning, voice queries, and GPS-based restaurant discovery for seamless on-the-go recommendations.
Dish customization recommendations: Use AI to suggest modifications to dishes ("make it less spicy," "add extra umami," "substitute X for Y") based on user preferences and what restaurants typically accommodate.
Nutritional intelligence: Integrate macro/micronutrient tracking to help users meet health goals (high-protein, low-carb, etc.) while still enjoying foods that match their taste preferences.