DietaryVision AI

An offline-first, multimodal health assistant powered by Gemma 4, protecting travelers with severe allergies via AI menu triage and instant translation.

Comment

💡 Inspiration

For the 400+ million people worldwide living with severe food allergies, celiac disease, diabetes, or hypertension, traveling internationally isn't just difficult—it can be a life-threatening medical hazard. A single mistranslation on a foreign menu can lead to a hospital visit. Travelers are forced to rely on rudimentary translation apps that don't understand complex clinical dietary restrictions, cross-contamination risks, or hidden culinary ingredients. We wanted to build a true "pocket medical triage node" to give these travelers their freedom back.

⚙️ What it does

DietaryVision AI is a decentralized, hybrid-edge health safety assistant powered by Gemma 4. It protects travelers by serving as a localized medical triage node with four core features:

  • Multimodal Menu Triage: Users upload photos of foreign menus. The AI exhaustively analyzes every dish, cross-referencing ingredients against the user's localized "Health Passport" to categorize items into strictly SAFE, MODIFY, or AVOID.
  • Automated Medical OCR: Users snap a photo of their doctor's notes or lab results. The system automatically parses the clinical data and populates the user's Health Passport.
  • Meal Diary & Tracker: Users photograph their actual plates before eating. The AI identifies the food, estimates sodium and caloric load, and issues a hard medical warning if it detects a passport violation.
  • Multilingual Emergency Cards: The app translates the user's specific clinical restrictions into massive, highly visible wallet cards in 10+ local languages to show restaurant staff.

🛠️ How we built it

NutriTrip Architecture Diagram We engineered DietaryVision AI specifically for low-bandwidth, remote travel environments:

  • The Brain (Hybrid AI): To bypass the physical RAM limitations of standard traveler laptops, we utilized an Ollama daemon to intelligently proxy heavy multimodal inferences to the Gemma 4 31B Cloud model, with Google Gemini acting as a robust fallback for complex vision tasks.
  • The Backend: A lightning-fast FastAPI Python server handling the routing, prompt construction, and AI orchestration.
  • Data Privacy: All user accounts, medical profiles, and "Hacker Terminal" system logs are stored in a localized SQLite/PostgreSQL database, guaranteeing zero third-party transmission of highly sensitive health records.
  • The Frontend: We built a zero-dependency Single Page Application (SPA) using Vanilla JavaScript, HTML, and custom CSS to ensure the app is incredibly lightweight.

🚧 Challenges we ran into

  1. Network Latency with Massive Images: Modern smartphone photos are 5MB to 10MB. Sending these over spotty hotel Wi-Fi crashed the browser. We solved this by building a Custom Client-Side Image Compressor using the HTML5 Canvas API, instantly shrinking photos to lightweight 200kb payloads before they touch the network.
  2. LLM Output Truncation: Initially, asking the AI to analyze massive menus caused it to run out of output tokens and skip dishes. We heavily optimized the JSON schema and prompt engineering to force exhaustive, streamlined generation.

🏆 Accomplishments that we're proud of

  • Engineering a seamless hybrid Edge-to-Cloud proxy architecture that delivers heavy 31B parameter reasoning on constrained devices.
  • Building a gorgeous, startup-ready dark mode UI entirely from scratch with vanilla CSS Grid and Flexbox.
  • Creating a tool that solves a literal life-or-death problem for marginalized travelers.

🧠 What we learned

We gained deep insights into prompt engineering for strict JSON schema adherence in open-weights models, handling multimodal vision pipelines, and optimizing client-side performance for real-time AI applications.

🚀 What's next for DietaryVision AI

We plan to introduce local on-device Llama.cpp integration for 100% offline text inference, expand the nutritional database for obscure regional cuisines, and package the frontend into a native offline mobile app using React Native.

Built With

Share this project:

Updates