JomMakan

JomMakan is a simple Streamlit app that lets users upload a food photo, then usesGoogle Gemini Vision to generate a healthy score and improvement tips.

Comment

Inspiration

Malaysia's food culture is rich, vibrant, and deeply social — "Jom Makan" literally means "Let's eat!" in Malay. But while we love our nasi lemak, char kway teow, and mamak roti canai, it can be hard to know how nutritious our meals really are. We were inspired to build a tool that makes nutrition awareness effortless: just snap a photo and get instant, AI-powered feedback — no calorie counting apps, no manual food logging.

What it does

JomMakan lets you upload a photo of any meal and instantly receive:

  • 🏅 A Health Score (0–100) with a letter grade (A+ to D)
  • 📊 Macro breakdowns — Fiber, Sugar, and Protein levels visualised as animated progress bars
  • 🔥 Estimated calorie count, grounded with real data from the USDA FoodData Central database
  • 🍜 Detected food items displayed as chips
  • ⚡ Personalised healthy tips to improve the nutritional value of your meal
  • 🎉 A confetti burst for excellent scores (80+)

How we built it

  • Frontend & App Framework: Streamlit with a fully custom mobile-first CSS UI (glassmorphism header, animated SVG gauge, shimmer skeleton loader, confetti)
  • AI Vision Model: Google Gemini 2.5 Flash (gemini-2.5-flash) via the google-genai SDK — used both for food image analysis and for calorie-refined re-scoring
  • Calorie Grounding: USDA FoodData Central API to look up per-food calorie data, which is fed back into Gemini for a refined analysis pass
  • Containerisation: Docker with a minimal Python 3.12-slim image
  • Deployment: Google Cloud Run (Asia Southeast 1), with secrets managed via Google Secret Manager
  • CI/CD: GitHub Actions with Workload Identity Federation (keyless OIDC authentication) for zero-credential automated deploys

Challenges we ran into

  • Reliable JSON extraction from Gemini: The model occasionally returned markdown-wrapped or partially malformed JSON. We built a robust extraction and normalisation pipeline (extract_json + normalize_result) to handle all edge cases gracefully.
  • Two-pass calorie refinement: Orchestrating the flow of image analysis → USDA lookup → Gemini re-score required careful state management and fallback logic so a failed second pass never breaks the first result.
  • Mobile-first Streamlit UI: Streamlit is built for data dashboards, not polished consumer apps. Achieving a native-app feel (sticky header, full-width cards, animated gauges, shimmer skeletons) required extensive CSS injection and careful layout overrides.
  • Keyless Cloud Run deployment: Setting up Workload Identity Federation between GitHub Actions and Google Cloud for the first time had a steep learning curve, but results in a fully secure, key-free CI/CD pipeline.

Accomplishments that we're proud of

  • A production-ready, publicly accessible app deployed on Google Cloud Run at jommakan-171023813626.asia-southeast1.run.app
  • A genuinely beautiful, mobile-first UI built entirely within Streamlit's constraints — animated SVG donut gauge, shimmer loading skeleton, confetti celebration, and smooth card entrance animations
  • A two-stage AI pipeline that combines Gemini's vision capabilities with real-world USDA nutrition data for more accurate calorie estimates
  • A fully automated CI/CD pipeline using GitHub Actions and Google Cloud Workload Identity Federation — no long-lived service account keys anywhere

What we learned

  • How to use Gemini's multimodal vision API to extract structured nutritional data from food images
  • How to build a multi-step LLM pipeline that enriches AI output with external API data (USDA) and feeds it back for a refined second pass
  • Practical techniques for customising Streamlit's UI far beyond its default components using injected CSS and raw HTML
  • How to set up keyless, OIDC-based authentication between GitHub Actions and Google Cloud Run for secure automated deployments

Built With

Share this project:

Updates