Kerly

Kerly helps you pick your next lunch or dinner nearby by matching real restaurant menus to the latest U.S. nutrition pyramid and your preferences.

Comment

Inspiration

Nutrition advice is everywhere. But when you’re hungry, you don’t want a lecture. You want a decision.

The latest US nutrition guidance is clear. Balance matters. Quality matters. Personalization matters. But in the real world, that guidance rarely turns into action. Not because people don’t care. Because the moment is messy. Time is short. Options are overwhelming.

Kerly starts with one simple truth: health outcomes are driven by what you eat next. So we built Kerly to make that next choice easier. Kerly translates official guidance into an immediate, practical recommendation based on your preferences, constraints, and what’s nearby.

Less thinking. More doing. Better meals, one decision at a time.

What it does

Kerly helps users decide their next meal by combining the latest US nutritional guidelines with personal preferences. It evaluates meals against the modern food pyramid, calorie and macro needs, and user constraints, then recommends options that are both nutritionally sound and realistically appealing. Instead of generic “eat healthier” advice, Kerly recommends specific dishes from places nearby.

How we built it

The system is built as an AI-first recommendation architecture with Gemini at the core.

  • Frontend: Built in AI Studio as the client-facing experience.
  • Backend: Python-based Litestar REST API deployed on Render.
  • Core intelligence: Multiple backend agents powered by Google Gemini (google-genai), with Gemini driving:
    • Menu extraction and enrichment pipelines
    • Nutrition and macro resolution
    • Final meal recommendation reasoning from ranked candidates combined with user context

Gemini 3 usage

Kerly uses Gemini 3 as the core intelligence layer across the full recommendation pipeline. We use Gemini 3 for:

  • menu extraction/enrichment from messy restaurant sources
  • nutrition and macro resolution
  • recommendation reasoning over ranked candidates plus user constraints.

We run Gemini in multi-turn, stateful interactions with structured JSON outputs, grounding tools, retries, and guardrails so results are reliable enough for real user decisions.

  • Data layer: PostgreSQL as the system of record, using SQLAlchemy / Advanced Alchemy.
  • Async processing: Celery with Redis for background jobs such as ingestion and processing pipelines.
  • External integrations: Yelp as an initial venue and menu data source, followed by Gemini-based enrichment and recommendation.
  • Serving pattern: The frontend calls backend REST endpoints; the backend applies retrieval then uses Gemini for higher-level decisioning to return the final recommendation.
  • Backend admin interface: We also built an internal admin dashboard to manage restaurants and menu data, monitor ingestion quality, and quickly debug or correct recommendations during iteration.

In short, the platform consists of a Render-hosted backend and an AI Studio frontend, with Gemini acting as the primary intelligence engine across the entire meal recommendation pipeline.

Challenges we ran into

  • Getting high-quality menu data: We initially tried the Yelp API, but the menu coverage and structure weren’t reliable enough for accurate nutrition analysis. Many restaurants lacked detailed menus or had inconsistent and incomplete data, which forced us to explore alternative ingestion paths such as menu URLs, PDFs, images, and scraping-friendly sources.

  • Tuning agent parameters: Achieving consistent agent behavior required running many experiments. Small changes in prompts, thresholds, or tool usage often led to large differences in output quality, making iteration speed and experiment management a significant challenge.

  • Building trustworthy evals: We needed evaluation at multiple layers:

    • Menu ingestion validity: verifying that menu URLs actually worked, loaded correctly, and weren’t broken or blocked.
    • Extraction accuracy: ensuring menu items were correctly extracted and not hallucinated.
    • Nutrition accuracy: calorie and macro calculations ultimately require expert-grounded evaluation, since approximate results can still be misleading in a health-focused system.

Accomplishments that we're proud of

  • Built a real end-to-end AI product (not just a demo): We shipped a working full-stack experience, a deployed backend plus a user-facing frontend in Google AI Studio so the project is usable, not just a concept.

  • Turned the new nutrition guidance into a practical decision engine: We translated the latest US nutrition recommendations into computable rules and a “next meal” recommender that produces actionable outputs.

  • Production-style Gemini integration: We used Gemini the way you’d use it in a real system, multi-turn Interactions, structured outputs (schema-controlled JSON), tool grounding (google_search, url_context), plus retries and guardrails for reliability.

  • Hybrid intelligence that stays controllable: We combined deterministic scoring/filtering with Gemini reasoning, backed by a solid PostgreSQL data layer and async pipelines (Celery + Redis), so recommendations are both smart and predictable.

Validation, Safety, and Real-World Performance

Because Kerly influences real food choices, we treat reliability and safety as first-class product requirements, not nice-to-haves. We evaluate each stage of the pipeline and gate releases against measurable checks:

  • Constraint safety: Recommendations must satisfy hard dietary/allergen constraints before ranking.
  • Extraction trust: Menu URL reachability, extraction precision, and hallucination checks are tracked per source.
  • Nutrition quality: Calorie/macro estimates are compared against reference nutrition data and flagged when confidence is low.
  • Recommendation quality: We measure top-1/top-3 acceptance in pilot testing for “Would you actually order this?”
  • Operational readiness: We monitor latency, retry rate, failure rate, and cost per recommendation to keep the system usable and scalable.

This gives us a practical standard for shipping: Kerly is only “good” if it is nutritionally aligned, constraint-safe, and fast enough to be useful in the moment someone is deciding what to eat.

What we learned

  • How to build agentic systems (not just prompts): We learned to design a reliable multi-step agent pipeline, menu URL resolution → menu extraction → macro resolution → final meal recommendation where each step has clear contracts, validation, and recovery paths.

  • Gemini Interactions API is the right backbone for workflows: For multi-step agents, we found Gemini’s Interactions API to be a better core than the older generateContent-style approach because it simplifies stateful, turn-based execution and makes retries much more robust.

    • We run stateful turns with client.interactions.create
    • We pass previous_interaction_id to continue context across retries/steps

  • Structured outputs make LLMs production-usable: Using JSON Schema / Pydantic-backed response schemas and JSON mime types dramatically improved the consistency of machine-parseable outputs.

  • Tools + grounding reduce hallucinations: We learned to ground the agents by combining tools like URL context (read the actual menu URL) and web search grounding when needed (e.g., macro lookups), which improved accuracy and reliability.

  • Reliability requires LLMOps-style patterns: Parameter tuning (temperature/top_p/token limits/thinking settings), retries with corrective messages, fallback parsing, and timeout handling were essential to make the system stable under real-world menu data.

What's next for Kerly

Product priorities:

  • Expand coverage: Right now we have strong San Francisco data focused around Union Square. Next we’ll scale ingestion to more neighborhoods and then expand to other cities using the same pipeline.
  • Constraint-first engine: Add strict pre-checks for diets (seafood-only, etc.) so Gemini only ranks valid candidates.
  • Feedback loop + personalization: Add 👍/👎, “not this again,” “too far/too expensive,” then use it for a personalized reranker (history, cuisine preference, diversity memory).

Technical priorities:

  • Offline Gemini tagging: Pre-label menu items with dietary tags, allergen risk, etc; store in Postgres and filter before recommendation.
  • LLMOps + experimentation: Track adherence/retries/URL failures/latency/cost; add alerts.
  • Cost/latency controls: Cache repeated candidate sets and route simple requests to cheaper/faster model configs.
Share this project:

Updates