MugJai

MugJai: The AI assistant that grows with you.

Comment

About MugJai

Track: Collaborative Partner — All Things Agentic Hackathon
Tagline: An AI assistant that grows with you.

Inspiration

Setting up a Personal AI Assistant today is unnecessarily complex — Skills? Tools? Context management? Harness engineering? Yet most people simply want an easy, reliable partner to simplify daily life. That's why we built MugJai for users navigating the endless decisions that pile up when living on their own—from rent and personal finance to wellbeing.

MugJai transformed editorial content from static web guides into an interactive, in-app assistant. We believe editorial content in the AI era should not stay frozen on a page; trusted guides should answer questions, walk through real-world steps, and evolve with the user. Grounded in verified CMS data, MugJai moves beyond basic chat to guide step-by-step workflows and trigger actions directly inside the product.

This directly embodies the Collaborative Partner track: a proactive AI that retains context, learns from feedback, and grows with each user over time.

What it does

MugJai is a bilingual (zh-Hant-HK + en) Flutter app backed by Wagtail CMS and a Google ADK agent on Cloud Run.

For users:

  • Ask — natural-language questions grounded on published articles and checklists on topics such as rent, finance, and wellbeing, with citations.
  • Pick an assistant — General assistant for factual guidance; Support companion for a warmer, supportive tone and distilled social worker domain knowledge.
  • Master note — a rolling personal note with sections per assistant, so each assistant’s context stays separate.
  • Saved — the agent can save a checklist or article to your Saved tab after you confirm, for easier review later.
  • Feedback — thumbs up/down on every reply, with an optional comment on thumbs-down.
  • On-device Gemma voice (Optional) — for Support companion, Gemini runs tools in the cloud; Gemma 3 1B on your phone composes the final supportive reply locally.
  • Beyond chat: the agent mutates state — it updates master notes, writes bookmarks, and stores feedback in CMS.
Static editorial content MugJai
Search and read an article Ask in natural language
Fixed checklist page Agent cites a real checklist, shows preview steps, offers to save
No memory Persona-scoped master note persists

How we built it

Stack

Layer Technology
Mobile Flutter, Riverpod, Firebase Auth + App Check
CMS Wagtail, Django Ninja, Postgres + pgvector
Agent Google ADK, Gemini 3.5 Flash on Vertex AI
On-device AI Gemma 3 1B via MediaPipe GenAI
Cloud Cloud Run, Cloud SQL, Vertex AI, Cloud Trace, Secret Manager, Terraform

Architecture

flowchart TB
  Mobile[Flutter Ask + persona picker]
  Mobile -->|Firebase JWT + state_delta| ADK[Gemini router — ADK on Cloud Run]
  ADK --> CMS[Wagtail CMS]
  ADK --> Tools[TOOLS: search_cms load_skill checklists bookmarks master_note]
  Tools --> CMS
  ADK -->|VoiceHandoffV1 JSON| Mobile
  Mobile --> Gemma[On-device Gemma 3 1B]
  Gemma --> Mobile
  ADK --> Trace[Cloud Trace]

Design principles:

  1. CMS is the only knowledge source — Trusted editorial content, no open-web RAG / refuse if missing.
  2. Hide complexity — Users pick personas; editors manage skills and prompts in Wagtail.
  3. Cloud for trust — Tools, auth, daily caps, and grounding run on Cloud Run.
  4. Edge for supportive voice — Gemma phrases replies locally from a structured handoff; Gemini never delegates tool calling to the phone.

Grounding and search

CMS search embeds queries with Vertex AI and retrieves nearest neighbours from pgvector. The agent instruction layer (grounding.py) enforces citations, refusal copy, and beginner-friendly tone when profile ratings mark a topic as beginner-level. Editors maintain domain Agent Skills (e.g. tax-hk, rent-hk) as Wagtail snippets; Gemini loads them on demand via ADK SkillToolset.

Collaborative Partner features

Feature Implementation
AI Agent's Personas Wagtail AgentPersona + mobile picker; separate Ask thread per persona
Master note sections: { persona_slug: text } + ADK tools + Settings screen
Feedback AskFeedback model + thumbs UI → Wagtail admin
Bookmarks ADK save_bookmark → existing bookmarks API
Checklists list_checklists / get_checklist + preview cards in Ask
Observability Cloud Trace spans on cloud ADK (trace_to_cloud=True)

Split-brain voice (VoiceHandoffV1)

For Support companion with the on-device model installed:

  1. Mobile sends voice_inference: on_device with the Ask request.
  2. Gemini runs all tools (search_cms, load_skill, list_checklists, get_checklist, get_user_profile, save_bookmark, update_master_note).
  3. The router emits only a JSON handoff payload — no user-facing prose.
  4. The phone parses the SSE stream, validates the contract, and streams Gemma’s reply into the Ask bubble.

If the model is not installed, the app falls back to Gemini supportive voice on the server in a single request. If handoff parsing or on-device compose fails, we show a localized error and do not retry — avoiding a double hit on the daily Ask cap.

Challenges we ran into

Grounding vs helpfulness. Models want to answer even when CMS has no match. We tightened refusal rules, mandatory tool use, and eval tests so figures are never invented or hallucinated.

Tools that looked available but were not. CMS helper tools were initially wired inside ADK SkillToolset in a way that did not expose them to the model on every turn. We moved critical tools (including master note updates) to the top-level agent tool list so “I’ll remember that” actually calls update_master_note.

Master note evolution. A single blob did not fit multiple assistants. We migrated to persona-scoped sections with backward-compatible API payloads and synced helpers on CMS and agent sides.

On-device Gemma vs server orchestration. Server-side Gemma sub-agents added cost and deployment complexity. We replaced them with an explicit VoiceHandoffV1 contract: Gemini orchestrates in the cloud; the phone composes voice locally. Parsing raw JSON from SSE streams, normalizing streaming artifacts, and handling fallbacks without double-billing the Ask cap took several iterations.

CI and signing. TestFlight and Play builds surfaced path bugs in signing scripts, ProGuard rules, and keychain setup on GitHub Actions runners — separate from the agent, but necessary to demo on real devices.

AI agent evolves. Feedback and master notes improve the product loop for editors.

Accomplishments that we're proud of

  • A production-shaped stack — Flutter app, Wagtail CMS, ADK agent, Terraform deploy to GCP — not a notebook demo.
  • Collaborative Partner end-to-end: personas, persona-scoped master notes, separate chat threads, feedback, bookmark actions, checklist referral.
  • Gemini + Gemma split with a documented handoff schema, mobile parser, and graceful fallbacks.
  • Editorial trust model — citations, refusals, regulated disclaimers, CMS-only knowledge.
  • CMS-managed personas and skills — change tone and playbooks without redeploying the mobile app, making it very easy to maintain.
  • Observability on cloud — Cloud Trace shows tool spans for every grounded Ask turn.
  • Bilingual — zh-Hant-HK and en across content, personas, and UI.
  • TestFlight and Play builds shipping while the hackathon work landed (v0.1.1).

What we learned

  • Action beats chat in judging. Saving a checklist, updating a master note, or writing a feedback row is more convincing than a fluent paragraph alone.
  • Decouple voice from orchestration. Tool calling belongs in a capable cloud router; smaller on-device models can still add value for tone and privacy if the contract is strict.
  • Memory needs structure. One chat log and one note field are not enough when users switch between “help me budget” and “I feel overwhelmed.” Persona-scoped sections and threads match how people actually think.
  • Hide the harness. Users should pick General or Support companion, not load_skill or VoiceHandoffV1. Editors and engineers own the complexity.
  • Grounding is a product feature. Refusal when content is missing builds trust — especially for regulated HK topics.

What's next for MugJai

  • Wire the Finance assistant and more persona end-to-end (UI placeholder exists today).
  • Use Ask feedback and master-note patterns to guide editorial priorities and persona prompt tuning.
  • More wellbeing and everyday-life editorial content domains in CMS, still fully grounded.
  • Marketing: launch it and promote it.

Built for the All Things Agentic Hackathon — Collaborative Partner track.

Built With

Share this project:

Updates

Submission history