Nexus AI — Multi-Agent Intelligence Hub

9 specialist agents · 23 skills · 4-layer memory · Google ADK — a PT student's always-on multimodal AI, built for real clinical study.

Comment

Inspiration

Physical therapy school is academically brutal.

Every week I'm expected to read 10–20 peer-reviewed papers, organize clinical notes by anatomical subject, memorize hundreds of therapeutic techniques, and prepare for board exams — all while doing clinical placements. I tried using ChatGPT, Notion, Zotero, and five other tools. None of them talk to each other. None of them know what "旋轉肌群" means in a clinical context.

So I built Nexus — not as a hackathon project, but as the tool I actually needed. By the time this hackathon started, it had already been running on my machine for weeks. The hackathon gave me a reason to make it production-grade.

What it does

Nexus AI is a 24/7 multi-agent AI assistant that coordinates 9 specialist agents and 23 built-in skills across Web UI, Telegram bot, REST API, and a Google ADK standalone agent — all from a single FastAPI server on Google Cloud Run.

You talk to it in plain Chinese or English. No commands, no prefixes. The orchestrator figures out what you need.

Real examples from my daily use:

「幫我找前十字韌帶復健的相關論文」 → Searches PubMed with automatic MeSH term expansion ("前十字韌帶" → "Anterior Cruciate Ligament[MeSH]"), returns real PMIDs with authors, journals, and links 「把這些論文存進我的骨科筆記」 → Saves to a SQLite study notes database, organized by PT subject category 「筆記 考試 解剖學」 → Generates quiz questions from your saved anatomy notes using Gemini Send a knee anatomy diagram photo on Telegram → Vision agent identifies structures in Traditional Chinese with confidence score 「每天早上6點 生成晨報」 → Sets a recurring schedule; morning summary delivered via Telegram push every day at 6AM

The 9 agents:

  • Reasoning — Chain-of-thought with 3-step self-verification
  • Research — Real-time web search + synthesis
  • Coder — Code generation, debugging, sandboxed execution
  • Knowledge — 4-layer adaptive memory retrieval
  • Shell — Sandboxed command execution with allowlist
  • Vision — Multimodal image analysis (Gemini 2.0 Flash)
  • Web — Live URL fetch and content extraction
  • File — Secure local file operations
  • Optimizer — System self-monitoring and budget control

Google ADK Agent (adk_agent/): A standalone ADK-powered agent with web search, PubMed literature search, clinical calculator, and weather tools. Run locally with adk web adk_agent.

How we built it

Architecture:

  • FastAPI backend with full WebSocket streaming for real-time agent responses
  • Three-stream output — simultaneous reasoning / thinking / final answer streams displayed in the UI's "Analysis Log" panel
  • Three-layer NLP routing: Layer 1: Trigger keywords (instant, zero LLM cost) Layer 2: Regex intent patterns (handles 80%+ of real requests) Layer 3: LLM semantic fallback (only when needed)
  • 4-Layer Adaptive Memory: Working Memory — 7 attention slots, LRU eviction Episodic Memory — SQLite with FTS5 full-text search Semantic Memory — ChromaDB vector store + NetworkX knowledge graph Procedural Cache — response dedup with 1-hour TTL
  • Agent Conference — multiple agents debate in parallel rounds with consensus detection for higher-quality answers on hard problems
  • Auto-Scheduler — natural language cron scheduling, fires via Telegram push
  • Token Budget Controller — hard daily limit, atomic JSON state, per-request cap, warning callback at 80%
  • Google ADK Integration — adk_agent/ with root_agent exposing 4 tools: search_web, search_medical_papers, compute, get_weather

Tech Stack: Python 3.11, FastAPI, asyncio, SQLite FTS5, ChromaDB, LiteLLM, Google Gemini 2.0 Flash (Google GenAI SDK), Google ADK (google-adk), python-telegram-bot, D3.js v7, Google Cloud Run, Google Cloud Build

Challenges we ran into

Async orchestration without deadlocks — Coordinating 9 agents in parallel meant any slow agent could block the pipeline. Solved with asyncio.gather(), per-agent 25-second hard timeouts, and graceful degradation that still returns a partial answer.

Memory search speed — The 4-layer memory needs to complete in under 2 seconds to feel responsive. Solved by running working memory (hash-keyed cache) first and only escalating to vector search when the cache misses.

Skill false-positive triggers — With 23 skills, the wrong skill would sometimes activate on ambiguous input. Solved with word-boundary regex for ASCII triggers, separate CJK character counting for Chinese text (Chinese doesn't have spaces), and a scoring system that counts all matching intent patterns instead of stopping at the first match.

Security surface — Supporting shell execution, URL fetching, and file operations in an AI system is inherently risky. Implemented: SSRF protection (block private IPs, cloud metadata endpoints), shell command allowlist + dangerous argument pattern detection, filesystem sandbox, path traversal rejection in PDF reader, LIKE injection escaping in SQLite queries, and timing-safe API key verification with secrets.compare_digest().

Natural language time parsing — The scheduler had to understand both 「早上9點半」 (Chinese) and 9:30 AM (English) and 09:30 (24-hour) — all in one regex chain without a date library.

Accomplishments that we're proud of

  • 9 specialized agents with sub-30s response guarantee, even on complex multi-step tasks
    • 23 fully functional skills with graceful degradation
    • Physical therapy domain knowledge baked in: MeSH vocabulary expansion for 30+ PT terms, PT subject taxonomy (anatomy, orthopedics, neurology, cardiopulmonary...)
    • Agent Conference: multi-agent debate reaching consensus in 2–3 rounds
    • Google ADK integration — standalone ADK agent (adk web adk_agent) qualifying for the Live Agents category
    • Production security: SSRF protection, shell allowlists, path traversal blocking, rate limiter, budget controller
    • Deployed 24/7 on Google Cloud Run with Telegram push notifications running autonomously

What we learned

Building a multi-agent system revealed that the hardest problem isn't the AI — it's orchestration.

Specifically: managing timeouts across parallel agents, maintaining memory coherence when multiple agents read and write the same session context, and preventing one slow external API (PubMed, Yahoo Finance, etc.) from cascading into a full system timeout required far more careful async design than expected.

We also learned that NLP routing for Chinese text is fundamentally different from English. Chinese has no word boundaries, so split() returns 1 word for 「幫我找前十字韌帶復健的相關論文」 — a full sentence. This broke our specialist detection heuristic until we counted CJK characters as individual "words" for routing decisions.

The biggest surprise: once the routing and memory are solid, the LLM becomes less important. The majority of requests in daily use hit deterministic skills (weather, calculator, academic search, notes) without ever calling an LLM. The intelligence of the system is more in the orchestration than in any single model call.

What's next for Nexus AI — Multi-Agent Intelligence Hub

  • Voice interface — Gemini's real-time audio streaming for hands-free clinical note-taking during patient sessions
    • Exam mode — structured 3-hour PT board exam simulation with the study notes system generating questions, tracking correct/incorrect, and targeting weak areas
    • Collaborative memory — share annotated PT notes with classmates through a shared ChromaDB index
    • Mobile app — native iOS/Android with the Telegram channel as the backend, adding push notifications for schedule reminders
    • Reinforcement learning routing — use session feedback (thumbs up/down) to fine-tune which agent handles which query type over time

Built With

  • asyncio
  • chromadb
  • d3.js
  • fastapi
  • gemini-2.0-flash
  • google-adk
  • google-cloud-build
  • google-cloud-run
  • google-genai-sdk
  • litellm
  • open-meteo
  • pubmed-api
  • python-telegram-bot
  • sqlite
  • websocket
Share this project:

Updates

Submission history