Chat Interface
Knowledge graph showing all domains and Relationships
Knowledge graph showing Cardiology Domain and Relationships
Knowledge graph showing all agents and Relationships

Inspiration

Cloud-hosted LLMs require API fees, stable fiber, and sustained electricity — three things that are unreliable or unaffordable in many African clinics. MAKO, my final-year engineering project, is a knowledge-graph-based multi-agent orchestrator: agents, their capabilities, the tools they own, and the handoffs between them all live in an RDF/Turtle knowledge graph, and the execution sequence is inferred from the graph at runtime rather than hard-coded. Its most advanced routing mode, v3-langraph, is genuinely autonomous — a ReAct-style LLM orchestrator decides the next hop one step at a time, querying the graph for valid transitions and validating each decision before committing to it. Demoed on a 71-agent healthcare knowledge graph (Cardiology, Emergency Medicine, Oncology, Neurology, Orthopedics, Lab/Pathology), every one of those hop decisions currently needs a cloud LLM call. HestiaHealth is what happens when that call becomes local.

What it does

HestiaHealth runs MAKO's hop-by-hop clinical routing decisions entirely offline. Given a patient case and the set of specialist agents available at that point in the graph, a local Qwen2.5-3B-Instruct model (GGUF, Q4_K_M quantization) — running through llama.cpp, no GPU, no internet — decides which agent should act next and justifies the decision, the same way MAKO's orchestrator would query a cloud LLM to do today. It runs comfortably within an 8GB RAM / integrated-graphics-only laptop profile.

How we built it

Started from MAKO's existing v3-langraph orchestrator (a LangGraph StateGraph governing one hop decision at a time, with an action-masking gate so a commit is only accepted immediately after a validate_decision success for that exact agent).

Selected Qwen2.5-3B-Instruct as the base model — small enough to leave real headroom under the 7GB memory budget, strong enough at structured, constrained output to reliably produce a parseable "next agent plus justification" decision.

Quantized to GGUF Q4_K_M (about 2GB) — the standard balance point between output quality and memory footprint.

Wired the model into llama.cpp (the only runtime ADTC's evaluation framework accepts — no Ollama, no llama-cpp-python server wrapper for the graded path), then actually ran MAKO's real, unmodified orchestrator against it — not a mock, not a hand-written prompt imitating it. Started llama-server on the submitted .gguf file, pointed MAKO's own KG_LLM_PROVIDER=openai plus KG_BASE_URL config at it, and ran "python -m orchestrator.run" with a real chest-pain case. It completed in about 6 seconds, fully offline: picked an entry agent from the knowledge graph, made a real hop decision validated against a graph edge (delegatesTo, accepted on its 3rd validate_decision attempt after two rejected proposals), and voluntarily terminated when it judged the case needed more input. The full unedited log and trace JSON are in the repo under mako-integration/.

Benchmarked the raw model itself with the official adtc-profiler tool, which drives llama-bench directly (this is separate from the orchestrator run above, and is what ADTC's own scoring actually measures): 30.33 tok/s generation, 3.84GB peak RSS, no thermal throttling, schema-valid submission report. Running the orchestrator and webui alongside the model adds some Python/LangGraph/FastAPI overhead on top of this baseline; we didn't publish a precise combined figure since ad-hoc memory sampling of a running llama-server process turned out to be unreliable on macOS (llama.cpp mmaps the weights, and OS-level RSS accounting for those pages doesn't match the profiler's own methodology).

Challenges we ran into

ADTC's evaluation framework mandates llama.cpp plus GGUF exclusively — no Ollama, despite MAKO's existing provider abstraction supporting it as a first-class backend. Had to re-validate the whole local-inference path against llama.cpp specifically.

A truncated model download (1.79GB instead of the expected 2.10GB) initially produced a "tensor data not within file bounds" error from llama.cpp — a silent curl-to-tail pipeline had swallowed the real exit code. Caught by comparing the downloaded file size against the Content-Length header before trusting the file.

Sizing the model against the 7GB RAM budget while keeping enough capability for structured decision-making meant explicitly rejecting both a 7B-class model (too much RAM pressure once KV cache and orchestrator overhead are added) and a sub-1.5B model (unreliable at holding the required output structure).

What we learned

The actual graded artifact in this challenge is narrower than the multi-agent system around it — ADTC's profiler benchmarks the raw GGUF model directly via llama-bench, not custom application code. But that doesn't mean the application layer doesn't matter — running MAKO's real orchestrator against the model, not just prompts imitating it, is what actually proves the two systems fit together.

What's next

Expanding past the healthcare domain to MAKO's other knowledge graphs (legal, and eventually the full multi-domain graph), and moving from the Q4_K_M single-model setup to a routing-aware benchmark across several small local models to see which is most reliable specifically on hop-decision-shaped prompts.