ZafyaLM: Offline Medical AI on 8GB Laptop with No GPU

Inspiration

Healthcare access across Africa is constrained less by clinical knowledge than by infrastructure: unreliable connectivity, intermittent power, and the cost of sustained cloud API usage. A nurse in a rural primary health center, a community health worker doing door-to-door outreach, or a patient trying to understand their own diagnosis often has no realistic path to a cloud-hosted medical AI assistant — not because the technology doesn't exist, but because the network and the electricity bill don't cooperate.

We wanted to build something that works the same whether the clinic has fiber, a patchy 2G signal, or no connection at all — on hardware people already own, not hardware they'd need to buy.

What it does

ZafyaLM is a fully offline, on-device medical Q&A and patient-education assistant that runs entirely on an 8GB laptop with no discrete GPU — the exact machine sitting in African clinics, classrooms, and homes today.

It speaks in two registers depending on who's asking:

• Clinicians get concise, differential-oriented, decision-support language — a quick second opinion during a busy clinic day with limited specialist access.
• Patients get plain-language explanations with practical next steps and adherence guidance (e.g. why to finish a full course of malaria medication even after symptoms improve).

Zero cloud dependency means zero per-query cost and zero connectivity requirement after the one-time model download.

How we built it

We built on Qwen2.5-1.5B-Instruct, quantized to GGUF Q4_K_M and served through llama.cpp — the only runtime accepted by the ADTC evaluation pipeline.

Model selection was evidence-based, not assumed. We benchmarked Qwen2.5-1.5B-Instruct head-to-head against Gemma-2-2B-it, both at Q4_K_M, on real representative hardware (a GitHub Actions ubuntu-latest runner — 4 vCPU x86-64, much closer to the ADTC reference laptop than our own dev machines). We measured:

• Throughput: Qwen 33.61 t/s decode vs Gemma 17.53 t/s
• Peak RAM: Qwen 2.65GB vs Gemma 3.30GB (against the 7GB budget)
• Output completeness: given the same 100-token budget, Qwen finished its answer; Gemma was truncated mid-sentence in two independent runs

Qwen won on every scoring dimension in the ADTC formula (Sacc/Sperf/Seff), so that's what we shipped.

We set up a reproducible GitHub Actions CI workflow (.github/workflows/benchmark.yml) that re-runs this exact comparison on demand, so the benchmark numbers in our report aren't a one-off — anyone can re-verify them.

Challenges we ran into

• Our first development machine (an older 2-physical-core Intel Mac) gave actively misleading benchmark numbers — tg128 throughput got WORSE going from 2 to 4 threads, a red flag that its BLAS backend was adding overhead rather than helping. We caught this by comparing against real x86 hardware via GitHub Actions rather than trusting the first numbers we saw.
• llama.cpp's prebuilt release tarballs don't include every example binary (llama-simple wasn't there, only llama-cli/llama-bench/ llama-server) — had to adjust our quality-check tooling to use llama-cli's non-interactive single-turn mode instead.
• HuggingFace's CDN gave intermittent HTTP/2 stream resets on large GGUF downloads; fixed by forcing HTTP/1.1 with resume support (curl --http1.1 -C -), which is now baked into download_model.sh so evaluators don't hit the same issue.

Accomplishments that we're proud of

• A model choice backed by real, reproducible benchmark data — not vibes — including catching our own unrepresentative hardware before it skewed the decision.
• A submission that runs cleanly through the official adtc-profiler end-to-end on the first fully-configured attempt: valid submission.json, measured_on: "participant_laptop", params_match: true.
• Dual-register design (clinician vs patient) baked into the prompt architecture from day one, matching how the healthcare_medical domain actually gets used in the field.

What we learned

Benchmark on hardware that resembles the target, not just whatever's on your desk — our first "result" would have led us to the wrong conclusion if we hadn't sanity-checked it against representative x86 hardware. Also: read the actual release artifacts before assuming a tool ships every binary its source repo does.

What's next for ZafyaLM

• Explore whether a multilingual model (e.g. an Aya variant) can fit within the RAM budget with a smaller context window, to genuinely qualify for the African Use Case Bonus without risking OOM disqualification
• Expand the domain validation set with more triage-style prompts covering common presentations in African primary care (malaria, TB, hypertension, maternal health)
• Package a simple offline-first chat UI on top of the llama.cpp server binary for non-technical clinic staff

Built With

• bash
• gguf-q4-k-m-quantization
• github-actions
• hugging
• huggingface
• llama.cpp
• python
• qwen2.5-1.5b-instruct