Inspiration
In many rural African communities, the person making the first medical decision is not standing beside a hospital server rack.
It may be a community health worker with an ordinary laptop, intermittent electricity, unreliable internet, and a patient sitting directly in front of them.
That worker may need to distinguish routine childhood fever from danger signs, recognise when a pregnant woman needs urgent referral, explain dehydration care to a parent, or communicate with a patient in Kiswahili.
We wanted to know whether modern language models could actually be useful in that environment.
The obvious answer was to make the model very small.
Almost every constraint pointed in that direction: 8 GB of RAM, CPU-only inference, no cloud, and an evaluation that rewards both memory efficiency and throughput. A tiny model is easier to fit, easier to run, and easier to benchmark.
But healthcare made us uncomfortable with that answer.
For safety-critical clinical decision support, speed alone is not enough. The system has to understand varied language, follow context, communicate uncertainty, handle multilingual questions, and reason across situations it has not seen verbatim. Deterministic rules are valuable for hard safety boundaries, but they are not a replacement for model capability.
So we asked the opposite question:
Instead of shrinking the model until it fits the laptop, could we redesign inference so that a much stronger model only keeps the part it needs in RAM?
That question became Jamii Afya.
Jamii Afya means "community health" in Kiswahili. It is a fully offline clinical decision-support assistant designed around rural African primary care — but the technical problem underneath it became much larger:
Can a 35-billion-parameter sparse language model run usefully on a low-memory, CPU-only laptop?
We decided to chase that problem.
What it does
A health worker can ask Jamii Afya a question in English or Kiswahili.
The system first extracts clinically relevant facts and runs deterministic safety rules. Emergencies and danger signs are handled before slow model reasoning can delay them.
When authoritative guidance is needed, Jamii Afya selectively retrieves from a curated offline clinical knowledge base covering primary-care domains such as maternal health, paediatrics, infectious disease, dehydration and emergency care.
The language model then explains the situation conversationally.
Every response passes through another deterministic safety layer that checks for unsafe escalation advice, unsupported diagnosis, medication requirements, false authority attribution and other known failure modes. A failed answer gets one bounded regeneration attempt; if it still fails, the system falls back to a safe deterministic response.
The important part is what model is behind that interface.
Jamii Afya runs Qwen3.6-35B-A3B, a sparse Mixture-of-Experts model with roughly 35 billion total parameters.
The final GGUF is 12.26 GB on disk.
Yet during the official ADTC profiler run, the inference system used only:
- 2.50 GB peak RSS
- 2.44 GB steady-state RSS
- 16.0 tokens/second generation
- CPU only
- no GPU
- no cloud inference
The model is larger than the memory it is running in because Jamii Afya separates model storage from the active inference working set.
The whole model does not need to live in RAM at once.
How we built it
We did not start with this architecture.
We started where most constrained-device LLM projects start: smaller models, aggressive quantization, and increasingly careful optimisation.
But we kept running into the same trade-off.
The smaller the model became, the easier the systems problem became — and the less comfortable we were relying on it for open-ended clinical language tasks.
So we changed the problem.
Qwen3.6-35B-A3B is a Mixture-of-Experts model. Although it contains about 35B total parameters, only a small subset of its experts is activated for each token.
No weight level finetuning was done.
That means most of the model is inactive at any particular moment.
The first idea was simple:
Why should inactive experts occupy RAM?
We researched existing work and discovered that we were not the first people to ask that question. There are already systems exploring expert offloading, SSD-backed MoE inference and expert caching.
That was an important moment in the project.
Instead of abandoning the direction, we stopped trying to claim that moving experts from storage was itself new and asked a narrower question:
How far can we push the memory boundary if the entire system is designed around a strictly bounded expert working set?
Our final approach combines several ideas into one operating point.
1. Execute fewer experts without destroying router calibration
Qwen3.6 normally activates eight routed experts per token.
Simply changing eight to four damaged quality because the router probabilities were trained under a different normalization regime.
Recent research showed that execution count and normalization reference do not have to be the same.
Jamii Afya therefore uses K4/16:
- execute the top 4 experts,
- but normalize using the probability mass of the top 16.
The real router remains intact.
We execute half as many routed expert MLPs while preserving much more of the router's original calibration.
2. Quantize the part of the model that dominates storage
We did not uniformly crush every tensor to the lowest precision.
The routed expert bank dominates the model's size, so we requantized the routed expert tensors to Q2_K while leaving more sensitive dense/shared components in their existing types.
This gave us a much better quality/storage trade-off than simply applying an extremely low-bit quantization to everything.
3. Treat the expert bank as storage, not resident memory
This is where the architecture changes fundamentally.
The complete 12.26 GB GGUF remains on disk.
Jamii Afya maintains a fixed-size expert staging pool in RAM:
- 755 bounded expert slots
- 80 globally pinned hot experts
- asynchronous expert staging
- direct storage reads for missing experts
When the router selects an expert that is not resident, the required tensor data is loaded into one of the bounded slots.
When the cache is full, another expert is evicted.
The number of slots never grows with the size of the model.
So instead of:
model size ≈ required RAM
we move toward:
required RAM ≈ dense state + KV state + bounded active expert working set
That distinction is the core of the system.
To the best of our knowledge, our contribution is not any single one of these techniques in isolation, but the exact systems composition and extreme-memory operating point:
Qwen3.6-35B-A3B + real router + K4/16 execution + routed-expert-only Q2_K + explicit bounded expert staging + CPU-only llama.cpp inference below 3 GB RAM.
We implemented the runtime as a pinned and reproducible patch set on llama.cpp rather than building a separate PyTorch demonstration.
Challenges we ran into
The hardest part of this project was that almost every promising idea was wrong before one finally worked.
Our first instinct was to solve the constraint by shrinking the model.
That works extremely well for benchmarks. A 0.6B or 1.5B model can be incredibly fast and memory efficient.
But optimisation targets have a dangerous property: eventually you begin building for the metric instead of the user.
For a rural health worker, the purpose of the system is not to achieve the smallest RSS number. The memory reduction only matters if useful capability survives it.
That pushed us toward the much harder 35B route.
Naively halving the active experts hurt quality.
K4/4 looked obvious: if we execute four experts, normalize over four experts.
It was not.
Separating execution from normalization — K4/16 — recovered much of the quality while retaining the compute reduction. That became one of the central design choices.
Putting the whole model at extremely low precision also hurt quality.
The better answer was selective quantization: aggressively compress the enormous routed expert bank while preserving the rest of the model where possible.
Then we discovered that "lazy loading" was not the same thing as bounded memory.
An operating system can mmap a model and page data lazily, but that does not give us a hard working-set architecture. Under pressure, residency can still behave in ways we do not control.
We needed explicit slots, explicit staging, explicit eviction and a fixed memory budget.
Our first official profiler run appeared to destroy the entire project.
It reported approximately 15 GB peak RSS.
For a system whose central claim was sub-3-GB inference, that number was brutal.
We traced the profiler instead of changing the architecture.
The problem turned out to be the execution path: the profiler's llama-bench
process had silently entered the normal resident path instead of our bounded
executor.
After fixing the integration and adding a hard preflight that proves the bounded cache is actually active before profiling, we ran the official profiler again.
This time:
2,502.49 MB peak RSS.
The architecture had worked.
The measurement path had not.
That distinction probably taught us more than any successful experiment in the project.
Accomplishments we're proud of
The number we are most proud of is not simply the throughput.
It is the ratio between the model we store and the memory we require to execute it.
| Metric | Result |
|---|---|
| Model | Qwen3.6-35B-A3B |
| GGUF size | 12.26 GB |
| Runtime | llama.cpp, CPU only |
| Peak RSS | 2,502.49 MB |
| Steady-state RSS | 2,436.26 MB |
| Generation throughput | 16.0 tok/s |
| ARC-Easy | 0.72 acc_norm / 50 samples |
| Active routed experts | 4, normalized over top 16 |
| Expert cache | 755 slots + 80 pinned experts |
| Network required during inference | None |
The profiler did not merely see environment variables claiming that bounded mode was enabled.
Before the measured run, our preflight exercised the same patched llama-bench
binary and recorded real expert-cache traffic: tens of thousands of expert requests,
hits, misses, evictions, asynchronous loads and storage reads.
The 35B model was genuinely executing through the bounded expert runtime.
We are also proud that the final architecture did not emerge from pretending failed experiments worked.
We tried approaches that were faster and approaches that were simpler.
We attempted a weight-level fine-tuning pipeline and documented why the available 2×T4 hardware could not complete the final run.
We abandoned optimisation directions when measurements showed they were not worth their complexity.
The submitted weights are therefore not falsely described as fine-tuned. Jamii Afya's adaptation comes from its runtime architecture, clinical system prompting, structured guidance retrieval and deterministic safety system.
What we learned
The biggest lesson was that constraints can change the question you ask.
At the beginning, "8 GB laptop" sounded like a model-selection problem:
Which tiny model can fit?
By the end, it had become a systems question:
Why should model size determine resident memory at all?
Sparse models contain an enormous amount of parameter capacity that is inactive for any individual token.
Once storage and working memory are treated as separate resources, a different design space appears.
A 12 GB model file does not necessarily imply 12 GB of RAM.
A 35B model does not necessarily imply server hardware.
And optimisation does not always have to mean making the model smaller.
The second lesson was equally important for the healthcare side of Jamii Afya:
probabilistic intelligence and deterministic safety do different jobs.
The model is useful because language and clinical context are messy.
The safety system is deterministic because some boundaries should not depend on the model being clever that day.
Neither layer is enough by itself.
What's next
Jamii Afya is still a research prototype, not a clinically validated medical device.
There has been no clinician validation yet, and the current system should not be treated as a substitute for professional medical assessment. Kiswahili generation also needs review by native speakers and healthcare professionals.
But the systems result opens a direction that goes beyond this hackathon.
Today we demonstrated one 35B sparse model operating with a roughly 2.5 GB measured working set.
The next research question is broader:
How little DRAM does a large Mixture-of-Experts model actually need when parameter storage and active working memory are deliberately decoupled?
With more compute and time, we want to test the method across multiple MoE model families, map the full quality/throughput/memory Pareto frontier, study expert-cache policies, and measure behaviour across ordinary consumer SSDs and low-end CPUs.
For healthcare, that could eventually mean stronger local models reaching places where cloud AI is structurally unreliable.
But the same idea applies far beyond healthcare.
Education systems in schools without dependable internet. Agricultural assistants at the edge. Local-language tools. Private enterprise inference. Large models on machines that were never supposed to run them.
The original goal of Jamii Afya was simple:
make useful medical AI work on the laptop already in the room.
Somewhere along the way, that became a deeper question about what kind of machine a large language model actually needs.
And our answer, so far, is:
much less RAM than we thought.
Log in or sign up for Devpost to join the conversation.