• 

  flow

• 

  front-page

Jamii Afya — Bringing Frontier Intelligence to African Doorsteps

Inspiration

This is not a project about making small models available to Africa. That is already possible.

The world is already moving toward frontier intelligence. If Africa is still stuck celebrating 0.5B, 1.5B, or 4B models because they fit on a laptop, we are not solving the real problem. We are designing around our infrastructure gap and quietly accepting that the gap should remain. That is painful, and it is not a serious long-term strategy for the continent.

Small models can be useful for narrow, stable tasks. They are not frontier intelligence. They do not offer the same breadth of reasoning, language, planning, coding, and context handling as a much stronger model. Scaling studies and compute-optimal training research make the relationship between model capacity, data, and capability difficult to ignore. Fine-tuning can specialise a model; it cannot turn a small base checkpoint into a frontier model. LoRA and related methods adapt a fixed pretrained parameter budget, and recent research on reasoning distillation has documented a learnability gap in very small models.

The point is not that fine-tuning is useless. The point is that even after fine-tuning and wrapping a model in a polished harness, a very small model can still be a polished imitation of a narrow slice of a stronger model. It does not become the stronger model. A successful narrow demo is not the same thing as access to frontier intelligence.

The next thing Africa needs to do is not keep fine-tuning smaller models. It is to bring frontier intelligence a step closer to our hands and doorsteps.

That is the direction behind Jamii Afya. We took a considerably stronger, 35-billion-parameter sparse Mixture-of-Experts model and made it run on commodity CPU-only hardware with no cloud and no internet during inference. The official ADTC profiler measured 2,502.49 MB peak RSS and 16.0 tok/s while the full model artifact remained 12.26 GB on disk.

This is a genuine systems breakthrough for Africa, but it is not magic and it is not finished. Even this 35B system is still a failure against the real target: Which is, running frontier intelligence on hardware we actually have . It is nevertheless a considerably more meaningful failure in the right direction than treating a tiny model as Africa's permanent ceiling simply because it is convenient to run.

The runtime is domain-portable. Healthcare is the current ADTC submission use case; the underlying system can be adapted to agriculture, education, coding, enterprise work, local-language tools, and other African ecosystems with a new system prompt, offline reference corpus, and—when justified—future domain adaptation.

The African use case

In many African communities, the person making the first medical decision is not standing beside a hospital server rack. It may be a community health worker with an ordinary laptop, intermittent electricity, unreliable internet, and a patient sitting directly in front of them.

That worker may need to recognise danger signs in a child, understand when a pregnant woman needs urgent referral, explain dehydration care to a parent, or communicate clearly in the user's language. The problem is not only access to medical knowledge. It is access to useful intelligence at the moment a decision has to be made, on hardware that can actually be afforded and maintained.

Africa still has a severe access problem. The ITU reports that only 38% of Africa's population used the internet in 2024, while rural internet use was far behind urban use. In 2024, mobile broadband still did not reach 25% of Africa's rural population. Cloud AI can be excellent and still be structurally absent from the room where it is needed.

That is why healthcare is the use case for this submission. It gives the systems problem real stakes: a community health worker may need capable, multilingual decision support without a specialist, a reliable network, or a large hospital information system nearby.

What it does

Jamii Afya is an offline clinical decision-support assistant for community health workers and nurses in rural and peri-urban African clinics. A user can ask a question in English or other African languages and receive a model-generated explanation on a commodity laptop, without a cloud API or an internet connection during inference.

The production path is intentionally simple:

conversation + user question
        ↓
short system prompt
        ↓
optional offline RAG context when retrieval is useful
        ↓
Qwen3.6-35B-A3B sparse model
        ↓
model answer

There is no application-side diagnosis classifier, urgency labeler, deterministic answer card, response rewriter, fixed fallback, or hidden clinical workflow deciding what the model is allowed to say. The system prompt establishes the role and boundaries; optional local references supply evidence when relevant(RAG); the model produces the answer.

The prompt asks the model to explain uncertainty, surface urgent actions early, avoid pretending to have examined a patient, and discuss medication names, doses, or prescriptions only when the user explicitly asks about medication or treatment. The UI hides internal model reasoning and shows the final answer, with a quiet activity indicator while generation is in progress. This keeps the interface usable without exposing private reasoning or turning the answer into a transcript of internal instructions.

The healthcare scenario is deliberate. A community health worker may not have a specialist, a reliable network, or a large hospital information system nearby. Local inference can keep patient information on the device and continue working when connectivity is absent. It is decision support, not a diagnosis and not a substitute for a qualified clinician.

How we built it

We did not begin by trying to make a 35B model fit. We began with the standard constrained-device instinct: smaller models, aggressive quantization, and careful runtime optimisation. That route is useful when the task is narrow. It was not enough for the open-ended language, context, multilingual communication, and clinical uncertainty we wanted to preserve.

The change in direction was fundamental. Instead of treating total parameter count as the RAM requirement, we treated the model file as storage and the currently needed expert tensors as a bounded working set.

No weight-level fine-tuning was performed. The submitted weights are not presented as a fine-tuned healthcare model. We worked on the runtime, sparse execution, quantization, prompt, offline retrieval path, packaging, and measurement. That matters because it makes the result honest and portable: the same systems can later be adapted to agriculture, education, coding, enterprise work, local-language assistance, or another domain with a different prompt, reference corpus, and—if resources justify it—domain-specific fine-tuning.

Start from a stronger model

The base model is Qwen3.6-35B-A3B, a sparse Mixture-of-Experts model with approximately 35B total parameters and roughly 3B active parameters per token. The model card describes 256 experts, with eight routed experts and one shared expert activated in its normal configuration.

This is not a claim that the model is magically frontier-grade in every task. It is a claim about the direction of capability. We chose to start from a much stronger general base than the tiny models that are easy to run, then redesign the inference system around the hardware constraint.

Execute fewer experts without throwing away router information

The normal model activates eight routed experts per token. Simply changing that to four and renormalizing over only those four damaged quality because the router's probability distribution was not trained for that new reference set.

Our K4/16 operating point separates execution count from normalization mass:

• execute the top 4 routed experts;
• normalize their contribution using the probability mass of the top 16.

The real router remains in the path. We execute less routed-expert work without pretending that the model's router was trained for a different probability normalization regime.

Quantize where the storage pressure actually is

The routed expert bank dominates the model's storage footprint. We therefore requantized the routed expert tensors to Q2_K, while preserving more suitable types for sensitive dense, shared, router, embedding, and state-space components.

This is a targeted storage trade-off, not a claim that every tensor tolerates the same precision. Uniformly crushing the entire model was easier to describe and worse to use.

Treat the expert bank as storage, not resident memory

The complete 12.26 GB GGUF remains on disk. The runtime maintains a fixed expert staging pool in RAM:

• 755 bounded expert slots;
• 80 globally pinned hot experts;
• asynchronous staging for missing experts;
• explicit slot reuse and eviction;
• direct reads from the model file when a selected expert is not resident.

When the router selects an expert that is absent, its tensor data is staged into one of the bounded slots. When the cache is full, another slot is reused. The number of slots does not grow with the total model size.

The design therefore moves toward:

required RAM ≈ dense state + KV state + bounded active-expert working set

instead of:

required RAM ≈ total model size

The contribution is not the claim that expert offloading, sparse execution, or storage-backed inference was invented here. Prior work exists. The contribution is the measured composition and operating point:

Qwen3.6-35B-A3B + real router + K4/16 execution + routed-expert-only Q2_K + explicit bounded expert staging + CPU-only llama.cpp inference below 3 GB RSS.

Keep the deployment path real

The repository packages the runtime for Linux, macOS, and native Windows. It uses CPU-only llama.cpp inference, a versioned prompt, optional offline BM25 retrieval, model hash verification, a pinned runtime build, and a browser UI. The repository also documents the platform assumptions instead of presenting a single successful machine as universal proof.

Users should read ARCHITECTURE.md before running or evaluating the repository. It is the source of truth for the sparse executor, memory measurements, platform assumptions, benchmark paths, and limitations.

Challenges we ran into

The hardest part was accepting that the most obvious engineering answer was not the answer we wanted to ship.

Small models fit. The capability ceiling did too. A 0.5B, 1.5B, or 4B model can be genuinely useful when the task is narrow. But if the goal is broad, open-ended intelligence, stopping there because the model is convenient is not a strategy for closing Africa's capability gap. It is a strategy for making the gap permanent. A polished narrow fine-tune is still not a route to frontier intelligence.

Fine-tuning did not change the underlying constraint. Fine-tuning can make a model speak a domain's vocabulary or imitate a desired answer style. That is valuable. But it does not add the broad pretrained capacity that was not present in the base checkpoint. Distillation can transfer useful behaviour, but the student still has its own capacity and learnability limits. This is why our primary work was not another fine-tuning exercise; it was inference systems engineering.

Naive expert halving hurt quality. K4/4 looked obvious and was not enough. The K4/16 separation preserved more router information while reducing executed experts.

Uniform ultra-low-bit quantization hurt quality. Selectively compressing the storage-heavy routed bank was a better trade-off than treating every tensor as equally disposable.

Lazy loading was not bounded memory. Memory mapping can defer reads, but it does not create an application-controlled resident working set. We needed explicit slots, staging, reuse, eviction, and a hard cache shape.

The first profiler path appeared to disprove the project. An early run reported approximately 15 GB peak RSS because llama-bench had entered the normal resident path instead of the bounded executor. We traced the measurement path, fixed the integration, and added a preflight that exercises the same patched binary and records real expert-cache traffic before the expensive run.

The corrected ADTC profiler measurement confirmed the bounded path at 2,502.49 MB peak RSS and 16.0 tok/s. Individual observed profiler runs were 16.46 and 15.5 tok/s; which is in line with the profiler's 16.0 toks/s.

Accomplishments we are proud of

The result we are most proud of is not simply a throughput number. It is the fact that a model with a 12.26 GB artifact can operate with a roughly 2.5 GB measured resident working set on CPU-only commodity hardware.

Metric	Result
Model	Qwen3.6-35B-A3B
GGUF size	12.26 GB
Peak RSS	2,502.49 MB
Steady-state RSS	2,436.26 MB
Generation throughput	16.0 tok/s
ARC-Easy	0.72 acc_norm / 50 samples
Active routed experts	4, normalized over top 16
Expert cache	755 slots + 80 pinned experts
Inference network requirement	None
Weight-level fine-tuning	None

The profiler preflight did not merely inspect an environment variable claiming that bounded mode was enabled. It exercised the same patched llama-bench binary and recorded expert requests, hits, misses, evictions, asynchronous loads, and storage reads before the full measurement.

We are also proud of what we refuse to claim. We did not invent MoE offloading. We did not fine-tune the submitted weights. We do not have clinician validation. We do not claim that a 35B model on a laptop has the latency, reliability, or quality of a production frontier service. Our claim is narrower and stronger:

we integrated, measured, and documented a substantially larger sparse model running within a laptop-scale working set, and we made the complete path available offline.

What we learned

The first lesson was that constraints can change the question. At the beginning, “8 GB laptop” sounded like a model-selection problem:

Which tiny model can fit?

By the end, it had become a systems problem:

Why should total parameter count determine resident memory at all?

Sparse models contain parameter capacity that is inactive for any individual token. Once storage and working memory are treated as separate resources, a different design space appears. A 12 GB model file does not automatically imply 12 GB of resident RAM.

The second lesson was strategic. Small models are not worthless. They are often the right tool for classification, extraction, routing, and other narrow jobs. But they should not become Africa's default definition of what local AI is allowed to be. We should not idolise a 1.5B or 4B model merely because an interesting fine-tune makes it look impressive on one task. Narrow success is not the same as frontier access.

The third lesson was that adaptation and capability are different things. Fine-tuning can specialise a model. Retrieval can add current local facts. A system prompt can establish behaviour. None of those should be confused with increasing the model's native capacity. Jamii Afya's runtime is therefore the important reusable layer: healthcare is one deployment domain, not the boundary of the technology.

The fourth lesson was that benchmark architecture matters as much as inference architecture. A benchmark can report a number for the wrong execution path if the harness does not verify the runtime configuration. Preflight checks, hashes, pinned commits, cache markers, and explicit limitations are part of the system.

What's next

First, we want to turn this from a competition submission into a serious research program. The amount of systems engineering involved—the bounded expert working set, storage-backed staging, routing changes, selective quantization, cross-platform packaging, and profiler validation—deserves to be written up as an academic paper, reproduced, challenged, and extended. With enough funding and support, this could become a much larger research effort for African ecosystems rather than a one-off demo.

Second, we want to map the full quality/throughput/memory frontier across more Mixture-of-Experts families, expert-cache policies, consumer SSDs, low-end CPUs, and real deployment environments. 35B is still a failure against the actual goal and a compromise, not a finish line. The goal is to keep pushing the frontier of what can run locally until the compromise is no longer so severe.

Third, we want to validate the healthcare deployment with clinicians and native African language speakers. The current system is a research prototype. It has no clinician validation, it is not a medical device, and it must not replace professional assessment.

Fourth, we want to take the same runtime into other domains. The model is not limited to healthcare. With domain-specific references and, when justified, future adapter or weight-level fine-tuning, the same foundation could support agriculture, education, coding, enterprise productivity, local-language tools, and private offline assistance. We are using healthcare here because it makes the stakes obvious.

The original goal was not to make Africa permanently satisfied with a model that fits the smallest possible machine.

The goal is to make stronger intelligence physically reachable on the machines already in African homes, clinics, schools, and businesses.

Jamii Afya is one proof that the direction is technically real. It is not the destination.

References

These references support the claims about scaling, fine-tuning, small-model limitations, connectivity, the base model, the sparse-execution direction, the clinical use case, and the competition requirements. They do not constitute clinical validation of Jamii Afya.

1. Kaplan et al., “Scaling Laws for Neural Language Models” — empirical scaling relationships between model size, data, compute, and language-model loss.
2. Hoffmann et al., “Training Compute-Optimal Large Language Models” — evidence that model size and training data both matter for compute-efficient capability.
3. Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models” — parameter-efficient adaptation that freezes pretrained weights and learns low-rank updates.
4. Li et al., “Small Models Struggle to Learn from Strong Reasoners” — evidence for a small-model learnability gap in direct reasoning distillation; this supports a cautious, not absolute, argument about small-model limits.
5. Qwen3.6-35B-A3B model card — base model architecture, total and active parameter counts, expert layout, and model-reported evaluations.
6. llama.cpp — the CPU/GGUF inference runtime extended by the pinned implementation.
7. Chen & Yao, “Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Models” — prior work motivating the separation between executed experts and normalization reference mass.
8. ITU, Measuring Digital Development: Africa region — Africa's internet-use, mobile-broadband, and urban-rural connectivity gaps.
9. WHO Pocket Book of Hospital Care for Children — public clinical reference for childhood danger signs and common illnesses.
10. WHO African Region: Community Integrated Management of Childhood Illness — community-health context for the African deployment scenario.
11. Africa Deep Tech Challenge 2026 rules and official submission template — competition scoring, hardware, provenance, and submission requirements.

Built With

• gguf
• llama.cpp
• lora
• peft
• python
• pytorch
• qlora
• qwen3
• transformers