-
-
server startup with plugins loading
-
API endpoints list
-
server running
-
llama-bench results
-
adaptive mode switching timeline
-
CPU variant comparison table
-
your local machine specs
-
quality scores chart
-
ADTC scores dashboard
-
STOTAL bar chart + radar
-
TPS and RAM charts
Inspiration:
The moment that started this project was not a technical paper. It was a conversation with a motorcycle taxi driver in Kampala who had been waiting three days for a mechanic to tell him why his Bajaj Boxer would not start in the mornings. The mechanic was two towns away. The answer — a weak battery struggling in cold morning temperatures — was something any AI assistant could have told him in ten seconds.
But not on his device. Not without internet. Not for free.
That gap is not about the AI models. The models exist. Phi-3-mini, Llama 3.2, Qwen — they are freely available and genuinely capable. The gap is purely about access infrastructure: cloud APIs require stable fiber, reliable electricity, and ongoing subscription costs that make no economic sense for a motorcycle taxi driver earning $8 a day in rural Uganda, or a mechanic running a small workshop in Arusha, or a matatu operator in Nairobi diagnosing a rough idle before the morning route.
The competition brief said it perfectly: the bottleneck is no longer research or raw model availability. It is access economics.
Nomad Runtime was built to remove that bottleneck entirely. Not by building yet another chatbot that requires an API key, but by solving the hard systems engineering problem underneath: how do you make a capable language model run correctly, efficiently, and reliably on the 8 GB commodity laptop that already exists on millions of desks across the continent — in schools, corner shops, transport offices, and village clinics — with zero cloud dependency, zero ongoing cost, and zero crashes?
What it does:
Nomad Runtime is an adaptive AI inference layer that sits between the operating system and the language model. Instead of running AI in a fixed, one-size-fits-all way, it continuously monitors the device and automatically decides the best way to run inference given the current hardware conditions.
The primary application built on top of it is Autex, an AI automotive diagnostic assistant. A mechanic describes a symptom, pastes an OBD fault code, or asks about a repair — and Autex provides a technically accurate diagnosis and repair pathway. The conversation runs entirely offline. No API key. No internet. No ongoing cost. The same assistant works whether you are in downtown Nairobi or a village in rural Zambia with no data signal.
Under the hood, Nomad Runtime has five layers:
1-Capability Detection : runs once on first boot. It scans the CPU model, core count, AVX2 and AVX-512 instruction sets, total RAM, GPU type, NPU presence, battery, and OS. From this it builds a permanent hardware ceiling: the system learns that this specific device can run Phi-3-mini and Llama 3.2-3B but not Llama 8B, and that ceiling is enforced from that point forward. The device never wastes a request attempting a model it cannot run.
2-Adaptive Mode Switching: runs every five seconds. It reads free RAM, CPU load, temperature, battery level, and network status, then selects one of four inference modes: Stallion (cloud API when online, highest quality), Workhorse (Llama 3.2-3B locally, balanced), Nomad (Phi-3-mini locally, efficient), or Guardian (deterministic rules, always available, zero RAM). Hysteresis prevents flickering — the system waits thirty seconds before upgrading a mode but downgrades within ten seconds for safety. The user never touches this. It just works.
3-Adaptive Learning: builds a profile of the device and its usage over time. If the system notices that RAM consistently sits below 3 GB — because the device owner always has Chrome and WhatsApp open — it pre-emptively biases toward the lighter model without waiting for a memory pressure event. If it learns that complex multi-code OBD diagnosis prompts consistently need longer, more detailed responses than the small model provides, it biases those queries toward the larger model when RAM allows. The system gets smarter about the specific machine and the specific user over days of use.
4-Plugin Architecture: means every inference engine is a peer plugin discovered automatically at startup from a plugins directory. Ollama, llama.cpp, ONNX Runtime, and the deterministic Guardian fallback are all plugins implementing the same three-method contract. Adding a new engine — TensorRT, ExecuTorch, or something that does not exist yet — means creating one new file. The router, API server, and dashboard require zero changes.
5-Guardian Mode: is the safety net that keeps Autex responding even when the device cannot run any model. When RAM drops critically low, the CPU is under extreme load, or the temperature approaches the throttling threshold, Guardian mode activates automatically. It uses deterministic keyword-matching rules to deliver safety-critical responses — brake warnings, overheating alerts, electrical fault notices — with zero model overhead. The application never crashes. It degrades gracefully instead.
How we built it:
The full stack is Python on the backend and a single-file HTML dashboard on the frontend, designed to run on Ubuntu 22.04 LTS with no framework dependencies beyond Flask.
The inference backend: uses llama.cpp directly via llama-cpp-python, compiled with AVX2 support for the ADTC Standard Laptop's CPU class. Phi-3-mini-4k-instruct quantized to Q4_K_M was selected after benchmarking four models across the accuracy/RAM/throughput tradeoff curve. Q4_K_M preserves over 98% of full-precision accuracy while fitting in 2.2 GB — leaving nearly 6 GB free on an 8 GB machine for the OS, application layer, and KV cache.
The device monitor: uses psutil for CPU, RAM, and temperature readings, with a fast non-blocking CPU measurement (primed at init rather than blocking 0.5 seconds per call) and a 1-second network probe timeout so the monitoring loop never falls behind its 5-second cadence even when fully offline.
The capability detector: reads /proc/cpuinfo directly for CPU model, vendor, and instruction-set flags (AVX, AVX2, AVX-512, FMA) on Linux, with macOS sysctl and Windows fallback paths. GPU detection tries lspci, nvidia-smi, and system_profiler in sequence, explicitly distinguishing discrete GPUs from integrated graphics since integrated-only is the expected ADTC hardware profile.
The adaptive learner: uses two exponential moving averages — one for device metrics (RAM, CPU, temperature), one for task quality satisfaction (response length relative to expected output for each prompt category). Both converge in eight to twelve observations in fast-demo mode, which translates to roughly ninety seconds of background polling on real hardware. Learned state persists to disk as JSON so it survives restarts.
The mode switcher: implements hysteresis as a two-speed timer: thirty seconds to commit an upgrade, ten seconds to commit a downgrade. The capability ceiling from first-boot detection is applied as a hard cap before hysteresis runs — structurally preventing the system from proposing a mode the hardware cannot support, regardless of how favourable a momentary snapshot looks.
The plugin registry: discovers InferencePlugin subclasses at startup by scanning the plugins directory, importing each file, and using Python's inspect module to find classes. If a plugin file fails to import or its constructor raises, it is skipped with a logged warning — one broken plugin never prevents the others from loading. If a plugin's infer() method raises an uncaught exception (a contract violation), the registry quarantines it for the rest of the session rather than propagating the crash upward.
The ADTC emulator: produces statistically realistic benchmark results without requiring physical target hardware. Performance profiles for each model/mode combination were derived from published llama.cpp community benchmarks on i5-10th through 12th-gen hardware, with per-prompt quality modifiers calibrated to the automotive diagnostic domain. It runs all five ADTC CPU variants (i5-10th, i5-11th, i5-12th, Ryzen 5 3600, Ryzen 5 5600) and produces a full STOTAL projection with the exact competition formula.
Challenges we ran into:
The double-counting bug in the adaptive learner. The device profiler was initially fed by two separate code paths — the background monitor thread and the recommend() method inside the adaptive layer. This meant every chat request that went through the adaptive path counted twice toward the EMA, making the system learn false patterns at double speed. The fix was architectural: the background monitor owns the feed, and recommend() reads from it. Discovered through a live integration test that showed the profiler reaching "chronic low RAM" conclusions on a machine that was clearly healthy.
Confidence never clearing the threshold. Even after the double-counting fix, the device profiler was stuck reporting "not enough history yet" indefinitely. The root cause was a hardcoded minimum-window floor of 5 samples inside _recompute(), combined with a fast-demo mode that set LEARN_EVERY to 4. The first recompute cycle at tick 4 was silently skipped because the window had exactly 4 items — one below the hardcoded floor. A unit test reproduced it in isolation; the fix was making the floor track LEARN_EVERY rather than a literal 5.
The plugin registry loading stale bytecode. During development, changes to plugin files were sometimes not reflected in the running server because Python was loading cached .pyc files from pycache. This caused confusing behaviour where a bug appeared fixed in the source but not in production. The fix was ensuring the plugins directory is always imported with a fresh module name that includes the plugin's stem, preventing the cache from being reused across code changes.
Snapshot speed blocking the learning loop. The device monitor's snapshot() was taking up to 2.5 seconds per call — 0.5 seconds blocked on psutil's cpu_percent(interval=0.5), plus up to 2 seconds on the TCP network probe timeout. At a 5-second polling interval, this left less than 3 seconds per cycle for everything else, and the adaptive learner was receiving far fewer samples per minute than expected. Both were fixed: cpu_percent() switched to the non-blocking interval=None form (primed in init), and PING_TIMEOUT reduced from 2 seconds to 1 second.
The submission schema validation. The ADTC profiler schema for the reproducibility block uses docker_image_digest and git_commit_sha as field names, not docker_image and git_sha. Submitting with the wrong names caused an "Additional properties are not allowed" validation error. The fix was reading the actual schema from the installed profiler package rather than guessing field names. The submission.json now passes jsonschema validation against the live schema before the archive is built.
Stallion mode as a disqualification risk. The original design treated Stallion as just another mode with cloud routing. The competition rules explicitly require zero cloud dependencies during evaluation. If a judge triggered Stallion — or if it was selected by the adaptive layer during evaluation — the submission would be disqualified. The fix was introducing an ADTC_SUBMISSION environment variable (defaulting to true) that removes Stallion from MODEL_REGISTRY at startup and makes the capability ceiling hard-cap at Workhorse.
Accomplishments that we're proud of:
The capability detection scan completes in under one second and correctly identifies what a device can and cannot run before a single inference attempt is made. On the ADTC Standard Laptop profile, it correctly identifies that Phi-3-mini and Llama 3.2-3B are viable, that Llama 8B is not, and that the integrated-only GPU means no CUDA offload is possible. This is not a lookup table — it is a live hardware scan that generalises to any x86-64 machine.
The plugin architecture genuinely works as designed. Adding a new inference engine is one file. We proved this by implementing four plugins — llama.cpp, Ollama, ONNX Runtime, and the Guardian deterministic fallback — each of which passes the same health check and infer() contract. The ONNX Runtime plugin includes logic to detect Intel's OpenVINO execution provider, meaning it would automatically use the NPU on newer Intel Core Ultra laptops if present, with no changes to the router.
The adaptive learner correctly identified a chronically memory-constrained device in simulation and began biasing toward the lighter model before a memory pressure event occurred — the core behaviour the system was designed for. In testing, we simulated forty observations from a machine with 1.8 GB average free RAM, then asked the adaptive layer to evaluate a prompt where the rule engine would have picked Workhorse. The adaptive layer correctly capped it to Nomad with the reason "RAM typically only 1.9 GB free" — exactly the outcome specified in the original design.
The submission.json passes schema validation against the live ADTC profiler package, with all fields correctly named and typed. This is more important than it sounds: many submissions are disqualified not because the code is bad but because the metadata is malformed.
The Africa-specific content — six prompts across three languages (English, Swahili, Hausa) covering Bajaj Boxer motorcycles, NAPEP tricycles, Toyota Hiace matatus, Chinese minibuses, and Land Cruiser 70s running on adulterated fuel — represents real diagnostic scenarios from real vehicles on real African roads, not translations of Western automotive examples.
What we learned:
The ADTC scoring formula punishes RAM consumption more than most participants probably expect. Moving from Workhorse (Llama 3.2-3B, 3.8 GB) to Nomad (Phi-3-mini, 2.2 GB) costs twelve points on Sacc but gains twenty-four points on Seff and eight points on Sperf — a net gain of twenty points on STOTAL for a model that is only twelve points less accurate. On an 8 GB machine, RAM efficiency is not a secondary concern. It is a primary design constraint.
Hysteresis in mode switching is not optional. Without it, a system that polls every five seconds will visibly flicker between modes as RAM fluctuates during inference, which both degrades user experience and confuses the adaptive learner's training signal. The asymmetric timers — thirty seconds to upgrade, ten seconds to downgrade — reflect a genuine design principle: it is safer to stay in a lighter mode briefly than to attempt a heavier model and OOM-crash.
Capability detection belongs at the boot layer, not the request layer. The first version of the system evaluated hardware capability on every mode decision, which was both redundant and slow. The CPU does not change between requests. The total RAM does not change between requests. Moving capability detection to a one-time first-boot scan with disk caching eliminated an entire class of bugs where momentarily favourable RAM readings would cause the system to attempt a model it could not sustain across a full conversation.
The plugin contract's most important clause is "never raise." A plugin that raises an uncaught exception is worse than a plugin that returns an error result, because an uncaught exception propagates up through the router and takes down the entire inference path for every mode. Quarantining the offending plugin and returning a structured failure is what keeps the application running. Guardian mode is the proof of this principle taken to its logical conclusion — an inference backend with no external dependencies that cannot fail.
What's next for Nomad Runtime:
Fine-tuning on African automotive data. Phi-3-mini's out-of-box accuracy on automotive diagnostics is approximately 62/100 on the ADTC evaluation set. Fine-tuning on a curated dataset of African vehicle fault codes, local vehicle types, and diagnostic dialogues in English, Swahili, Hausa, and Amharic could push this to 75-80. We have applied for the Udutech GPU credits to do this. A fine-tuned model that actually knows what a NAPEP tricycle is, or what fuel adulteration does to a diesel injector pump, would be meaningfully more useful than a model that knows American and European vehicles well.
RAG over local repair manuals. The Phi-3-mini context window is 4,096 tokens. A retrieval-augmented generation layer that chunks local repair manuals — Toyota, Bajaj, Yamaha, Sinotruk — and retrieves the relevant section before each inference call would dramatically improve accuracy on specific fault codes without requiring fine-tuning. The model stays the same. The context gets smarter.
Expanding beyond automotive. The Nomad Runtime architecture is domain-agnostic. Autex is the first application, but the same adaptive inference layer could power an agricultural extension assistant for crop and livestock advisory, a basic medical triage assistant for community health workers, or a small business accounting assistant for informal traders. Any application that currently relies on a cloud API and fails when the internet does is a candidate.
NPU acceleration on newer hardware. Intel Core Ultra CPUs (Meteor Lake, Lunar Lake) include a dedicated Neural Processing Unit accessible via the OpenVINO Execution Provider in ONNX Runtime. The ONNX Runtime plugin in Nomad Runtime already includes the detection logic. When these CPUs become common in the African refurbished laptop market — which typically lags the primary market by three to five years — Nomad Runtime will automatically use the NPU with no code changes required.
Community plugin ecosystem. The plugin architecture was designed so that a developer who builds a better quantization scheme, a faster runtime, or a domain-specific model can contribute a single plugin file without touching the core. We want Nomad Runtime to become infrastructure that the African AI developer community builds on top of, not a closed system that requires forking to extend.
Note: I generated my video using artificial intelligence because I didn't have time, and also because I work alone without a team. I did everything on my own, and time was tight, so I created the video using AI.
Log in or sign up for Devpost to join the conversation.