Inspiration
Cloud-hosted coding assistants assume a stable, generous internet connection and a willingness to pay a recurring per-token or per-seat cost. For self-taught developers, bootcamp learners, and junior developers on modest hardware with metered or intermittent connectivity — a very common reality across Nigeria, Kenya, Uganda, and much of the continent — that assumption is a real barrier, not a minor inconvenience. I wanted a coding tutor and debugging assistant that works the same whether you're on fast fibre or no connection at all.
What it does
Nova Dev AI is a coding assistant that runs entirely on-device using llama.cpp against a quantized GGUF model (Qwen2.5-Coder-1.5B-Instruct, Q4_K_M, ~1.1 GB). After a one-time model download, it needs zero network connection to generate code, explain programming concepts, or help debug — your code and questions never leave your machine. It's paired with a small, curated local knowledge base (knowledge/*.md) covering programming fundamentals — variables and types, functions, loops, debugging strategy, and working offline. Before every response, a lightweight keyword-overlap retriever (rag_retrieve.py, stdlib-only, no embeddings model) finds the most relevant document and prepends it to the prompt sent to the model. This grounds the 1.5B model's explanations in a fixed, vetted curriculum rather than relying on pure open-ended generation — genuinely changing what the model sees and produces, not just a cosmetic add-on.
How I built it
Base model: Qwen2.5-Coder-1.5B-Instruct, from Qwen's official GGUF release on Hugging Face (Apache-2.0). It's a code-specialized model family, so it outperforms a general-purpose model of similar size on coding-tutoring, debugging, and generation tasks. Quantization: Q4_K_M (1.1 GB on disk) — a well-established balance between size and quality, comfortably inside the 8 GB RAM / 7 GB peak-usage ceiling for the target laptop profile. Runtime: llama.cpp, pure CPU inference, no discrete GPU required. Retrieval: a stdlib-only keyword-overlap scorer instead of an embeddings-based RAG pipeline, to avoid the RAM and complexity cost of a second model. Broader product layer: a separate, polished PWA I built on WebLLM/WebGPU (browser-based, MLC-format weights) implements the full chat UX, conversation history, and coding-mode prompts. This GGUF/llama.cpp submission is a deliberately separate, competition-compliant layer, since the official evaluation only supports llama.cpp.
Challenges I ran into
Balancing model size against the 7 GB hard disqualification ceiling. A larger Qwen2.5-Coder tier (3B/7B) would likely score higher on accuracy, but risks an out-of-memory disqualification on the 8 GB target machine once the KV cache and runtime overhead are accounted for. I chose the safer 1.5B model. Keyword-overlap retrieval is not semantic search. I found and disclosed a real limitation: a query like "I found a bug and don't know how to fix it" sometimes scored functions.md slightly higher than the more topically apt debugging-strategy.md, due to incidental term overlap — an accepted trade-off for staying dependency-light and RAM-cheap. Concurrency limits on my development device. Two concurrent assistant processes ran successfully on my ARM64 Android/Termux dev device, but four concurrent processes exceeded its practical resource capacity (exit code 247) — a device-specific observation, not a failure of the core single-process pipeline. I also caught and fixed a factual error in my own knowledge base (functions.md incorrectly implied a no-return Python function returns undefined; corrected to None) during functional testing.
Accomplishments that I'm proud of
A fully offline, dependency-light coding assistant that runs within an 8 GB RAM budget on CPU-only hardware. A genuinely load-bearing cross-disciplinary pairing between the coding-assistant domain and programming education, verified to actually change model output, not just described as a feature. Honest, disclosed limitations (retrieval mismatches, concurrency ceiling) instead of overstated claims — verified end-to-end with real functional and concurrency tests on-device.
What I learned
Small, code-specialized models paired with a narrow, well-curated local knowledge base can meaningfully outperform relying on the base model alone — without needing an embeddings model or any network access. I also learned to respect hard resource ceilings as a design constraint from day one, not an afterthought to optimize around later.
What's next for Nova Dev AI
Improve retrieval quality (e.g. a lightweight semantic scoring approach that's still RAM-cheap) to address the keyword-overlap mismatch I identified. Expand the local knowledge base beyond Python fundamentals to cover more languages and more advanced debugging patterns. Bring learnings from this offline llama.cpp layer back into the WebLLM/WebGPU-based product for a more consistent experience across both.
Log in or sign up for Devpost to join the conversation.