Zephyr

• 

  Web interface to chat with the model

• 

  Sample query to the model

• 

  Sample query to the model

• 

  African Language: Swahili

Inspiration

Computer Science students in African higher-education institutions face a dual barrier when learning to code: prohibitive mobile data costs and unreliable internet connectivity. While cloud-based LLMs like ChatGPT and GitHub Copilot exist, they are entirely inaccessible in offline or low-bandwidth environments. Furthermore, generalist cloud models often undermine the learning process by directly generating complete solutions for assignments, violating academic integrity and preventing students from grasping core logic. We needed a solution that was completely offline, capable of running on standard university laptops, and designed strictly to teach rather than solve.

What it does

Zephyr is a 100% offline AI coding assistant built specifically for educational environments. Operating entirely within an 8 GB RAM physical constraint, it features:

• Pedagogical Refusal System: Zephyr actively refuses to write complete assignment solutions, instead guiding students through logic, syntax errors, and conceptual understanding.
• Textbook Grounding (RAG): Explanations are grounded in an embedded offline vector database containing Think Python, 2nd Edition, ensuring academic accuracy.
• Execution Sandbox: Students can paste code into the web UI, which Zephyr runs locally in an isolated Python subprocess to explain outputs or diagnose tracebacks.
• Bilingual Support: The tutor dynamically translates technical explanations into Swahili to support localized learning.

How we built it

Zephyr is built on a highly optimized, local microservice architecture:

• Inference Engine: We selected Qwen2.5-Coder-3B-Instruct quantized to GGUF Q4_K_M running on llama.cpp. This achieved the optimal balance of coding accuracy while keeping the memory footprint under 3.5 GB.
• Backend Orchestration: A local Python FastAPI server acts as the central brain. It handles RAG retrieval (using ChromaDB and all-MiniLM-L6-v2), prompt assembly, and the isolated 3-second code execution sandbox.
• Frontend Interface: We built a lightweight, responsive web application using Streamlit. This allowed for rapid prototyping of the chat interface, Swahili toggles, and sandbox input, all served locally without needing an external web server.

Challenges we ran into

Operating under a strict 8 GB RAM and CPU-only constraint introduced significant engineering hurdles.

• Memory and Thermal Management: Early tests with 7B models caused Out-Of-Memory (OOM) crashes. We downscaled to a highly capable 3B model and capped the context window at 4096 tokens to prevent thermal throttling on budget CPUs.
• Offline Telemetry Crashes: By design, the application has zero network access. However, underlying dependencies in the vector database (ChromaDB) attempted to send telemetry data on startup, resulting in fatal crashes when the network request failed. We had to implement strict backend configurations to forcefully disable all background telemetry.
• Strict C-Extension Compatibility: While developing the RAG ingestion pipeline, we ran into deep metaclass compilation errors (tp_new) between Python 3.14 and older Protobuf C-extensions. We had to surgically bypass these binaries to force pure-Python fallbacks, ensuring the application could run across various Linux environments.

Accomplishments that we're proud of

• Performance Metrics: We achieved a generation throughput of 17.92 tokens per second and a peak memory footprint (RSS) of only 3.45 GB—well below the ADTC 7.0 GB disqualification ceiling.
• Robust Local Orchestration: Successfully binding a Streamlit frontend to a FastAPI asynchronous router, a vector database, and a C++ inference server, all operating seamlessly on a single local machine.
• Pedagogical Efficacy: Implementing a prompt architecture that successfully overrides the base model's instruction-following tendencies, forcing it to act as a strict tutor rather than an answer generator.

What we learned

Building Zephyr required deep technical execution across multiple domains. We gained practical experience in LLM memory profiling, GGUF quantization trade-offs, and vector database chunking strategies. Furthermore, building the execution sandbox taught us how to safely manage and isolate untrusted Python subprocesses using strict time and memory bounds.

What's next for zephyr

While the current Streamlit architecture proves the concept beautifully, our immediate next step is Zephyr Desktop.

We are actively migrating the frontend from Streamlit to a native desktop application using Tauri, Rust, and React. This transition will drop our GUI RAM overhead from hundreds of megabytes down to just ~40 MB, allowing the LLM to utilize more system resources. The native desktop version will also introduce Rust-powered Inter-Process Communication (IPC) for real-time memory token tracking and local SQLite integration so students can save and persist their chat histories across semesters.

Built With

• fastapi
• llama.cpp
• python
• qwen2.5-code