TinyArmCopilot: SLLM Debugging on Arm

Inspiration

Developers often debug with sensitive source code, logs, and internal documentation. Sending that context to a cloud model is not always practical, private, or fast. We wanted to explore whether a small language model could become useful as a local debugging copilot on Arm-powered devices.

TinyArmCopilot is an offline SLLM debugging assistant built around llama.cpp and quantized GGUF models. It analyzes code, error logs, and project documentation locally, then returns grounded debugging suggestions with retrieved source snippets.

What It Does

TinyArmCopilot lets a developer import a folder of Markdown files, source code, and logs into a lightweight local RAG index. When the user asks a question or pastes an error message, the system retrieves relevant local context and sends a compact prompt to a quantized small language model running through llama.cpp.

The project focuses on Arm optimization by measuring and improving:

• Model size on disk
• Runtime memory usage
• Time to first token
• Tokens per second
• Retrieval latency
• Answer usefulness for debugging workflows

How We Built It

We built the prototype with a Python-based local RAG pipeline, GGUF-format SLLMs, and llama.cpp as the inference backend. Documents are chunked, embedded, indexed, and retrieved locally. The retrieved context is compressed into a prompt designed for debugging tasks, so the model can produce practical next steps without requiring a cloud API.

The system is designed to run on Arm64 environments such as Arm-powered laptops, development boards, or Arm cloud instances.

What We Learned

We learned that small local models become much more useful when paired with a focused retrieval pipeline and carefully structured prompts. We also learned that optimization is not only about raw model speed: reducing context size, caching embeddings, choosing the right quantization level, and tuning thread/batch settings can significantly improve the end-user experience.

Challenges

The biggest challenge was balancing model quality with local performance constraints. Smaller quantized models are fast and memory-efficient, but they need high-quality retrieved context to answer technical questions well. Another challenge was designing benchmarks that reflect real developer workflows instead of only measuring synthetic token generation speed.

What Is Next

Next, we plan to add more Arm-specific benchmark profiles, support more GGUF SLLMs, improve answer evaluation, and package the tool as a reusable local debugging assistant for open-source projects.

Built With

• arm64
• chroma
• faiss
• fastapi
• gguf
• language
• llama.cpp
• local
• markdown
• models
• or
• python
• rag
• sentence
• small
• sqlite
• transformers