Offline Code Agent

Inspiration

In professional software development, engineers frequently face the daunting task of understanding large, unfamiliar codebases. This process is time-consuming and often hindered by a lack of documentation. Furthermore, in many corporate or high-security environments, developers cannot use cloud-based AI tools due to data privacy concerns. We were inspired to build a tool that solves this problem: a powerful, private, and completely offline AI assistant that can become an expert on any codebase, boosting developer productivity and preserving data security.

What it does

The Offline Code Agent is a fully local pair programmer. A developer first points the agent to a local code repository, which the agent then indexes using syntax-aware parsing. Once indexed, the developer can use a simple web UI to:

Ask complex questions about the code's architecture, functionality, or logic.

Generate pytest unit tests for any specific function or class.

The entire process, from indexing to answering questions, runs 100% offline, ensuring that the user's source code never leaves their machine.

How we built it

The agent is built on a Python backend with a Flask web server. The core of the application is a Retrieval-Augmented Generation (RAG) pipeline: AI Models: We use Ollama to run the gpt-oss:7b model for reasoning and the nomic-embed-text model for generating embeddings. Code Parsing: The tree-sitter library allows for intelligent, syntax-aware chunking of source code into functions and classes. Vector Database: ChromaDB stores the vector embeddings of the code chunks for efficient retrieval. Frontend: The UI is built with simple HTML, CSS, and vanilla JavaScript, with the fetch API handling the real-time streaming of the AI's response.

Challenges we ran into

We faced a series of significant technical hurdles that required extensive debugging: Model Unavailability: The required gpt-oss models were not initially available on the public Ollama registry, forcing us to pivot to codellama and later switch back. Hardware Limitations: We discovered that large language models (like the 20b and 34b versions) have substantial RAM requirements, which led us to select the more efficient 7b model for broader compatibility. Dependency Hell: We encountered numerous installation issues with the Python environment, including corrupted NumPy installations and version conflicts with tree-sitter and its dependencies, which we ultimately solved by rebuilding the environment with pinned, stable library versions. Browser Caching: We also had to solve common web development issues, such as browser caching that prevented UI updates from appearing correctly.

Accomplishments that we're proud of

We are incredibly proud of building a complete, end-to-end, and genuinely useful AI application from scratch. Overcoming the persistent environment and dependency issues was a major accomplishment that tested our problem-solving skills. The final product is a polished, stable, and powerful tool that successfully fulfills our original vision of a secure, offline AI assistant for developers. The "Write a Unit Test" feature is a particular point of pride, as it demonstrates a truly "agentic" capability.

What we learned

This project was a deep dive into the practicalities of running large language models locally. We learned: The critical importance of a clean, well-managed development environment. The trade-offs between model size, performance, and hardware requirements. The power and complexity of building an effective RAG pipeline, from intelligent data chunking to efficient retrieval. The necessity of robust error handling and clear user feedback, both in the terminal and in the UI.

What's next for Offline Code Agent

The current application is a powerful proof-of-concept. The next steps would be to expand its agentic capabilities and improve its integration into developer workflows.

More Agentic Actions: Add new features like "Refactor this function" or "Generate documentation for this entire file."

VS Code Extension: Rebuild the UI as a Visual Studio Code extension for seamless integration directly within the developer's editor.

Expanded Language Support: Add tree-sitter grammars for more languages like C++, C#, and Rust to make the tool even more versatile.