Inspiration Cloud-based AI assistants often expose sensitive enterprise and personal data to third-party servers, introducing severe privacy risks and unpredictable API costs. We were inspired to answer a critical question: Can we build a powerful, context-aware AI assistant that runs 100% offline on standard consumer hardware? We built BORZ to prove that privacy-first, local AI is not just possible, but highly effective.
What it does BORZ is a fully local, privacy-first AI assistant and document intelligence web app. Users can upload private documents (PDFs, Markdown, TXT), and BORZ locally indexes them into a custom Retrieval-Augmented Generation (RAG) pipeline. Users can then ask complex questions, and BORZ will extract data and summarize the documents entirely offline, acting as a secure, simulated alternative to cloud-reliant assistants like Alexa.
How we built it We engineered the backend using llama.cpp to run a highly compressed Qwen2.5 1.5B GGUF model directly on standard CPU hardware. For the RAG pipeline, we used FastEmbed to generate offline embeddings and stored the vectors in a lightweight local SQLite database to bypass the need for heavy vector DB servers. Finally, we wrapped the entire architecture in a sleek, custom-styled Streamlit web application.
Challenges we ran into Running LLM inference and a complete RAG embedding pipeline simultaneously on constrained hardware (a Ryzen 5 CPU with limited RAM) caused heavy memory bottlenecks. We overcame this by ruthlessly optimizing the document chunking process (350 words with a 50-word overlap) and using the GGUF quantization format to drastically reduce the model's footprint without losing accuracy.
Accomplishments that we're proud of We successfully eliminated the need for cloud APIs. We are incredibly proud of achieving near-instant offline retrieval speeds using nothing but an SQLite database and raw CPU power. Our custom glass-morphism Streamlit UI also elevates the user experience to feel like a premium, production-ready product.
What we learned We learned how to deeply optimize mathematical vector storage without relying on external cloud databases, and how to effectively manage LLM context windows natively. We also mastered dynamic path routing and Streamlit session states to ensure seamless cross-platform deployment.
What's next for BORZ AI The next phase is optimizing the local voice-input subsystem (Faster-Whisper) to run in real-time within the Streamlit UI, effectively turning BORZ into a fully voice-activated, offline smart-home assistant. We also plan to integrate local tool-calling capabilities to allow BORZ to securely interact with local operating system files.
Log in or sign up for Devpost to join the conversation.