KnowledgeForge AI

A Production-Grade, 100% Offline Retrieval-Augmented Generation Ecosystem

Comment

KnowledgeForge AI

💡 Inspiration

In an era where enterprise data privacy is paramount, uploading sensitive documents to cloud-based LLMs is a massive security risk. We noticed that while open-source models were becoming incredibly powerful, the tools to run them securely on local hardware were either fragmented, painfully slow, or required an internet connection to function properly.

I was inspired to build KnowledgeForge AI—a completely air-gapped, production-grade Retrieval-Augmented Generation (RAG) ecosystem that brings the intelligence of cloud AI directly to your local machine, ensuring that your data never leaves your hard drive.

⚙️ What it does

KnowledgeForge AI is a 100% offline document analysis and chat engine. It allows users to ingest complex files (PDFs, DOCX, CSVs, TXT, HTML) and instantly chat with their data with pinpoint accuracy.

It provides three distinct ways to interact with the intelligence:

The Web Dashboard: A beautiful, responsive Streamlit interface for seamless document management. The Native Desktop Edition: A standalone Windows application utilizing a PyWebView wrapper, bringing the backend directly to your desktop. The Programmatic API: A Jupyter Notebook environment for developers who want raw access to the retrieval engine. When a user asks a question, the engine retrieves the most relevant chunks, reranks them using a heavy-duty Cross-Encoder, and streams the answer back alongside verified source citations, page numbers, and exact confidence scores.

🛠️ How I built it

The architecture is entirely Python-centric and built for extreme speed: Hybrid Retrieval Strategy: I combined Dense Vector search using FAISS (for semantic understanding) with Sparse Keyword search using BM25 (for exact terminology matching) to guarantee maximum recall. Embeddings & Reranking: Documents are embedded locally using all-MiniLM-L6-v2. To ensure the LLM isn't confused by irrelevant context, I implemented the BAAI/bge-reranker-base Cross-Encoder to re-score and filter the retrieved chunks. LLM Inference: The generation layer is powered by Ollama running quantized models (like Gemma). The Engine: I built a custom Recursive Overlap Chunker to preserve semantic sentence boundaries during ingestion. Deployment: The entire stack is containerized with Docker, mounting persistent volumes for the databases and HuggingFace caches to prevent redownloads.

🚧 Challenges I ran into

Database Bottlenecks: Initially, I used ChromaDB, but I encountered severe SQLite locking issues and latency spikes on local Windows environments. I solved this by ripping out ChromaDB and engineering a custom, lightning-fast FAISS integration from scratch, resulting in near zero-latency lookups. Host Port Conflicts & Firewalls: Running local servers often leads to port conflicts (e.g., ghost processes blocking port 11434). To prevent the app from crashing on the judges' or users' machines, I built a Global Try-Catch Shield. If the network blocks the **Ollama* server, the UI gracefully intercepts the crash and renders a helpful markdown guide. Absolute Offline Capability: To combat extreme firewalls that block local socket connections entirely, I integrated llama-cpp-python as a fallback engine. This allows inference to run directly in the Python RAM, bypassing the host's networking layer completely!

🏆 Accomplishments that I'm proud of

Achieving true Hybrid Search with a Cross-Encoder reranker completely offline on consumer hardware. Creating a codebase that is essentially "foolproof." The error handling ensures that no matter how restrictive the user's environment is, the UI never throws an ugly Python traceback. Successfully wrapping a complex Python FastAPI backend into a clean, single-click Native Desktop executable using PyWebView.

🧠 What I learned

This project was a masterclass in advanced RAG architectures. I learned the profound difference between simple cosine similarity and actual Cross-Encoder reranking. I also gained deep experience in optimizing Docker for Windows environments, managing low-level socket connections, and handling multi-threaded PyTorch memory limits.

🚀 What's next for KnowledgeForge AI

Multi-Modal Support: Upgrading the ingestion pipeline to understand charts, diagrams, and images embedded inside PDFs using local Vision Language Models (VLMs). Multi-Agent Orchestration: Implementing autonomous sub-agents that can actively crawl local directories and update the vector database in real-time. Enterprise Clustering: Distributing the FAISS index across multiple nodes to support gigabytes of corporate data at lightning speed.

Built With

  • all-minilm
  • bge-reranker
  • css3
  • docker
  • docker-compose
  • faiss
  • fastapi
  • gemma
  • html5
  • huggingface
  • javascript
  • jupyterlab
  • llama-3
  • llama-cpp
  • markdown
  • ollama
  • pymupdf
  • python
  • python-docs
  • pytoch
  • pywebview
  • rank-bm25
  • rest-api
  • sentencetransformers
  • streamlit
Share this project:

Updates