Divinity AI

Inspiration

General-purpose AI models like ChatGPT and Gemini can sometimes generate information that sounds convincing but is actually incorrect. When answering questions about religious texts, they may cite the wrong verse, misattribute sources, or produce explanations that have no reliable basis. For people seeking trustworthy guidance, these mistakes can be misleading because there is often no way to verify where the information came from.

We wanted to build a system based on a simple principle: "If it's not in the source, don't say it." Every answer must be grounded in an authentic, verifiable passage from the Quran or a trusted hadith collection, complete with a clear citation. Instead of relying on the AI's memory or making educated guesses, the system only answers using information that can be traced directly to its source.

What it does

DivinityAI is a Retrieval-Augmented Generation (RAG) system that answers questions using only the Quran and six major authenticated Hadith collections. Instead of relying on an LLM's internal knowledge, it retrieves relevant passages, verifies every citation against the canonical corpus, and generates responses grounded exclusively in those sources. The system filters out unrelated questions, improves retrieval with hybrid search and query rewriting, detects hallucinations, and applies safety checks for sensitive religious topics. Every answer includes clickable, verifiable citations, supports Arabic, English, and Indonesian, and runs entirely on free-tier LLM APIs without requiring a GPU.

How we built it

Backend: Django 5.2 + Django REST Framework serving as the API layer. The RAG pipeline is a modular, stateless orchestrator (PipelineService) that wires together independent stages — intent routing, query rewriting, hybrid retrieval, citation verification, evidence sufficiency checking, grounded generation, and safety checks.

Vector Database: ChromaDB in client-server mode, with two collections (quran_collection and hadith_collection) storing 6,236 Quran ayahs and ~34,000 hadith narrations.

Embeddings: Ollama running embeddinggemma for dense vector embeddings, combined with rank_bm25 (BM25Okapi) for sparse retrieval. Results are merged via Reciprocal Rank Fusion (k=60).

LLM: OpenRouter API with Gemini 2.5 Flash for generation/intent/hyDE and Llama 3.3 70B for evidence checking and hallucination detection. The system is designed to run on free-tier APIs — up to ~200-250 queries/day at $0 cost.

Citation Verification: A pure Python cascade — exact string match → NFKD normalization + tashkeel stripping → rapidfuzz fuzzy matching → LLM semantic check as last resort. No LLM bias in the most reliable verification stages.

Frontend: React 19 + Vite + Tailwind CSS v4 with an ornate Islamic-themed UI featuring gold accents, Arabic calligraphy, a crescent-and-star motif, and Georgian serif typography. Citations are rendered as inline gold badges, and sources are expandable with both Arabic and English text.

Infrastructure: Docker Compose with Nginx reverse proxy, multi-stage Dockerfiles for dev/prod, and a comprehensive Makefile for orchestration.

Challenges we ran into

Hallucination is a multi-layered problem. Early on, we realized that a single guard (e.g., just telling the LLM "don't hallucinate") is useless. We needed four independent layers: corpus lock, source tagging, deterministic citation verification, and a post-generation LLM check. Each layer catches what the previous one misses. The citation verifier — pure Python, no LLM — ended up being the most reliable guard.

Running on free-tier APIs without a GPU. We couldn't afford hosted embedding APIs or GPU instances, so we designed the entire pipeline around Ollama running locally for embeddings and free-tier LLM APIs (Gemini 2.5 Flash, Groq Llama 3.3) for generation. This required careful rate-limit budgeting and a fallback chain across providers.

Balancing retrieval quality with latency. The full Phase 2 pipeline makes up to 7 LLM calls per query. Keeping end-to-end latency under 8 seconds while maintaining citation accuracy above 95% required careful prompt engineering, parallelization where possible, and knowing when to skip expensive steps (e.g., no HyDE for simple verse lookups).

Accomplishments that we're proud of

It works

What we learned

Deterministic beats probabilistic for verification. We initially planned to use an LLM for citation verification, but switched to pure Python string matching (exact → normalized → fuzzy). It's faster, cheaper, and — crucially — it cannot hallucinate. The LLM is only invoked as a last resort for semantic similarity checks.

RAG pipelines need orchestration, not just a chain. Intent-based routing (verse lookup vs. hadith search vs. fiqh guidance) dramatically improves both retrieval quality and user experience. A verse lookup should use BM25 exact matching; a fiqh question needs HyDE + sub-query decomposition across both corpora. One-size-fits-all retrieval doesn't work.

Prompt engineering is a first-class engineering discipline. The grounded generation prompt is the single most important component. We spent significant time refining the system prompt — rules 1-7 in the generation prompt (answer only from context, cite every reference, admit when you don't know, don't issue fatwas) are the difference between a trustworthy system and a dangerous one.

What's next for Divinity AI

Evaluation benchmark. Build the planned 500-QA-pair evaluation dataset with expert-annotated ground truth, and integrate RAGAS metrics (faithfulness, answer relevancy, BERTScore) into a CI pipeline so every change is measured against retrieval and generation quality targets.

User accounts and history. Add optional user accounts to save query history, bookmark sources, and build personalized study collections.

Built With

• django