The problem
Legal and civic-tech tools increasingly answer real questions for real people through LLMs. But an LLM answer is only as trustworthy as the retrieval behind it - and most teams ship RAG systems with no quantified evidence that the right source text is actually being found. For anything touching rights, compliance or public assistance, "it seems to work" is not an acceptable governance standard.
What it does
rag-eval-system is an evaluation harness that audits whether a RAG pipeline's answers are genuinely grounded in their source documents. It runs a hand-labeled golden dataset (35 questions over a 10-document internal policy handbook - leave, expenses, WFH, performance, IT security, code of conduct and more) through every combination of chunking strategy and retrieval method, and reports Hit Rate, MRR and NDCG for each. The results are CI-gated: if retrieval quality regresses, the build fails.
Current benchmark results (@k=5): BM25 sparse retrieval hits 1.000 hit rate / 0.943 MRR on fixed chunking, beating dense, hybrid and reranked variants - a counter-intuitive, reproducible finding about when lexical retrieval wins. The harness also documents where an intervention (lexical reranking) made things worse, because an audit that only shows improving numbers is not a credible audit.
How it's built
Python + FastAPI service, Docker-packaged, with full CI. Retrieval: BM25 (sparse), TF-IDF+SVD (dense), RRF fusion hybrid, pluggable reranker (lexical now, cross-encoder swap point built in). Generation: Claude with an extractive fallback so the pipeline runs with no API key. Evaluation: python -m src.eval.run_eval --k 5 regenerates the full metrics table.
What I learned
Governance-grade evidence changes design decisions. The instinct to default to hybrid retrieval doesn't survive measurement on a lexically precise corpus - policy and legal documents share exact vocabulary between question and clause, so BM25 alone wins. The eval harness is what makes that visible, and it's exactly the kind of auditing step that legal-tech and civic-tech AI needs before anyone should rely on its answers.