RAG Eval Sidekick

An AI-powered evaluator: stop eyeballing RAG outputs - GPT-5.6 scores, diagnoses, and auto-tunes your pipeline automatically.

Comment

Inspiration

Every team building a RAG (retrieval-augmented generation) system hits the same wall: once it's running, how do you actually know if it's good? Most teams eyeball a handful of answers and move on. There's no fast way to tell whether a bad answer came from retrieval pulling the wrong chunks, or generation hallucinating something the context never said. I wanted to build the tool I'd actually want on my own team — something that scores RAG outputs systematically and tells you why something failed, not just that it did.

What it does

RAG Eval Sidekick takes any RAG pipeline's output — a question, the chunks it retrieved, and the generated answer — and scores it on three dimensions using GPT-5.6 as an LLM judge:

  • Faithfulness — are the answer's claims actually supported by the retrieved chunks?
  • Answer relevance — does the answer address the question that was asked?
  • Context precision — did retrieval pull chunks that were actually relevant?

Failing triples get a plain-English diagnosis distinguishing retrieval problems from generation problems. Beyond scoring, it also:

  • Auto-tunes retrieval settings by sweeping chunk-size and top-k combinations and recommending the best configuration
  • Tracks regressions by saving labeled evaluation runs and comparing them over time
  • Generates test data on demand — upload any document, and GPT-5.6 suggests evaluation questions, including ones specifically designed to stress-test retrieval

It works with any RAG system's outputs, regardless of what vector database, embedding model, or framework built it — I don't require any integration, just paste in your triples or hit the API.

How I built it

I built this iteratively with Codex, one component at a time rather than all at once. Each piece was scaffolded, tested against real data, and reviewed before moving to the next: the mini RAG pipeline first, then the scorer, then the diagnoser, then the FastAPI backend and Streamlit frontend. Once the core worked end-to-end, I expanded it with three more features — bring-your-own- data support, the auto-tuner, and regression tracking — followed by a one-command Docker setup so anyone can run it without wrestling with Python environments.

The source corpus for testing comes from Wikipedia articles on Transformers, attention mechanisms, and BERT, chosen specifically because they're technical enough that retrieval actually has real work to do — not just three paragraphs where "top-3 of five chunks" trivially always wins.

Challenges I ran into

The faithfulness rubric was initially too lenient. Early on, I tested the scorer against a deliberately fabricated example — an answer that included a confident but completely made-up statistic. It scored 0.75, the same range as a genuinely good answer with only a minor omission. I rewrote the rubric to explicitly treat any specific, falsifiable fabricated claim as a hard cap on the score, regardless of how small or incidental it seemed relative to the rest of the answer. Re-tested against the same example, faithfulness dropped to 0.4 — the fix actually worked.

The evaluator caught a failure I didn't plan. While testing what I expected to be a "clean" example, the scorer flagged a genuine faithfulness issue: the answer stated facts about BERT's pre-training tasks that weren't actually present in the specific chunks retrieved for that question, even though they're true facts about BERT in general. That was an unplanned, organic catch — real evidence the evaluation logic generalizes rather than just detecting the failures I deliberately engineered.

Building the auto-tuner efficiently. A naive sweep across chunk sizes and top-k values would mean re-embedding the entire corpus for every configuration. Codex optimized this by batching all question embeddings once and reusing chunk embeddings across every top-k value tested for a given chunk size — cutting redundant API calls significantly.

Accomplishments that I'm proud of

Catching a real bug in my own evaluation logic. The faithfulness rubric initially scored a confidently fabricated statistic the same as a genuinely minor omission (0.75 either way). I caught it, rewrote the rubric to explicitly cap scores for specific falsifiable claims, and verified the fix dropped that same example to 0.4 — proof the fix actually worked, not just a guess that it did.

An evaluation harness that catches failures I didn't plan. During testing, the scorer flagged a genuine, undirected faithfulness issue in what I expected to be a clean example — the retrieved chunks didn't actually support a specific claim in the answer, even though the claim was true in general. That's real evidence the tool generalizes, not just detects the failure modes I deliberately engineered.

A genuinely useful auto-tuner, not just a scorer. Most eval tools stop at "here's your score." Mine sweeps chunk-size and top-k combinations, scores each one, and tells you which configuration is actually best and by how much — turning the tool from a report card into something that helps you fix the problem.

Zero-friction setup. A single docker compose up --build gets both services running together, backend and frontend talking over Docker's internal network, with persistent SQLite history that survives container rebuilds. Anyone can clone the repo and be evaluating RAG outputs in under a minute.

A tool that works with any RAG pipeline, not just my own. Sidekick doesn't care what vector database, embedding model, or framework built the outputs it's evaluating — paste in any triple, or call the API directly, and it works.

What I learned

That LLM-as-judge scoring is only as good as the rubric behind it — a vague rubric produces vague, overly lenient scores, and getting it right takes real iteration against adversarial test cases, not just a first draft. I also learned that a small, deliberately-controlled test set (with known good and known bad examples I'd hand-verified) is more useful for validating an eval tool than a huge dataset I couldn't manually check — scale doesn't substitute for ground truth I actually trust.

What's next for RAG Eval Sidekick

Confidence-aware scoring. Right now every score is presented with equal certainty. A natural next step is surfacing when the LLM judge itself is uncertain — borderline cases where the model's confidence is genuinely low — so I know which scores to trust at face value and which deserve a closer manual look.

Batch CSV/JSONL upload. Today I evaluate one triple or a small JSON list at a time. Supporting bulk upload of hundreds or thousands of production RAG logs would make this usable at real team scale, not just for spot-checks.

AI-suggested fixes, not just diagnosis. Beyond explaining why an answer failed, the natural next step is having GPT-5.6 propose a corrected version of the answer using only the actually-retrieved chunks — turning Sidekick from a diagnostic tool into something that actively helps fix broken RAG outputs.

Deeper retrieval strategy comparisons. The auto-tuner currently sweeps chunk size and top-k. Extending it to compare different embedding models, chunking strategies (semantic vs. fixed-size), and reranking approaches would make it a genuinely comprehensive RAG optimization tool.

Team-level dashboards. Regression tracking currently lives per-machine in SQLite. A shared, hosted version where a whole team could see evaluation history and trends over time — not just one person locally — would make this useful for ongoing production monitoring, not just individual testing.

Built With

Share this project:

Updates