RAG Params Finder

A practical RAG sweep tool for comparing vector databases, embeddings, chunking, and retrieval methods in real-world setups.A free pre-eval tool before you run expensive evals on your RAG app!

Comment

Inspiration

This project inspired by a chain of projects (two to be precise), the quest to find the quality of data or dataset led to creating this project. It predecessor was a lighter version that runs in the browser (https://github.com/neomatrix369/pre-rag-explorer-dashboard).

What it does

It helps find the best parameters one needs to set our RAG applications with to get optimal retrievals. In essence its evaluating the retrieval model (i.e. vector database retrievals).

Point it at a data source and a set of persona-organized questions. It sweeps:

13 embedding models (Voyage AI hosted, plus local sentence-transformers / SIE open-source embeddings) 5 chunking methods (Fixed, Recursive, Token, Sentence, Semantic) 3 retrieval methods (Dense, Sparse, Hybrid)

Every combination gets scored against the questions, results are stored, and a dashboard surfaces the best-performing configuration with live phase indicators, elapsed/ETA timers, a Search Explorer, and results export — so you know which config actually works before you build the RAG app around it.

How we built it

It was built it in stages first was a prototype of a lighter version (https://github.com/neomatrix369/pre-rag-explorer-dashboard) and following that came this more robust version.

It was built with a single large prompt then split into smaller slices of a plan. Each plan was executed, and each plan delivered end-to-end.

With simple solutions -- support for one vector database, local embedding model, a couple of chunking methods and couple of retrieval methods.

A variety of coding agents were used and finally Codex!

Challenges we ran into

Lots of technical issues with keeping various services running and connecting to them correctly. Docker container failures.

Plenty of requirements -- all of them exciting, which ones to pick first and apply?

Coding agents i.e. Codex failing on us, running slow or missing out on spec definitions.

Parallelism vs. Bayesian search vs. product identity. Grid search sweeping every combination is the product's stated core feature (⭐ CORE FEATURE, "evidence across every combination, side by side"). Swapping it for sequential Bayesian sampling would change what the tool fundamentally is, so the call was to keep Bayesian opt-in and additive, deferred-not-rejected for future evolution, rather than a wholesale replacement.

Accomplishments that we're proud of

Addition of parallelisation and Bayesian search method to the project were big achievements making it faster and smarter.

What we learned

Moving slow and small steps at a time is the fastest way to move.

Coding agents can slow us down, so use them smartly and use low effort simple models to solve coding issues - they respond fast.

What's next for Rag Params Finder

We have a big backlog of goodies we want to add to the functionality i.e. support for additional vector databases, embeddings models, chunking methods and retrieval methods --- while keeping the existing functionalities work and preventing any form of regressions!

Lots to keep us busy and excited.

Built With

Share this project:

Updates