Self Improving RAG via Hyperparameter Optimization

Inspiration

We were inspired by the tedious reality of building RAG applications: developers spend endless hours manually tuning "magic numbers." Should chunk_size be 500 or 1000? Is top_k=5 enough? Does MMR retrieval actually help?

Usually, these decisions are made by gut feeling or trial and error. We wanted to utilize the new concept of multi-agent systems to solve this problem.

What it does

Its an autonomous, self-improving RAG pipeline. It doesn't just answer questions; it continuously experiments on itself to get better at answering them.

Executes a RAG Pipeline: It ingests documents, chunks them, embeds them, and retrieves answers using an initial configuration.
Self-Evaluates: A dedicated "Evaluator" agent scores the answers against a golden dataset (ground truth) on metrics like Relevance, Completeness, and Accuracy.
Optimizes Hyperparameters: An "Optimizer" agent analyzes the history of scores and configuration changes. It "reasons" about cause and effect (e.g., "Increasing chunk size improved completeness but hurt relevance, let's try a paragraph-based strategy next") and suggests a new configuration.
Rebuilds & Repeats: The system automatically rebuilds its vector index if necessary and runs the next iteration, effectively "learning" the optimal parameters for the specific dataset.

How we built it

We architected the system as a closed feedback loop with three specialized Gemini agents:

The Generator (Gemini model): A flexible RAG engine capable of hot-swapping strategies. It supports dynamic chunking (fixed, sentence, paragraph), embedding models , and retrieval logic (Similarity vs. MMR) on the fly using ChromaDB.
The Evaluator (Gemini model): Acts as the judge. It provides structured feedback and numerical scores ($0.0 - 1.0$) for every answer.
The Optimizer (Gemini model): The brain. It views the experiment history as a trajectory and predicts the next best hyperparameter set ($\theta$) to maximize the utility function.

We built the backend in Python, using Google GenAI SDKs for the models and Plotly for visualizing the optimization curve.

Challenges we ran into

The "Oscillation" Problem: The optimizer would rapidly improve performance by 80% in just 3 iterations, but then get stuck "thrashing" between configurations as it tried to overfit to specific hard questions.
State Management: Persisting state in a self-modifying system is hard. Changing the chunking strategy requires completely rebuilding the vector database. We had to implement smart caching to ensure we didn't waste time re-indexing the same configurations.

Accomplishments that we're proud of

Autonomous Improvement: We achieved a jump from a baseline score of ~80% in just 3 iterations without human intervention.
Self-Correction: The system successfully identified that "sentence-based chunking" was hurting performance and autonomously reverted to "fixed-size chunking" in subsequent iterations.
Complex Hyperparameter Space: We successfully tuned 10 different interacting parameters simultaneously, including discrete choices (retrieval strategy) and continuous values (temperature, threshold).

What we learned

Evaluator Alignment is Key: An automated optimization loop is only as good as its judge. If the Evaluator is too lenient, the Optimizer learns to exploit it rather than actually improving answer quality.
Latency vs. Quality: We discovered that sophisticated retrieval strategies like MMR (Maximal Marginal Relevance) significantly improved answer diversity but came with a latency cost—a trade-off the optimizer had to balance.
LLMs as Optimizers: Large Language Models are surprisingly good at "reasoning" through hyperparameter spaces, acting almost like a semantic Bayesian optimizer.

What's next for Self Improving RAG via Hyperparameter Optimization

Frontend Visualization: We are finalizing a Streamlit dashboard to watch the "evolution" of the system in real-time, plotting the trajectory of scores against parameter changes.
Larger Evaluation Sets: To stop the oscillation, we plan to scale the golden dataset from 10 to 50+ questions to smooth out the noise in the reward signal.
Cost-Aware Optimization: We want to add "cost" and "latency" as negative terms in the optimizer's utility function, so it finds the most efficient pipeline, not just the most accurate one.