🚀 Inspiration
Traditional RAG systems are powerful, but they often follow a simple retrieve → generate pipeline. When something goes wrong, it can be difficult to understand why the system produced a particular answer, whether the retrieved context was actually sufficient, or where the workflow failed.
This inspired us to build AgentLens – Observable Agentic RAG, an Agentic RAG system designed not only to generate better answers, but also to make the entire AI workflow observable, traceable, and easier to debug.
💡 What We Built
AgentLens transforms a traditional RAG pipeline into an intelligent, iterative workflow using multiple specialized agents. The system starts with a user's question and passes it through a Planner Agent, Chain of Thought, Tree of Thought, Query Rewriter, Search Fanout, Retriever, Sufficient Context Agent, Feedback Query Generator, Iteration Manager, and Synthesis Agent.
Instead of retrieving information once and immediately generating an answer, the system evaluates whether the retrieved context is sufficient. If important information is missing, it automatically generates additional search queries, retrieves more relevant information, and repeats the process before producing the final grounded response.
The retrieval layer uses Sentence Transformers for semantic embeddings and FAISS for efficient vector similarity search, while Llama-based models accessed through Groq power the reasoning and generation stages.
🔍 Observability as a Core Feature
Our key differentiator is the end-to-end observability framework built around the Agentic RAG pipeline.
Every agent execution can be monitored through distributed traces containing information such as agent name, execution status, start time, end time, and latency. We also implemented LLM monitoring, error tracking, replay logging, session tracking, health monitoring, token usage and cost estimation, workflow visualization, and execution timelines.
This allows developers to answer questions that traditional RAG systems often leave unanswered:
- Which agent caused the failure?
- How long did each stage take?
- Was the retrieved context sufficient?
- Why did the system perform another retrieval iteration?
- Where did latency increase?
- What happened during a previous execution?
The goal is to make AI systems easier to understand and improve, rather than treating them as black boxes.
⚙️ How We Built It
The project was implemented in Python using a modular multi-agent architecture. Documents are processed into chunks, converted into embeddings using Sentence Transformers, and indexed using FAISS. The Planner and reasoning agents analyze the user's query, while the Query Rewriter and Search Fanout improve retrieval coverage.
After retrieval, the Sufficient Context Agent evaluates the available evidence. If the context is insufficient, the Feedback Query Generator creates targeted queries and the Iteration Manager initiates another retrieval cycle. Once sufficient evidence is available, the Synthesis Agent produces the final grounded response.
Alongside this pipeline, custom tracing and monitoring components capture the execution of each stage without tightly coupling observability logic with the individual agents.
🧩 Challenges We Faced
One of the biggest challenges was coordinating multiple agents while maintaining reliable structured outputs. LLMs can occasionally return malformed JSON, so we implemented JSON extraction and repair mechanisms.
We also had to handle retrieval edge cases such as invalid FAISS indices, API failures, and repeated retrieval iterations. Building the observability layer introduced another challenge: tracking every agent execution consistently without making the underlying AI logic difficult to maintain.
These challenges helped us understand that building an AI application is not only about selecting a powerful model. Reliability, monitoring, error handling, retrieval quality, and debugging are equally important.
📚 What We Learned
Through this project, we learned how to design and connect multiple AI agents into a complete workflow, implement semantic retrieval using embeddings and vector search, and build iterative context validation into a RAG system.
More importantly, we learned that observability is essential for production AI. An AI system should not only provide an answer; developers should also be able to understand how the system reached that answer and identify what happened when it failed.
🌟 Why It Matters
AgentLens combines agentic reasoning, adaptive retrieval, context self-evaluation, and production-oriented observability in a single system.
Our vision is to move from AI systems that simply generate answers to AI systems that are reliable, measurable, debuggable, and continuously improvable.
Don't just ask what the AI answered. Ask what happened while it was thinking.
Log in or sign up for Devpost to join the conversation.