RAGReader

Inspiration

We realized that not every document benefits from the same Retrieval-Augmented Generation (RAG) strategy. Different document types often require different retrieval methods to achieve the best results. This inspired us to build a system that automatically evaluates multiple RAG approaches and identifies the most effective retrieval pipeline for a given dataset.

What it does

RAGReader is an advancement over traditional dense-only RAG systems. Instead of relying on a single retrieval approach, it automatically evaluates Dense Retrieval, Sparse Retrieval, Hybrid Search, and Reranking methods. The system compares their performance using retrieval metrics such as Precision, Recall, and F1 Score, as well as generation quality metrics including ROUGE and RAGAs. This helps users discover the most effective retrieval strategy for their documents.

How we built it

• LLM Orchestration: OpenAI GPT, Anthropic Claude, Google Gemini
• Embedding Models: OpenAI Embeddings, MiniLM
• Retrieval Methods: Dense Retrieval, Sparse Retrieval, Hybrid Search, Rerankers (e.g., Cohere/BGE)
• Framework: LangChain
• Frontend: React, Vite, Tailwind CSS
• Backend: Django REST Framework for AI and RAG APIs
• Infrastructure: Celery, Docker, and WebSockets for asynchronous processing and real-time updates

Challenges we ran into

Building a scalable backend was one of our biggest challenges. We designed the system using Django REST Framework, Celery, and Docker to support asynchronous document ingestion, embedding generation, retrieval, reranking, and evaluation workflows. Managing long-running AI tasks while maintaining a responsive user experience required careful separation between API responses and background processing.

Accomplishments that we're proud of

• Successfully created an intuitive user experience for evaluating multiple RAG strategies.
• Implemented a comprehensive evaluation framework covering both retrieval and generation quality.
• Built a scalable architecture capable of handling complex document processing workflows.
• Gained a deeper understanding of RAG evaluation methodologies and retrieval optimization.

What we learned

Throughout the project, we learned:

• How to evaluate RAG systems using metrics such as Precision, Recall, F1, ROUGE, and RAGAs.
• The importance of separating synchronous API endpoints from asynchronous background jobs.
• How to use Celery effectively for large-scale document processing.
• Implementing real-time communication through WebSockets.
• Designing scalable AI application architectures.

What's next for RAGReader

• Improve platform stability and reliability.
• Add support for more retrieval and reranking techniques.
• Automate retrieval strategy recommendations based on document characteristics.
• Enhance evaluation visualizations and reporting.
• Optimize performance for larger document collections and enterprise-scale deployments.

Built With

• django
• openai
• openrouter
• rag
• react