Document Q&A Agent

Our Document Q&A Agent uses a local vector store and RAG pipeline to instantly parse documents and answer natural language queries about any clause, combining speed, accuracy, and privacy.

Comment

Inspiration

I had already written basic RAG code by hand and built agents that reason over data, but I had never had a single tool that answers questions from one document and refuses to guess. Contracts, laws, syllabi and textbooks all share the same problem. You need one specific clause, and a general chatbot will confidently invent it. I wanted an agent whose most important behavior is saying "not found in this document" when the PDF does not contain the answer.

I also wanted to learn how to work with an AI coding agent: give clear requirements, test the result, and tune it until it meets the need.

What it does

Document Q&A Agent lets you upload a PDF (up to 40 MB) and ask questions that are answered only from that document.

  • Upload a PDF. It is read, split into chunks, embedded, and stored in a local Chroma vector database.
  • Ask in chat. The app retrieves the 7 most similar chunks, and Gemini writes a connected answer using only those chunks.
  • Ask something the PDF does not cover (e.g. "Who won the 2018 World Cup?") and you get exactly: not found in this document.
  • Summary-style questions ("give me a summary") use chunks sampled evenly across the whole document.
  • Conversations are saved, listed in a sidebar (first question and time), and can be continued or deleted.
  • Replacing or deleting the PDF makes older conversations read-only, since they belong to a different document.
  • Structured documents are split at their headings ("Article 32", "Clause 7", "Unit 2", "Section 4.2"), so "what does clause 7 say" works.

How we built it

  • Frontend: React site (port 4000) with a sidebar, drag-and-drop upload, and a chat with waiting indicators and clear error messages.
  • Backend: FastAPI (port 8000), organised into:
    • chunking/ reads pages with pypdf (in parallel for large PDFs) and splits with a heading-aware RecursiveCharacterTextSplitter. Each chunk is labelled with its heading.
    • rag/vectorstore.py embeds chunks locally with bge-small-en-v1.5 and stores them in Chroma.
    • rag/chat.py runs top-7 similarity retrieval, a strict grounding prompt, and Gemini. It tries several Gemini models in order as fallbacks.
    • sessions.py saves conversations as JSON files.
  • Measuring instead of guessing: tests/eval_retrieval.py takes a list of questions with known answers and checks whether the answer text is in the top 7 chunks, with no Gemini calls and no quota. Retrieval quality is simply

$$\text{hit rate} = \frac{\text{questions whose answer is in the top-7 chunks}}{\text{total questions}}$$

I ran it before and after every change.

Challenges we ran into

  • "Not found" for real questions. "What does article 32 say?" failed on a 402-page Constitution PDF. Embeddings alone could not match article numbers. I tested four chunkings and three embedding models on the same question list. Re-chunking alone did not help (4/14). Heading-aware chunks plus bge-small reached 18/20.
  • Telling two causes of "not found" apart. "Article 457" correctly returned not found because the PDF ends at Article 395. "Article 32" was a real retrieval failure. Measuring was the only way to separate them.
  • Speed. Semantic chunking took over 4 minutes on a 402-page PDF. Switching to a recursive splitter, preloading the model and parallelising page reads cut this to about 35–90 seconds.
  • Over-fitting to one document. My first chunker was tuned to the Constitution. I rewrote it to recognise numbered and keyword headings in any document, with a plain-splitter fallback. On generated PDFs in five styles (contract, syllabus, textbook, notes, report), heading-aware chunking scored 57/60 vs 48/60 for plain chunking.
  • Free-tier limits. Gemini's free tier allows about 20 requests/day per model, so the chat falls back across several models and fails fast on an invalid key.
  • Misleading errors. A stopped API was being shown as "this document is invalid". It now says "cannot reach the server".

Accomplishments that we're proud of

  • An agent that says "not found in this document" instead of hallucinating. In the build-time test, 5/5 covered questions were answered correctly and 5/5 uncovered questions returned exactly "not found in this document".
  • A measured retrieval pipeline, not guesswork: 19/22 on the Constitution and 57/60 across five different document styles.
  • Embeddings and vector storage run fully locally. Only the final answer step calls Gemini.
  • A 400-page book is ready to question in about 90 seconds, and a 20–50 page PDF in about 10 seconds.
  • A complete, polished flow: upload, ask, save, continue, delete, read-only history.

What we learned

  • Measure before you change. A fixed question list showed which "improvements" actually helped and which did not.
  • Not every miss is a bug. A correct refusal and a retrieval failure look the same in the UI.
  • Chunk boundaries matter more than model choice. Labelling chunks with their headings changed results more than swapping retrievers.
  • How to work with an AI agent: state requirements precisely, verify each step, decide the trade-offs yourself, and iterate.

What's next for Document Q&A Agent

  • Better lookup of near-identical numbered sections (e.g. topic 2.3 vs 2.4) and bare number-only questions like "article 17".
  • A query-rewrite step so follow-ups such as "Explain that more simply" work.
  • OCR support for scanned PDFs, which are currently reported as invalid.
  • Multiple documents, plus citations that show the exact page or section behind each answer.
  • Pasted text as an input alongside PDF upload.

Built With

Share this project:

Updates

Submission history