Insight verifier

Inspiration

We’ve all seen the trend: everyone is rushing to slap LLMs on top of their documents. “Ask your PDF,” “Chat with your data.” It sounds great until you realize these systems answer with total confidence, whether or not the answer is actually in the source. A single hallucinated number in a financial summary or a misquoted clause in a contract isn’t just an error—it’s expensive. And worse, you usually can’t tell which parts of the answer to trust and which are made up. We kept hitting the same wall: The model gave me an answer, but where did it come from? Is it even true? So we flipped the script. Instead of just generating an answer, we generate the answer and trace every single claim back to its exact evidence in the source. Think of it as reverse-mapping insight back to origin. An answer shouldn’t be a black box—it should come with receipts.

What it does

Insight Verifier takes a source document (PDF, DOCX, TXT, or CSV) and a question–answer pair, then verifies the answer claim by claim. Here’s how it works:

Breaks it down: It splits the answer into individual, atomic claims.
Finds the proof: For each claim, it retrieves the most relevant passage, table, or exact cell from the source.
Checks the facts: It judges whether each claim is supported, contradicted, or unsupported by that evidence.
Does the math: If a claim involves numbers (like “gross profit was $4.2M”), it doesn’t just look for the text—it recomputes the figure from the underlying cells, since derived numbers rarely appear verbatim.
Proves the negative: For claims like “the contract doesn’t mention indemnity,” it scans the entire document to confirm absence.

The result? A per-claim verdict with the exact source location (e.g., Page 4, or Table 1, Row 4, Column 2), the quoted evidence, and an overall Groundedness Score. We calculate groundedness (GG G) as the fraction of claims that are fully supported: G=1n∑i=1n1[ci is supported by the source]G = \frac{1}{n}\sum_{i=1}^{n} \mathbb{1}\left[c_i \text{ is supported by the source}\right] G=n1∑i=1n1[ci is supported by the source] If every claim traces cleanly to evidence, G=1G = 1 G=1. Hallucinations pull that score down. The final output is a defensible, exportable audit trail (JSON or CSV) that maps every claim to its proof.

How we built it

Ingestion We parsed every format into a common chunk representation. We used pypdf for PDFs and python-docx for DOCX files (including tables). Text and CSV files used built-in parsers. Crucially, table cells became “first-class” chunks, carrying their row/column coordinates and headers. This makes the evidence self-describing: instead of just “4200000,” we know it’s revenue: fy2024: 4200000. Retrieval We chunked and embedded the source using sentence-transformers (specifically all-MiniLM-L6-v2, which runs fully locally and privately) into an in-memory FAISS index. But embeddings are notoriously bad at numbers, so we used a hybrid approach: cosine similarity re-ranked with lexical exact-token scoring, giving digits double weight. Decomposition + Verification We used Groq (running llama-3.3-70b-versatile) via the OpenAI SDK in JSON mode. One call splits the answer into atomic claims. Then, one call per claim audits the retrieved evidence, returning a verdict, cited evidence indices, and an explanation. If the claim is derived, it recomputes the figures from the cells. Frontend Built with Next.js 16 and Tailwind v4. It’s a dark, polished single-page flow featuring drag-and-drop upload, a Q&A form, a live groundedness gauge, and per-claim verdict cards. Each card shows the quoted evidence, source locations, and any recomputed figures. We also added an animated verification pipeline and one-click JSON/CSV export. Backend Powered by FastAPI and Python 3.11, managed with uv. In development, the client proxies /api/* to the server. We wrote 12 pytest tests covering parsers, claim heuristics, JSON salvage, and the full end-to-end pipeline with a stubbed LLM.

Challenges we ran into

Verifying the Verifier The naive fix—asking a second LLM “is this supported?”—just moves the hallucination up a level. A wrong “verified ✅” stamp is more dangerous than no stamp because people stop double-checking. So, we always surface the actual evidence span for human confirmation. The judge is even forbidden from citing evidence that wasn’t explicitly provided. Numbers Don’t String-Match Verifying “revenue was $4.2M” isn’t a text search problem. That figure is often derived from a table and never appears verbatim. Checking numeric claims means recomputing them from the underlying cells. This was the hardest but most valuable part. We also found our embedding model ranked unrelated passages above chunks that literally contained “net income reached 950000”—fixed by adding hybrid lexical scoring. Proving a Negative Claims like “the contract doesn’t mention indemnity” can’t be confirmed by retrieving a single passage. You have to establish absence across the whole document, which is exactly what our full-document scan does. Models Echo Their Prompts Groq’s strict JSON mode hard-fails when the model quotes evidence back before emitting JSON. We hit this on a real query. Our fix? Try strict mode first, then fall back to a plain-mode retry that salvages the JSON with a balanced-brace extractor. We also fixed token-budget overflow (by adaptively doubling limits) and evidence truncation that was silently cutting off figures. Library Landmines python-docx returns the same cell element for multiple grid positions in merged layouts. To fix this, we made evidence attribution position-based and deterministic. Latency vs. Rigor Generate-then-verify roughly doubles the model calls. To keep it fast, we kept verification single-pass per claim and made the judge budget-efficient while staying strict.

What we learned

Grounding isn’t a wrapper. You can’t just bolt it on generically. The hard 20%—numbers, tables, images, absence—is exactly where verification matters most and where naive citation fails.
Receipts are the product. People don’t just want a yes/no verdict; they want a defensible, exportable record mapping every claim to its evidence. The verdict is just a summary; the receipts are the real value.
Inspectability is key. A verifier you can’t inspect is no better than the model you were trying to check. Every claim ships with its evidence and its recomputation so the human always has the last word.
Local + Fast is the sweet spot. Using local embeddings plus a fast hosted LLM keeps documents private, costs near zero, and inference speed high.

What's next

OCR for scanned PDFs and images: The biggest missing ingestion path.
PDF table extraction: So numeric claims get cell-level evidence in PDFs, too.
Multi-document retrieval: Verify against a whole corpus, not just a single file.
Batch verification: Audit an entire document’s worth of Q&A pairs at once.
In-document highlighting: Click a claim and see the span highlighted in the original file view.
Confidence calibration: Per-verdict confidence scores, so borderline judgments are flagged for human review rather than asserted.