Inspiration
Real-world knowledge—such as financial reports, medical records, and technical manuals—lives in long, visually rich PDFs where critical clues are scattered across text paragraphs, complex tables, charts, and page layouts. Existing RAG systems face a rigid dilemma: OCR-based text chunking loses layout and visual semantics, while whole-page visual retrieval floods Vision-Language Models (VLMs) with irrelevant regions and high token costs. Even worse, models often guess answers from parametric memory without retrieving the complete supporting evidence chain. We wanted to build a multimodal document RAG system that retrieves compact, verifiable, and complete evidence groups on demand.
What it does
Our system is an adaptive, hypergraph-guided Multimodal Document RAG framework for long-document question answering. Instead of returning a fixed Top-$k$ list of isolated chunks or full pages, it:
Parses visually rich PDFs into multi-granular units (pages, headings, paragraphs, tables, and figures). Organizes them into a Document Evidence Hypergraph, binding each chart or table with its caption, section header, and explanatory text as a self-contained evidence unit. Uses an adaptive controller at query time to progressively activate, expand, and crop only the relevant visual regions and textual context under a strict token budget, producing answers with traceable page-and-region citations.
How we built it
Multimodal Ingestion & Parsing: Used layout-aware PDF parsing (MinerU / PaddleOCR + PyMuPDF) to extract bounding boxes, structured tables, high-resolution region crops, and section hierarchies. Evidence Hypergraph Construction: Built structural hyperedges (page and section containment) and cross-modal contextual hyperedges (linking figures/tables with captions and surrounding text) without expensive LLM indexing calls. Hybrid Retrieval & Budgeted Controller: Combined late-interaction visual page retrieval (ColPali / ColQwen) for initial grounding with an agentic controller that iteratively expands evidence along hyperedges and renders high-resolution region crops + OCR text for the VLM reader (Qwen-VL).
Challenges we ran into
The Granularity-Noise Trade-off: Feeding full page images diluted the VLM’s attention on small table cells, while feeding only cropped regions lost global page context. Combining low-cost page summaries with high-resolution local crops + OCR solved this. Shortcut Reasoning & Text Bias: VLMs tend to over-rely on text passages or guess answers after finding only the first hop of a multi-hop query. Designing explicit evidence-coverage verification and non-redundant query checks was key to forcing complete evidence retrieval.
Accomplishments that we’re proud of
Built an end-to-end pipeline that preserves both macro document structure and fine-grained visual details (charts, multi-column tables) without expensive full-corpus LLM graph extraction. Replaced static Top-$k$ retrieval with budget-aware, progressive evidence selection—reducing redundant visual tokens while improving multi-hop and cross-page QA accuracy. Made every generated answer auditable by linking claims back to exact document pages and bounding-box regions.
What we learned
Retrieval quality alone does not guarantee answer quality in multimodal RAG. How evidence is grouped and delivered to the VLM generator—preserving the n-ary binding between a visual element, its caption, and its table headers—matters just as much as finding the right page.
What’s next for multimodal document RAG
Reinforcement Learning for Evidence Control: Training the evidence-selection controller with first-match evidence-coverage process rewards (ECPR) so smaller models learn to retrieve complete evidence chains in fewer steps. Federated & Privacy-Preserving Progressive Disclosure: Extending our hypergraph framework to cross-institutional settings (e.g., hospitals and banks) where evidence is disclosed progressively—from de-identified summaries to localized region crops—while preventing quasi-identifier mosaic leakage.
Built With
- llm
- multimodal
- rag
