VerdictOS

Inspiration

A typical mid-market M&A deal involves 500 to 2,000 documents, four to eight specialist teams, 60 to 90 days of review, and $ \$2\text{M} $ to $ \$5\text{M} $ in advisory fees. Despite that investment, 70–90% of M&A transactions fail to deliver expected value, and undiscovered risk during due diligence is consistently named as a primary cause.

The failure isn't a lack of effort it's structural. No team reads the entire document bundle. Specialists work in silos, so a founder's IP assignment in one contract and the same founder's departure clause in a completely different agreement never get connected. Conclusions go unchallenged. And the final output is a prose report that can't be traced back to the actual clause it came from.

We also looked at existing AI document tools and found the same pattern everywhere: an LLM wrapped around a vector database, retrieving the semantically closest passage instead of the exact one, and confidently citing clauses that don't say what the model claims they say. In legal and financial due diligence, "approximately right" is not good enough a hallucinated citation in an M&A report is a liability, not a convenience.

That gap between what due diligence should be (exhaustive, adversarial, verifiable) and what it actually is (sampled, siloed, and unverifiable) is what VerdictOS sets out to close.

What it does

VerdictOS reads every document in a deal bundle, analyzes it from sixteen specialist perspectives in parallel, then makes its own findings argue against each other before any of them reach a human.

The pipeline runs in seven phases:

Ingest & Index documents are parsed, chunked with full section-and-cross-reference awareness, tagged by clause type, and indexed with BM25 (no vector embeddings anywhere). A knowledge graph links entities people, companies, defined terms across every document in the bundle.
Smart Dispatch & Analysis a planner agent activates only the specialist agents relevant to the documents present (out of 16: IP, Litigation, Tax, Finance, HR, Cyber, ESG, and more), and each one queries the index using its own domain-scoped retrieval strategy.
Findings Aggregation every finding is schema-validated, cross-document risks are merged via the knowledge graph, and findings are routed into 8 strategic debate dimensions but only if there's enough evidence to debate.
Adversarial Debate 6 personas (Proponent, Critic, Devil's Advocate, Valuation Skeptic, Integration Realist, Regulator's Eye) argue every finding for up to 3 rounds, with 5 independent reliability gates checking citations, confidence, and contradictions along the way.
Consensus Mapping a debated point is classified as Settled when $ \geq \frac{4}{6} $ personas agree with verified evidence, Contested when the split has evidence on both sides, and Unresolved when no source evidence exists all by counting persona stances, with zero extra LLM calls.
Judge Synthesis a judge agent reads only the Contested and Unresolved points (Settled findings are excluded entirely) and produces a confidence-weighted verdict.
Verdict + Human Loop the final output is a Go/No-Go brief, a human escalation list, and an evidence gap report, all traceable to verified source clauses. Humans can resolve escalations, dispute findings, or upload missing documents to trigger a targeted re-analysis never a full re-run.

Every claim in the final verdict can be traced back to an exact passage in the original documents. Nothing is asserted that wasn't verified.

How we built it

VerdictOS is built entirely on open-source infrastructure:

LLM layer: Llama 3.1 (70B for reasoning and debate, 8B for lightweight entity disambiguation) served via Ollama
Retrieval: Whoosh/Elasticsearch BM25 with section-aware metadata and clause-type filtering deliberately no vector database
Knowledge graph: spaCy NER + NetworkX, with a three-tier entity resolution pipeline: rule-based matching resolves $ \approx 80\% $ of entities, RapidFuzz fuzzy matching adds $ \approx 15\% $, and an LLM call handles the remaining $ \approx 5\% $ of genuinely ambiguous cases
Schema contracts: Pydantic v2 enforced at every agent boundary, turning the pipeline into a deterministic data flow rather than a chain of free-text prompts
Orchestration: Python asyncio with a semaphore bounding concurrency at $ N = 40 $ all 16 specialist agents and all active debate dimensions run in parallel
API & frontend: FastAPI with WebSocket streaming for live pipeline visualization, Next.js + Tailwind for the UI
Storage: PostgreSQL with append-only audit tables human overrides are additive, never destructive to the original AI output
Deployment: Google Cloud Cloud Run for the API and frontend, a preemptible GPU VM (g2-standard-4, NVIDIA L4) running Ollama, Elasticsearch, and Celery workers, We treated the system architecture itself as a living document every design decision was stress-tested against a "what actually breaks in production" lens before it was implemented.

Challenges we ran into

BM25's blind spot for cross-references. A clause that says "the entity assumes all exposures listed in Section 4" is invisible to keyword search if you're searching for "exposure" in a different section. We fixed this with section-aware chunking every chunk stores its section ID, the sections it references, and the contract's own defined terms, turning "Section 4" into a direct lookup instead of a failed keyword match.

One synonym dictionary, sixteen domains. Our first synonym expansion mapped "exposure" to "liability" globally which meant the Finance agent's queries also pulled in environmental contamination clauses, cyber breach clauses, and FX hedging clauses, all of which use the word "exposure" in completely unrelated ways. The fix was per-agent, domain-scoped synonym dictionaries combined with a compound query that filters by clause type, so the Finance agent hunting tax liabilities never sees an FX hedging clause regardless of keyword overlap.

A single dropped persona could fake a consensus. In multi-round debate, if the Valuation Skeptic's JSON output failed schema validation twice, it simply disappeared from that round and the remaining personas could agree just enough to mark a dimension "Settled," even though the opposing voice never spoke. We added a dropout-aware rule: a dimension can never be marked Settled if Critic, Valuation Skeptic, or Devil's Advocate dropped out it's automatically reclassified as Contested instead.

Bounding debate concurrency without losing parallelism. The theoretical maximum for one debate round is

$$ N_{\text{calls}} = 6 \text{ personas} \times 8 \text{ dimensions} = 48 $$

per round, or $ 144 $ across all 3 rounds which looks like it should take $ 144 \times t_{\text{call}} $ sequentially. In practice, all dimensions run concurrently and the semaphore caps simultaneous calls at $ N=40 $, so the wall-clock time per round is

$$ T_{\text{round}} \approx \left\lceil \frac{48}{40} \right\rceil \times t_{\text{call}} \approx 2\, t_{\text{call}} $$ roughly two LLM call durations per round, not 48.

Keeping the event loop from freezing. Building the knowledge graph with NetworkX and running spaCy NER over hundreds of documents is CPU-bound, synchronous work. Running it inside the async pipeline would block every agent worker for the duration. We solved this by making document ingestion a strict synchronous pre-flight phase that completes entirely before asyncio.run() is ever called the event loop only ever reads a finished index and graph, never builds one.

Fitting a 70B-class workload into a $ \$300 $ GPU budget. Llama 3.1 70B doesn't fit on a single L4 GPU (24GB VRAM) even at 4-bit quantization, which requires roughly $ 43\text{GB} $. We run Llama 3.1 8B as the default model on a preemptible g2-standard-4 instance with a scheduled start/stop window, and reserve a short-lived A100 spot instance for any demo that specifically needs the larger model.

Accomplishments that we're proud of

A retrieval architecture with zero vector embeddings that is fully deterministic and fully auditable every citation either exists exactly as claimed, or the system says so
A debate engine that guarantees termination by architectural constraint (a hard 3-round cap), not by hoping the model behaves
A human-in-the-loop design where AI output is never overwritten every human decision, dispute, or override is an additive, timestamped, attributed layer on top of the original analysis
An entire system built on open-source models and infrastructure, deployable for under $ \$100 $/month
A context compression scheme between debate rounds that achieves $ \approx 90\% $ token reduction while preserving every persona's sharpest argument verbatim

What we learned

We learned that BM25 is widely misunderstood including by us, initially. The full scoring function for a term $ t $ in a document $ d $ is

$$ \text{score}(t, d) = \text{IDF}(t) \times \frac{\text{TF}(t,d) \times (k_1+1)}{\text{TF}(t,d) + k_1 \times \left(1 - b + b \times \dfrac{|d|}{\text{avgdl}}\right)} $$

The length-normalization term in the denominator controlled by $ b $ actually penalizes long documents. A one-paragraph amendment buried in a 150-page contract scores higher per term match than boilerplate repeated across the whole document, because $ |d| $ is small relative to $ \text{avgdl} $. The real weakness of BM25 isn't length bias; it's structural blindness to cross-references, which is a solvable metadata problem, not a fundamental limitation of the algorithm.

We also learned that a knowledge graph built independently of agent retrieval straight from raw document text in a pre-flight pass is far more robust than one built incrementally from agent outputs, because it can't be "starved" by an agent that fails to retrieve the right chunk.

And we learned that the most trustworthy thing an AI system can do isn't claim to be right it's make every claim checkable, flag what it couldn't verify, and get out of the way when a human needs to make the final call.

What's next for VerdictOS

Schema-segmented debate calls and a random adversarial challenger sampling $ \approx 15\% $ of Settled findings to catch model-level blind spots
Fine-tuned clause classifiers (CUAD-based) to push clause-type accuracy from $ \approx 80\% $ to $ \approx 95\% $
Portfolio-level monitoring mode for private equity firms tracking covenant compliance across multiple holdings
Vertical-specific synonym and clause overlays for healthcare, energy, and real estate M&A