Inspiration
Construction procurement runs on documents that are supposed to agree — and routinely don't. The purchase order commits to 500 tons of A992 steel at $1,850/ton; the invoice bills 520 tons at $1,910; the submittal quietly swaps in a lower steel grade; the delivery slips past the date the project actually needs it. On a project moving billions in materials, each of these is real money, and today a human catches them by manually cross-checking PDFs — too late, if at all. We wanted to give every procurement team an expert analyst that never sleeps and never misses a line item.
What it does
Plumbline reads the procurement paper trail — POs, invoices, submittals, spec sheets — reconciles them against each other, and flags the costly mismatches before the invoice gets paid. It runs two engines over one report:
Structured 3-way match (deterministic). Every invoice line is checked against the PO it bills and the submittal that certifies it — quantity, unit price, extended total, SKU — plus duplicate invoices, invoices against un-approved POs, and delivery ETAs past need-by. It's robust to real-world SKU formatting noise ("STL-W14" ≡ "STL W14" ≡ "stlw14").
Semantic spec-conformance (NLI). A CPU natural-language-inference model reads the submittal's conformance statement against the PO's governing spec clause and catches grade/rating downgrades that aren't a field comparison but a language judgment.
Every finding is quantified in dollars of exposure and ranked, so the report opens with the number a construction CFO cares about: "$11.3M of billing errors caught before payment."
And because a project accountant needs to act on it with money on the line, an independent verifier (MiniCheck-770M) checks every claim the system makes against the source documents — a model separate from the generator, so it can't rubber-stamp its own hallucinations.
How we built it
Plumbline is built on an on-prem, fully open-source agentic GraphRAG stack: a local Qwen model via Ollama, a Neo4j vendor–SKU–spec knowledge graph, and Qdrant hybrid (dense + sparse) retrieval. A ReAct agent answers multi-hop questions over the graph. The mismatch auditor combines deterministic 3-way matching with a DeBERTa-MNLI cross-encoder for spec-conformance, and the grounding layer reuses MiniCheck-Flan-T5 on CPU — so the GPU is reserved for the generator and nothing ever leaves the machine.
To measure the auditor honestly, we generate a seeded, byte-for-byte reproducible corpus of 66 purchase orders across 4 projects and 8 vendors (202 documents), with 43 deliberately planted discrepancies and exact ground-truth labels — clean records generated to match exactly, so any flag on a clean record is a true false positive, not a generation artifact.
Measured results (plumbline eval --nli):
- Mismatch detection: 1.00 precision / 0.95 recall / 0.98 F1 across 8 discrepancy types
- Grounding: 100% of supported claims traced to source, 100% of fabricated clauses flagged on a 48-claim adversarial set
- 58 passing tests, ~0.5 ms/PO for the structured audit
Challenges we ran into
The honest one: corpus-wide contradiction detection is genuinely hard. Our first instinct was to run pairwise NLI across the whole document set, and it failed — high false-positive rate and low recall, exactly as the research warns. The insight that made Plumbline work was to narrow the problem: procurement documents have structure (POs, invoices, quantities, SKUs), so a deterministic 3-way match handles the bulk with perfect precision, and NLI is reserved only for the genuinely semantic question of spec conformance. Two engines, each doing what it's good at.
We also had to earn a CFO's trust in the headline dollar number — our first version double-counted overlapping findings on the same PO, so we cap each PO's exposure at what could actually be wrongly paid against it.
What we learned
- Match the technique to the sub-problem. The same task that's near-impossible as free-form contradiction detection is near-perfect as structured 3-way matching.
- Independent verification beats self-checking. A separate small model catching the generator's hallucinations is measurably better than asking a model to grade itself.
- Report the hard number. Our semantic spec-conformance layer catches only 60% of subtle downgrades, because NLI models are weak at numeric reasoning ("U-factor 0.41 vs. required 0.28"). We surface that ceiling instead of hiding it — it's where the next work is.
What's next
Real-document ingestion (PDF/OCR → structured fields) to handle messy invoices in the wild, a stronger numeric-aware spec checker, and a delivery-delay forecaster that cascades slippage onto the project's critical path — bridging into the Supply Chain track. Then a pilot on a firm's real purchase orders, which is the validation that matters most.
Log in or sign up for Devpost to join the conversation.