Quote Evidence Matrix

Language note

I am a native Japanese speaker and I am not fluent in English, so I used translation assistance to prepare the English version of this submission after reviewing its meaning in Japanese. The project decisions, implementation choices, and factual claims are my own.

Inspiration

I started with a practical question: when contractor quotes show different prices, are they actually promising the same work? A quote can look cheaper because a scope item is explicitly excluded, not mentioned, or too unclear to verify. A normal summary can flatten those differences. I wanted a comparison that shows the evidence behind each decision and refuses to turn missing information into confidence.

What it does

Quote Evidence Matrix turns four fictional roof quotes into a 9-by-4 evidence matrix. Each cell has one of four states: INCLUDED, EXCLUDED, NOT_MENTIONED, or UNREADABLE_OR_UNRESOLVED.

A user can open a strong cell to inspect the exact source quote, page, extraction status, verification result, and provenance. NOT_MENTIONED is allowed only when the relevant pages have sufficient extraction coverage and searches for the known aliases find no relevant match. If the evidence is incomplete, contradictory, or unsupported, deterministic code downgrades the cell instead of letting a confident-looking claim pass.

Unresolved cells become specific questions for the contractor. The public sample contains no contract totals, so all four price cards say Total not displayed. QEM does not invent a normalized price or choose a winner.

How I built it

I used Codex throughout the primary build thread to implement and repeatedly harden the synthetic fixture generator, versioned evidence contracts, deterministic verifier, record/replay browser interface, one-request safety gates, forensic recovery tools, judge tests, and reproducible release pipeline.

GPT-5.6 Sol was used separately in three governed synthetic experiments to propose candidate scopes and cell states. Deterministic code retained final authority over exact quotations, page anchors, alias coverage, state conditions, and provenance. The judge-facing sample is a local deterministic replay and makes zero runtime OpenAI API calls.

The application uses dependency-light Node.js 22 ESM with browser-native HTML, CSS, and JavaScript, plus JSON Schema, Git, and GitHub.

Challenges

The hardest distinction was EXCLUDED versus NOT_MENTIONED. A false exclusion can unfairly attribute a statement to a contractor, while a false “not mentioned” result can confuse unreadable evidence with genuine silence. I therefore required stronger evidence for strong states and allowed the verifier to downgrade model output.

I also had to preserve failed experiments honestly. One model run failed the local normalized schema, and another ended at the output-token limit. I kept those failures as audit evidence and regression cases instead of hiding them.

A final challenge was describing a useful result without turning one synthetic run into a general accuracy claim.

Results and what I learned

The public deterministic replay contains four fictional quotes, nine scopes, and 36 cells, with zero runtime API calls.

In one reviewed synthetic GPT-5.6 Sol run, the model returned 13 source-grounded scopes and 52 evidence-bearing cells. Human review found one under-split permit scope and four raw state errors. The formal experiment verdict was INCONCLUSIVE, with a PROMISING semantic signal. These are single-run synthetic diagnostics, not real-world accuracy or repeatability claims.

The main lesson was that a useful AI workflow does not need to pretend the model is always right. It can be more valuable when it makes uncertainty inspectable, preserves evidence, and refuses unsupported claims.

What's next

The next steps are production-grade PDF upload and OCR, a complete privacy and retention lifecycle, repeated evaluation on permission-cleared documents, broader browser and accessibility testing, and additional home-improvement domains. I would keep the same evidence boundary: no “true cost,” best contractor, or winner without the underlying evidence.

Built With

• codex
• javascript