Six of Two Hundred

An evidence-linked exception queue that checks AI-drafted customer replies against order records and policies, then sends only contradictions, missing evidence, or coverage gaps to a human.

Comment

Language note

I am a native Japanese speaker and I am not fluent in English, so I used translation assistance to prepare the English version of this submission after reviewing its meaning in Japanese. The project decisions, implementation choices, and factual claims are my own.

Inspiration

AI can draft a full queue of customer replies in minutes, but the last mile can still fail in a familiar way: a person is asked to skim every reply and gradually becomes a rubber stamp. Random sampling is not enough when the costly mistake is rare. I wanted a workflow that checks every material claim against the records that matter and gives the human a smaller queue with evidence attached.

What it does

Six of Two Hundred is an evidence-linked exception queue for AI-drafted customer replies. It checks material claims against local order, transaction, and policy evidence and uses four outcomes: EXCEPTION, NO_EXCEPTION_DETECTED_BY_CURRENT_CHECKS, INCONCLUSIVE_EVIDENCE, and NOT_CHECKABLE.

The main surface is an Outbox rather than an evaluation dashboard. It shows the small set that needs attention, a claim rail, and the evidence behind each result. Contradictions, missing or ambiguous evidence, and coverage gaps remain visible to a person. The product never calls a reply “safe” or “cleared.”

A separate Self-Test view keeps three evidence scopes apart: one live GPT-5.6 integration probe, a zero-network deterministic replay, and development fixtures whose human gold review is still pending.

How I built it

I built a synthetic commerce domain with an SQLite order store, transaction records, Markdown policies, a deterministic evidence layer, a guarded GPT-5.6 Sol path, and an HTML/JavaScript operator interface.

GPT-5.6 performs bounded semantic claim extraction and assessment and returns strict structured output. Local tools fetch order, refund-transaction, and policy evidence. Deterministic code then re-fetches every cited locator, checks evidence relevance and status consistency, creates the operator-visible explanation, and retains final routing authority.

I used Codex throughout the primary build thread as the engineering partner for the data spine, guarded tool loop, claim/evidence registry, canary and replay harnesses, budget and approval gates, failure regressions, release export, and UI. An earlier run, A2R1, produced structured output but failed the product's evidence-relevance gate because one claim did not cite every required transaction field. I used that failure to tighten the locator contract and regression suite. A2R2 then completed the bounded integration path.

Challenges

The hardest part was not getting a model to produce a verdict. It was proving that the verdict was connected to the correct records and failing closed when evidence was missing, ambiguous, or substituted from the wrong source.

I also had to keep three kinds of evidence from being mixed together: a live model integration result, a local deterministic replay, and development fixtures. I added approval and budget guards around the paid API path and chose not to run Stage B or create a final holdout when the remaining budget and deadline did not support a defensible evaluation.

Results and what I learned

I keep the measured results separated by scope:

  • Live A2R2: One synthetic case completed in 2 requests, 1 tool round, and 0 retries. The strict output passed. Deterministic code re-fetched 11/11 cited locators. Evidence relevance, status/evidence consistency, and presentation integrity passed; the fabrication count was 0; and the final route was EXCEPTION / human_queue. The recorded run cost was USD 0.04249375.
  • Local deterministic replay: 17/17 locators were re-fetched with zero product network attempts.
  • Development Self-Test: 20/20 fixtures were detected, but the gold labels still have HUMAN_REVIEW_PENDING. This was not a final holdout.

The synthetic Outbox demo contains 16 replies: 12 show No exception detected by the current checks, 2 are outside coverage, and 2 are routed for human attention.

The main lesson was that model confidence is not a sufficient product boundary. The useful boundary is a semantic layer that can propose an assessment and a deterministic layer that can reject unsupported evidence and control routing.

What's next

Next I would complete independent human review of the development gold labels, expand the claim/evidence registry to more reply types, measure operator handling time, and run a preregistered multi-item evaluation only under a separate budget and approval. Real customer data and automated sending remain out of scope until privacy, security, and operational review are complete.

Built With

Share this project:

Updates