• 

  Real generated HTML report: transition matrix plus expanded failed row — verdict flips to refuted after citing the declared distractor.

• 

  The same transition policy produces auditable outcomes: clean behavior passes, while brittle behavior fails CI with exit code 1.

• 

  Controlled evidence transformations reveal failures static tests miss: paraphrase drift, distraction, and unsupported claims.

• 

  GPT-5.6 performs bounded semantic analysis; deterministic rules own the verdict, reports, and CI exit status.

• 

  ReplayGuard brings metamorphic testing to evidence-grounded AI, turning controlled evidence changes into deterministic CI verdicts.

Inspiration

Evidence-grounded AI systems are usually evaluated one answer at a time: provide evidence, ask a question, and inspect whether the response appears correct.

That misses relational failures.

A system may answer correctly once but change when the same evidence is paraphrased, become distracted by irrelevant information, or preserve a conclusion after its supporting evidence disappears.

ReplayGuard applies metamorphic testing to this problem. Instead of requiring a perfect gold answer for every run, it defines how the target should behave when its evidence changes in controlled ways.

The model analyzes, while deterministic code decides.

GPT-5.6 performs bounded semantic interpretation, while deterministic rules own the final verdict, report, and CI exit status.

What it does

ReplayGuard fails your CI build when an evidence-grounded AI system’s answer stops following its evidence.

It begins with a baseline evidence fixture and generates four controlled replays:

• Equivalent evidence: The meaning is preserved, so the answer should remain materially stable.
• Distractor added: Irrelevant information is introduced, so the answer should resist distraction.
• Supporting evidence removed: Decisive support disappears, so the answer should weaken, change, or abstain.
• No evidence: The evidence packet is empty, so the target should declare insufficiency and avoid unsupported claims.

ReplayGuard then:

1. Runs the target against the baseline and transformed evidence packets.
2. Collects structured verdicts, claims, citations, confidence, and evidence-sufficiency fields.
3. Uses GPT-5.6 for bounded semantic comparisons where exact string matching would be inadequate.
4. Applies deterministic transition rules to classify each replay.
5. Produces machine-readable JSON and interactive HTML reports.
6. Returns exit code 0 for a passing contract, 1 for brittle behavior, or 2 for an invalid test or execution failure.

The public demo uses synthetic procurement-policy fixtures and intentionally clean and brittle targets. The fixtures are controlled, but the CLI, adapters, verdict engine, reports, tests, and CI behavior are real.

ReplayGuard does not formally verify a probabilistic model or certify universal factual correctness. It tests whether a target follows explicit behavioral contracts as its evidence changes.

How we built it

ReplayGuard is a Python 3.11+ command-line developer tool with a modular testing pipeline.

The implementation includes:

• strict Pydantic fixture and response schemas;
• deterministic evidence-transformation builders;
• sample and custom JSON HTTP target adapters;
• GPT-5.6 Structured Outputs for semantic assessment;
• a deterministic transition-verdict engine;
• JSON and self-contained HTML reports;
• reproducible clean and brittle demonstration targets;
• pytest coverage, Ruff validation, and GitHub Actions integration.

Each fixture defines the question, canonical claim, evidence documents, expected baseline behavior, allowed verdicts, and replay contracts. ReplayGuard constructs each evidence state, executes the target, and compares the resulting behavior with the baseline.

GPT-5.6 is constrained to questions such as whether two claims preserve the same meaning or whether a citation supports a claim. It does not decide whether the build passes. Deterministic code enforces the allowed verdict transitions, citation rules, support-removal behavior, abstention requirements, and process exit status.

Codex was the primary implementation environment for building and auditing the CLI, schemas, adapters, reports, tests, documentation, and CI workflow.

The evaluator was also regression-tested against a sanitized corpus of twenty responses retained from four earlier live GPT-5.6 runs. Those responses exposed two false-positive evaluator assumptions, which were corrected and preserved as deterministic regression cases.

Pilot validation

After the demo was frozen, ReplayGuard was evaluated in a preregistered pilot using 30 independent fixtures across six domains.

Two target prompts were tested with three replicates across a baseline and four evidence transformations, producing:

• 180 complete five-state runs
• 900 live GPT-5.6 calls
• zero execution errors
• zero missing prompt pairs
• zero replicate inconsistencies

Across all 180 no-evidence responses, both prompt arms abstained.

The hardened prompt also followed the strict empty-claims contract in all 90 of its no-evidence responses. The naive prompt abstained but returned a nonempty material_claims field in all 90, usually describing the absence of evidence. ReplayGuard therefore classified those responses as strict output-contract violations, not as continued support for the original claim.

The naive arm also produced 5 failures across 90 support-removal responses.

Separately, the released evaluator produced 10 false positives on equivalent-evidence transitions because its citation gate depended on stochastic baseline citation selection. Those disagreements were preserved, audited, and corrected in a separately reported study-level analysis.

The pilot remains exploratory rather than confirmatory. It evaluated the released ReplayGuard commit, and the full 900-call experiment was not rerun after the evaluator corrections. Native released-evaluator outcomes and corrected study-level outcomes are therefore reported separately rather than rewriting the original results.

Challenges we ran into

Semantic equivalence is not textual equivalence

An early evaluator treated expected claims too literally. A target could express the correct meaning through a natural paraphrase and still appear unstable.

ReplayGuard now separates textual form from material meaning. GPT-5.6 performs the bounded semantic comparison, while deterministic rules decide whether the transition passes.

Appropriate change can look like instability

When decisive support disappears, a reliable target should often change its verdict or return insufficient_evidence.

An early confidence rule compared responses across different verdict classes and incorrectly treated correct abstention as degradation. The evaluator had to distinguish inappropriate instability under preserved meaning from appropriate responsiveness after evidence loss.

The evaluator can fail too

The live-response corpus and 900-call pilot both exposed defects in ReplayGuard’s own test oracle.

Rather than removing or silently reclassifying those outcomes, the experiments were frozen, the disagreements were documented, and the cases were converted into regression tests.

An evaluator must be tested as rigorously as the system it evaluates.

Keeping the final control path deterministic

It would have been easier to ask a model whether each run passed. That would make the CI gate probabilistic and difficult to audit.

ReplayGuard instead uses GPT-5.6 for structured semantic interpretation and deterministic code for enforcement.

Accomplishments that we're proud of

ReplayGuard became a complete developer tool rather than only a testing concept.

The project includes:

• an end-to-end CLI;
• four controlled evidence replays;
• fail-closed fixture validation;
• deterministic CI exit semantics;
• structured GPT-5.6 analysis;
• custom target-adapter boundaries;
• JSON and interactive HTML reporting;
• clean and deliberately brittle demonstration targets;
• automated tests and passing CI;
• retained live-response regression fixtures;
• a completed 900-call exploratory pilot.

The strongest accomplishment was not simply catching a brittle target. It was allowing live behavior to reveal weaknesses in ReplayGuard’s own evaluator, preserving those failures, and improving the oracle without hiding the original outcomes.

What we learned

Static correctness testing is not enough for evidence-grounded AI. Reliability also depends on how a system responds when evidence is reworded, supplemented, weakened, or removed.

We learned that:

• transitions between outputs can be more informative than isolated outputs;
• semantic equivalence cannot be reduced to exact text matching;
• abstention and output-contract compliance are related but distinct behaviors;
• preserved meaning and removed support require different acceptance rules;
• probabilistic interpretation can be useful without making the final CI decision probabilistic;
• evaluator defects can resemble target defects unless both are independently audited;
• synthetic fixtures are effective for proving mechanics when their scope is stated clearly.

Most importantly, trustworthy AI testing requires explicit behavioral contracts. “The answer should be good” is not testable. ReplayGuard converts that expectation into inspectable transition rules.

What's next for ReplayGuard

The next stage is larger confirmatory validation across independent fixture clusters and domain-specific evidence contracts.

Future engineering work includes:

• domain-specific fixture libraries;
• adapters for production RAG pipelines;
• claim-to-evidence span mapping;
• configurable transition policies;
• pull-request annotations;
• historical baseline and drift monitoring;
• richer citation-consistency checks;
• fixture-authoring and review tools;
• preregistered confirmatory studies.

ReplayGuard is domain-agnostic at the fixture and adapter boundary, but responsible use in a new domain still requires domain-specific evidence, expected claims, transformations, and acceptance policies.

The long-term goal is to make evidence-grounded AI testing behave more like conventional software quality assurance: reproducible, inspectable, automatable, and strict enough to block a deployment when system behavior no longer follows its evidence.

Built With

• api
• ci/cd
• codex
• developer
• evaluation
• github
• gpt-5.6
• httpx
• metamorphic
• openai
• outputs
• pydantic
• pytest
• python
• ruff
• sdk
• structured
• tools