Post-Submission Technical Validation Update
Submission status: Original Build Week artifact intentionally unchanged
Submitted public repository: Bome2017/replayguard-build-week
Submitted package version: ReplayGuard 0.1.0
Purpose of this note: Explain post-deadline validation work without altering the judged submission
Submission-integrity statement
The public Build Week repository has intentionally not been updated with later executable work.
After submissions closed, OpenAI Build Week instructed entrants not to change the submitted description, video, repository, or other project materials during judging. I therefore preserved the submitted repository as the exact artifact available at the deadline.
The work described below occurred afterward in separate local repositories and benchmark workspaces. It is not a replacement for the submitted project, and judges should evaluate the Build Week entry from the frozen public repository.
What the submitted Build Week version contains
The submitted public version is ReplayGuard 0.1.0, a developer tool for metamorphic regression testing of evidence-grounded AI systems.
It implements four controlled evidence transitions:
- Replace decisive support with declared equivalent support.
- Add plausible but non-dispositive distractor evidence.
- Remove decisive support.
- Remove all usable evidence.
ReplayGuard then checks whether the target system changes—or refuses to change—in the required way.
The submitted implementation includes:
- typed YAML fixtures;
- deterministic perturbation construction;
- evidence-role and citation checks;
- clean, brittle, invalid, and execution-error outcomes;
- local deterministic targets;
- a JSON HTTP target boundary;
- optional GPT-5.6 semantic assessment;
- machine-readable JSON reports;
- standalone HTML reports;
- CI-oriented process exit codes;
- committed clean and intentionally brittle examples.
Why the submitted repository was not replaced
A later local implementation, ReplayGuard 0.3.0, contains corrected and expanded assurance functionality that did not exist in the submitted 0.1.0 code.
Pushing that implementation into the Build Week repository during judging would have changed the submitted executable artifact after the deadline. I therefore kept the public submission frozen rather than retroactively presenting later work as part of the competition entry.
The later implementation and benchmark results described below should be understood only as post-submission technical validation of the project direction.
Post-submission ReplayGuard version
The later implementation tested locally was:
- Version:
0.3.0 - Commit:
fc680c5851852f0bef662d0eb81aa0f3051261e0 - Source state: clean and independently frozen before comparison
- ReplayGuard test suite:
124 passed - Clean fixture: exit
0 - Intentionally brittle fixture: exit
1 - Certificate creation: exit
0 - Independent certificate check: exit
0
The later version adds capabilities not present in the submitted public 0.1.0 implementation:
- corrected evidence-transition semantics;
- exact fixture and response identity binding;
- exact selected-subject binding;
- certificate-v2 creation;
- independent semantic certificate recomputation;
- a type-distinct fail-closed downstream gate.
Expanded independent benchmark
A separate benchmark was created to test whether ReplayGuard merely passed its own examples or correctly implemented a broader evidence-transition contract.
Contract extraction
The frozen ReplayGuard source yielded:
- 87 extracted rules
- 82 gating rules
The original two fixtures exercised only part of that rule surface, so a broader corpus was constructed.
Independent semantic oracle
A separate semantic oracle was implemented without importing ReplayGuard, Promptfoo, or Giskard and without reading their verdicts.
Before it was allowed to judge the implementations:
- 15 of 15 intentional oracle mutations were killed
- passing controls remained accepted
- an initially over-derived citation rule was detected and removed before the corpus freeze
This mattered because the oracle initially appeared to expose a ReplayGuard failure, but inspection showed the oracle had imposed a restriction absent from ReplayGuard’s frozen specification. ReplayGuard was correct in that case.
Corpus
The final primary corpus contained:
- 24 development fixtures
- 16 sealed holdout fixtures
- 40 total fixtures
- a holdout generated only after the comparator implementations were frozen
The holdout seed was derived deterministically from:
- the frozen corpus commit;
- the frozen comparator commit;
- a fixed benchmark domain string.
No implementation was changed after holdout generation.
ReplayGuard result
ReplayGuard 0.3.0 matched the independent oracle on:
- 24/24 development fixtures
- 16/16 sealed holdout fixtures
- 40/40 total fixtures
Observed results:
- zero false positives;
- zero false negatives;
- zero fail-open cases;
- zero invalid-versus-execution-error confusion;
- semantic determinism across three full-corpus runs;
- zero ReplayGuard source modifications.
Comparison with public alternatives
The same evidence-transition protocol was reconstructed independently in two public evaluation frameworks.
Promptfoo 0.121.20
Promptfoo ultimately matched the oracle on all 40 fixtures, but doing so required:
- 1,094 cumulative lines of ReplayGuard-specific comparator code
- external orchestration;
- a custom provider;
- custom assertions;
- custom input and result handling.
Promptfoo exposed 69 assertion types in the inspected release, but none natively compared the outputs of two separate test executions.
Giskard OSS 2.19.2
Giskard ultimately matched the oracle on all 40 fixtures, but doing so required:
- 958 cumulative lines of benchmark-specific comparator code
- custom tests;
- a custom model wrapper;
- external orchestration;
- custom input and result handling.
Giskard provided two useful native relational behaviors out of the 21 capabilities classified for the comparison, but its RAG metrics were pointwise rather than cross-execution relational checks.
Functions not reproduced by either comparator
Neither completed comparator reproduced:
- exact selected-subject binding;
- certificate-v2 semantic recomputation;
- the type-distinct fail-closed consequence gate.
Benchmark verdict
The benchmark’s frozen primary verdict was:
SEALED_PROTOCOL_CONFIRMED_REPLAYGUARD_ADVANTAGE
This verdict does not mean ReplayGuard was more accurate than the completed reconstructions. All three implementations matched the oracle on 40/40 fixtures.
The measured advantage was narrower:
- ReplayGuard implemented the protocol without reconstruction.
- Promptfoo and Giskard required substantial protocol-specific code.
- Neither alternative supplied the complete protocol natively.
- ReplayGuard retained three integrated assurance functions that neither comparator reproduced.
The supported technical conclusion is:
ReplayGuard is the more complete ready-made implementation of this exact tested evidence-transition assurance protocol than the tested Promptfoo and Giskard OSS versions.
Important limitations
The post-submission benchmark used retained deterministic target responses.
It did not:
- call a live LLM;
- test a production RAG deployment;
- establish that ReplayGuard catches failures in a real operational system;
- establish commercial demand;
- establish superiority for evaluation tasks outside this protocol;
- establish that no private equivalent exists.
The result establishes implementation correctness and comparative architectural differentiation for the frozen protocol.
Relationship to the Build Week submission
The Build Week judges should evaluate the original public 0.1.0 repository, video, description, and submitted evidence exactly as they existed at the deadline.
This post-submission work is disclosed only to provide transparent context:
- the project continued to be developed separately;
- the submitted repository was deliberately left unchanged to protect submission integrity;
- later testing provided stronger evidence for the underlying technical approach;
- none of the later code or benchmark results are being represented as pre-deadline submission work.
Log in or sign up for Devpost to join the conversation.