Inspiration
AI-generated code can appear complete, but crucial pieces are often missing. It may lack clear acceptance criteria, test only the expected path, or cite documentation without proving the implementation meets the requirement.
I wanted to build something more useful than another model-generated confidence score: a release decision reviewers could examine, verify, and challenge.
What It Does
EvidenceGate evaluates code changes against explicit acceptance criteria using two independent evidence domains: internal and external.
Internal evidence comes from the repository itself, including source analysis, diffs, commands, and test results. External evidence comes from approved authoritative sources, with provenance preserved and citations tied directly to the claims they support.
A required claim passes only when both evidence domains support it.
The demo contrasts two implementations. The first patch looks plausible but is incomplete and fails the evidence gate. The corrected patch properly handles web search, returned-source metadata, native citation annotations, safe URLs, visible links, and adversarial cases, allowing it to pass.
How I Built It
Codex was my primary development collaborator. I used it to decompose requirements, design the architecture, implement the system in strict TypeScript, write tests, create adversarial fixtures, debug issues, prepare documentation, and organize the release.
I remained responsible for the product scope, evidence boundaries, authority policy, privacy requirements, and final submission decisions.
GPT-5.6 serves two distinct roles in the system. In Stage A, the Responses API web-search tool gathers cited narrative and returned-source metadata from approved domains. In Stage B, the model maps the supplied acceptance criteria and evidence IDs into structured assessments.
The model does not control the final release decision. Local validators reject fabricated evidence IDs, unsafe URLs, malformed citation ranges, and forged gate outcomes. A versioned policy layer computes the final result deterministically.
Challenges
The most difficult part was treating every trust boundary as adversarial.
For example, native citation annotations and consulted-source records may refer to the same page with slightly different URLs. EvidenceGate allows only a narrow, documented aliasing rule while preserving the returned-source record as the authoritative reference.
Another challenge was separating model-generated evaluation language from the actual release decision. A model may write “PASS” or "FAIL," but that wording should never be mistaken for the deterministic result. Reports clearly label research prose as untrusted context to prevent confusion.
Accomplishments
Built 173 offline tests covering evidence schemas, policy enforcement, spoofed domains, fabricated citations, prompt injection, stale or conflicting sources, output limits, report escaping, and publication safety.
Completed a live GPT-5.6 fail-to-pass workflow using official-domain web research, native citations, structured adjudication, and verified bundle hashes.
Produced self-contained HTML reports and canonical JSON bundles.
Created a public, no-build judging demo that requires no account or API key and clearly identifies its cached, validated fixture.
What I Learned
Repository truth and documentation truth are both necessary, but they are not interchangeable.
The most valuable role for a model in this system is bounded research and classification. The final release policy should remain deterministic, versioned, and open to inspection.
What’s Next
Once the OpenAI workflow is stable, I plan to add richer analyzers, signed bundles and attestations, stronger process isolation, accessibility regression testing, and narrowly scoped policy packs.
The core principle remains unchanged: every required claim must be supported by both internal implementation evidence and external authoritative evidence.
Log in or sign up for Devpost to join the conversation.