RepoWitness

• 

• 

Language note

I am a native Japanese speaker and I am not fluent in English, so I used translation assistance to prepare the English version of this submission after reviewing its meaning in Japanese. The project decisions, implementation choices, and factual claims are my own.

Inspiration

I built RepoWitness because a postmortem action item can be marked “Done” without anyone checking whether the repository contains evidence of the promised change. Ticket state and repository evidence are not the same. I wanted a tool that makes this gap inspectable instead of treating a workflow status as truth.

I chose the name RepoWitness because the tool acts as a witness to what the repository can—and cannot—support.

What it does

RepoWitness audits a Markdown postmortem against a local Git repository. For each remediation claim, it returns one of three bounded outcomes:

• supported by repository evidence;
• no supporting evidence found within the completed search scope;
• not testable from the available evidence.

It records the paths that were searched and excluded, the Git range, executed tools, failures, evidence anchors, and provenance. A supported result must have a mechanically verified file, line, and commit anchor.

RepoWitness never turns “I did not find it” into “it does not exist.” It also does not infer deployment, runtime, feature-flag, or production state from Git.

How I built it

Codex was my primary engineering collaborator throughout the build. It helped design and implement the bounded read-only file and Git tools, the evidence packet, anchor verification, search-scope accounting, provenance records, the deterministic verdict gate, replay system, security regressions, browser QA, release checks, and the static evidence board.

I reviewed the product boundaries, approved external operations, and repeatedly audited the generated evidence before release.

GPT-5.6 Sol was used once for a narrow semantic assessment over fixed, host-supplied candidates in a synthetic fixture. It did not access the repository and did not issue the final verdicts. Deterministic host code verified the evidence anchors, enforced the completed-search-scope boundary, and retained final verdict authority.

The fastest judge path is an API-key-free static demo. Judges can open demo/index.html directly without rebuilding the project or making a network request.

Challenges I ran into

The hardest problem was negative evidence. “No supporting evidence found within this search” is a defensible statement. “The implementation does not exist” is usually not.

I therefore had to make every negative result depend on an explicit, completed search scope. I also had to treat the target repository as potentially hostile. The implementation fails closed on unsafe paths, symlinks, malicious Git configuration, signature programs, lazy network fetches, malformed model output, and unverified evidence anchors.

Another challenge was keeping the language model useful without allowing it to become the source of truth. The model can assess fixed candidates, but deterministic code can reject its output and always owns the final result.

Results and what I learned

On the bundled synthetic demo, the deterministic evaluation matched all 5 expected action-item outcomes.

In one approved live GPT-5.6 semantic micro-smoke, the final host gate matched all 3 expected verdicts, with 0 false SUPPORT results and 0 fabricated anchors. It used one request, one attempt, and no retry.

The final main branch passed 206 tests, together with desktop and mobile browser QA.

These are limited synthetic results. They are not a real-world benchmark and do not establish general accuracy or production-state verification.

The main lesson for me was that abstention can be a product feature. A trustworthy developer tool should show what it searched, what it verified, and where its evidence ends, rather than hiding uncertainty behind a confident answer.

What’s next

I would like to test RepoWitness on more independently designed fixtures, add structured human review, verify Linux and Windows support, and study how well the claim-planning layer transfers to real postmortems.

A future version could run as a CI or scheduled review step, but broader accuracy claims should wait for real-world and holdout evaluation.

Built With

• codex
• python