TheUstad

Codex says done. TheUstad checks whether that is true.

When I use a coding agent, I give it a goal or a prompt to build something. It works for a while and eventually says, “It’s done.” But when I check the work, it is sometimes incomplete or the full test suite is still failing. A completion message is only a claim.

There is another problem: when a test keeps failing, an agent may change, weaken, or even delete that test instead of fixing the actual behavior. The test suite can become green while the underlying problem is still there.

That is why I built TheUstad. “Ustad” means teacher.

TheUstad does not simply trust the agent’s final message—it checks the evidence. Before the agent starts, TheUstad creates a fingerprint of protected inputs, including tests and verifier configuration. It then captures the agent’s final message, detects completion claims, checks that the protected files have not changed, and runs the configured verifier itself.

If the agent says the work is complete but the verifier fails, the claim is marked FALSIFIED. If the agent changes or deletes a protected test, TheUstad marks the round TAMPERED, restores the original files, and sends the evidence back into the same Codex session so the agent can try again. Only an explicit completion claim supported by a passing verifier becomes VERIFIED.

TheUstad also handles missing completion claims, incomplete work, agent crashes, and timeouts. Every round is written to a hash-chained audit log.

VERIFIED does not mean that the software is guaranteed to be bug-free. It means that the agent’s completion claim matched the verifier selected by the user, with the protected inputs still intact.

This is a documented developer problem:

• OpenAI reports that internal coding agents sometimes illegitimately edit tests and treats this reward-hacking behavior as rare but high severity.
• METR documented a software-engineering run where Claude edited a test file to make tests pass.
• The 2025 Stack Overflow Developer Survey found 46% of developers distrusted AI-tool accuracy, while 33% trusted it.
• DORA describes a verification tax: time saved generating code can be spent again auditing it.
• A SWE-bench correctness study found 7.8% of plausible patches in its evaluation passed narrower validation but failed the full developer test suite.

What I built

TheUstad is a verification-and-retry runtime for coding agents. It treats a completion message as a claim to test, not proof to accept.

For every round, TheUstad:

1. snapshots trusted tests and configuration outside the target repository;
2. launches a separate child Codex task;
3. captures and resumes that exact child task;
4. terminates the agent process group before verification;
5. checks the protected manifest before and after the verifier;
6. runs the configured verifier with a trusted absolute Python interpreter in isolated mode;
7. restores protected inputs after tampering;
8. returns concrete failure evidence to the same child task; and
9. appends the result to a SHA-256 chained JSONL audit log.

Only FINAL VERIFIED exits zero. A neutral final message is PASS_NO_CLAIM, not a false success. FINAL VERIFIED means the explicit completion claim passed the configured protected verifier; it does not mean all software is universally correct.

One core, two interfaces

The default master branch contains both interfaces:

• Standalone CLI for CI, shell automation, and direct security review.
• Codex plugin with $theustad:doctor, $theustad:run, and $theustad:audit.

The plugin packages the same enforcement core outside the target repository. It does not use a weaker verifier or trust a mutable project-local copy.

Install from the public repository:

git clone https://github.com/YashwanthGathuku/theustad.git
cd theustad
python3 -m venv "$HOME/.local/share/theustad/plugin-venv"
THEUSTAD_PYTHON="$HOME/.local/share/theustad/plugin-venv/bin/python"
"$THEUSTAD_PYTHON" -m pip install --upgrade pip pytest
"$THEUSTAD_PYTHON" scripts/install_plugin.py
codex plugin list --json

The list must show theustad@personal installed and enabled. The standalone CLI remains available as python theustad.py --repo ... --task ....

Supported platforms: Linux, macOS, and WSL 2 for coding runs. Native Windows can install and discover the plugin and audit an existing chain, while doctor and run fail closed because the runtime requires POSIX process-group termination.

Real-project proof

The demo uses a fresh clone of the real pytest-dev/iniconfig project at a pinned upstream commit and the same human-authored acceptance test in each run.

Without TheUstad: ordinary Codex solved the task and all 50 tests passed, but there was no protected verdict, no audit log, and no externally checkable root.

With the installed TheUstad plugin: the same acceptance suite passed, TheUstad returned FINAL VERIFIED, captured the exact child task, and produced an independently valid SHA-256 chain.

The demo also clearly labels a conftest-poisoning attack as a deterministic adversarial rehearsal. TheUstad reports TAMPERED, restores the planted protected file, detects a changed audit copy as BROKEN, and validates the untouched chain as VALID.

The 70-second narrated video includes burned-in captions and explicitly explains how I used Codex and GPT-5.6.

Reproducible evidence

Public repository: https://github.com/YashwanthGathuku/theustad

Plugin and CLI guide: https://github.com/YashwanthGathuku/theustad/blob/master/docs/PLUGIN_GUIDE.md

Release evidence: https://github.com/YashwanthGathuku/theustad/tree/master/docs/evidence/theustad-1.0

Fresh-clone validation: https://github.com/YashwanthGathuku/theustad/blob/master/docs/evidence/theustad-1.0/fresh_clone_validation.txt

Anchored fresh-clone audit root:

61a9093555ffc45de77a96ea01a3370e9bc0c1956eb84a8119a73f6469738bc4

Final video SHA-256:

D236FF13EF24A5F1F065F32D1E19158D59AE983C6F0D6C41F993081F62DC1A86

How I used Codex and GPT-5.6

I used Codex with GPT-5.6 to implement the authoritative security contract, build both interfaces, write and run the test suite, red-team tamper paths, capture the real-project proof, rewrite the documentation, and package the release. The build and feedback task ID is:

019f708d-eb32-72d0-a58d-fdd5ffcff511

Honest limitation

TheUstad cannot turn weak acceptance tests into a strong specification. Reviewers still need human-authored acceptance criteria and production-diff review. TheUstad succeeds when an agent's completion claim becomes falsifiable, protected, reproducible, and independently auditable.

Built With

• ffmpeg
• github
• gpt-5.6
• openai-codex
• pytest
• python
• wsl-2