About the Project

About Us

My name is Paul, and I am a student from Lithuania. I am building PatchWitness together with a friend. We both enjoy working with AI tools, testing ideas, and starting projects that sound simple until we actually begin building them.

Working together was basically co-op mode. One of us would build something, and the other would immediately try to break it. This was useful because whenever one of us said, “It should work,” the other one treated that as a challenge instead of proof.

We have both used Codex for coding and were impressed by how quickly it can work across a large project. At the same time, we noticed that AI tools can sound very confident even when they misunderstood the task or missed an important problem.

We kept coming back to one question:

“Okay, but how do we know it actually works?”

That question became the main idea behind PatchWitness.

Inspiration

AI coding agents are becoming very good at creating patches, writing tests, and explaining why their changes should work. The problem is that the same agent is usually also the one deciding that the task is finished.

It is similar to writing your own exam, checking it yourself, and then being surprised that you received 100%.

We also realised that a passing test does not always prove that the patch fixed anything. The test might already have passed before the code was changed. In that case, it proves that the test passes, but it does not prove that the patch caused the requested behavior to change.

Because of this, PatchWitness compares the same behavior before and after the patch.

The strongest result is:

[ \text{Original FAIL} \rightarrow \text{Patched PASS} ]

This shows that the behavior failed before the patch and passed afterward.

It does not prove that the whole project is completely free of bugs, because no normal testing tool can honestly guarantee that. However, it gives much stronger evidence that the patch fixed the specific behavior that was tested.

That is why we called the project PatchWitness. The patch makes its case, but it still needs a witness. Some bugs are repeat offenders, so putting them on trial felt appropriate.

What PatchWitness Does

The user selects a Git repository and writes a task, for example:

Reject expired API tokens without breaking valid authentication.

PatchWitness first starts a Builder AI. The Builder reads the repository and makes the requested change inside an isolated Git worktree. This means it does not directly change the user’s main project folder.

After the Builder finishes, PatchWitness saves the patch, changed files, commands, and available execution results.

Then it starts a separate Verifier AI.

The Verifier can inspect the original task, the repository, the final patch, and the real test results. It does not receive the Builder’s private reasoning, confidence, or explanation of why the patch should work.

We thought this separation was important because two AI agents agreeing does not automatically make them correct. Sometimes it only means the same mistake received a second opinion.

The Verifier turns the task into smaller things that can actually be checked. In the token example, it might check whether expired tokens are rejected, whether valid tokens still work, and whether unrelated authentication behavior stayed unchanged.

PatchWitness runs these checks against both versions of the repository and compares the results.

A FAIL → PASS result means the tested behavior changed and is now working. A PASS → PASS result means existing behavior was preserved. PASS → FAIL means the patch introduced a regression, while FAIL → FAIL means the original problem is still unresolved.

We did not want to show a made-up result such as “97% confidence.” A confident AI can still be confidently wrong.

Instead, the user can see what changed, which tests were used, what happened before and after the patch, and whether anything else broke. The report also keeps the commands and outputs, so the result can be checked again instead of only being trusted because the AI sounded convincing.

One thing that was important for us was not letting the AI decide by itself whether the task was successful. The models can suggest code changes and tests, but PatchWitness is the one that actually runs them and saves what happened. If an AI says that a test passed, there still needs to be a real output and exit code behind it.

The models can make suggestions, but PatchWitness keeps the receipts.

How We Built It

We built PatchWitness as a local developer tool using Next.js, React, TypeScript, Git worktrees, Vitest, Playwright, Codex, and support for other compatible or local AI models.

The general process is quite simple to explain. PatchWitness checks the repository, creates isolated versions, lets the Builder make a patch, asks the Verifier what should be tested, runs the checks on both versions, and then generates a report.

In the actual project, it was not nearly as simple.

Almost every “small change” unlocked another side quest, and some of those side quests had side quests of their own. At some point, the word “small” became more of a motivational phrase than a measurement.

We used Codex with GPT-5.6 as our main development tool. It helped us work on repository validation, Git worktrees, Builder and Verifier sessions, provider settings, command policies, reports, automated tests, and the user interface.

The biggest advantage was that Codex could follow one change through several parts of the system. For example, changing how a verification result was stored could also require changes to TypeScript types, schemas, API routes, saved data, UI components, reports, and tests.

Codex made those connected changes much faster. However, we still made the main product decisions, especially that execution results should have more authority than an AI explanation.

Codex helped build the homework, but it was not allowed to grade it.

PatchWitness also supports other models. A user can use Codex as the Builder and another provider or local model as the Verifier. The roles can have different permissions, time limits, network access, and fallback options.

Supporting other models was harder than we first expected. Some models passed a basic connection test and then failed when they had to inspect files, use tools several times, or return structured results.

It was similar to passing the job interview and then discovering that the job also requires doing the job.

Because of this, PatchWitness checks whether a model can actually do the work needed for its role before using it in a real verification run.

Challenges and What We Learned

One of the hardest parts was keeping the Verifier genuinely independent.

Using two different prompts was not enough. The Builder and Verifier needed separate sessions, context, permissions, responsibilities, and stored results. The Verifier can inspect the patch, but it cannot see the Builder’s explanation of why the patch should work. It is also normally read-only, so it cannot quietly change the patch it is supposed to judge.

Another difficult part was running code from real repositories.

At first, “run the tests” sounded like one of the easier features. Then we remembered that tests are still code, and code has never been known for respecting simple plans.

Tests can access files, start other processes, connect to the internet, run forever, or produce huge amounts of output. To reduce these risks, we added isolated worktrees, command restrictions, path checks, timeouts, output limits, network controls, and restricted Verifier permissions.

These protections reduce risk, but they do not replace a complete operating-system sandbox. We thought it was better to say that clearly instead of pretending that running unknown code is completely safe.

We also had to decide how PatchWitness should handle failed or incomplete runs.

It would have been easy to make every finished run look successful, but changing the color to green does not fix the result.

PatchWitness records when a command times out, a test does not compile, a model returns invalid output, the patch changes files outside the expected scope, or an existing feature stops working.

A failed verification is still useful. It tells the user that the patch has not earned trust yet. A green result without evidence is like showing the victory screen while the boss still has half of its health.

The biggest thing we learned is that execution is more trustworthy than explanation.

[ \text{PASS} \rightarrow \text{PASS} ]

shows that existing behavior was preserved.

[ \text{FAIL} \rightarrow \text{PASS} ]

shows that the tested behavior changed.

Both results are useful, but they prove different things.

We also learned that uncertainty should not be hidden. Sometimes the most honest result is simply:

“There is not enough evidence yet.”

It is not the most exciting answer, but it is much better than making one up.

What We Are Proud Of

We are proud that PatchWitness is not only another tool that writes code with AI.

It adds an independent verification layer between an AI-generated patch and the person deciding whether to accept it.

PatchWitness does not claim that it can prove a patch has no bugs. It also does not replace developers, code review, security testing, or human judgment.

Its promise is more realistic: