Inspiration
AI makes it easier to write code quickly, but every change still needs to be checked. I built ProofPatch around a question that comes up in code review: did this change break something that worked before? I wanted a reviewer to see the exact test that answers that question and be able to run it again.
What it does
ProofPatch compares two Git commits. Gemini reads the diff, the relevant Python files, and a description of the expected behavior, then generates a targeted pytest test. ProofPatch runs the same test against both commits in Docker. When it passes on the base commit and fails an assertion on the new commit, the dashboard shows the regression alongside the diff, test code, and failure output. You can also supply your own test.
How I built it
I used Streamlit for the dashboard, FastAPI for the backend, Git to read files from each commit, and Docker to run pytest in separate environments. I built a reusable GitHub Actions workflow that takes the base and head commit IDs from a pull request and calls the same API with a supplied test. I verified that workflow logic locally with simulated PR events.
Challenges I ran into
A failed run can mean several things. A test might catch a real regression, or it might never run because a dependency is missing or Docker timed out. I separated assertion failures from execution errors so ProofPatch can report an inconclusive result when it lacks enough evidence. I also tested cases where both commits already contain the bug and where a change fixes it.
Accomplishments that amproud of
I caught a regression in a real, three-file Git repository. A $60 order received a $10 discount, but a one-character change to the free-shipping rule made checkout charge $55 instead of $50. The same test passed on the working commit and failed on the buggy commit. My local CI simulation failed for the buggy comparison and passed for the clean one.
What I learned
A useful generated test needs a clear statement of what the code should do. The diff helps Gemini decide where to look, and the results from both commits show whether the test actually caught a change in behavior.
What's next for ProofPatch
My next milestone is to verify the GitHub Actions check on a live pull request, then add Gemini test generation to that CI workflow. I also want to make ProofPatch useful while people are writing code: paste a snippet, highlight a section, or share a screen to start investigating. In conversation mode, developers could ask why a test failed, reason through the bug with the assistant, and turn what they learn into a regression test for code written by a person or another AI.
