Inspiration

AI coding assistants can generate security patches in seconds, but how can developers know whether those patches actually work?

A patch might look correct while leaving the original vulnerability exploitable. It might even introduce new bugs that break existing functionality.

We built PatchProof to bridge the gap between AI-generated code and verifiable security. Instead of trusting an AI's explanation of a fix, PatchProof attempts to reproduce the vulnerability, applies a patch, and executes the same tests again to produce concrete evidence.

What it does

PatchProof is an AI-powered security auditing and remediation application for public Python GitHub repositories.

Users submit a repository URL, and PatchProof automatically:

Analyzes source code to identify a potential security vulnerability.

Generates a security test designed to reproduce the vulnerability, alongside a regression test for normal functionality.

Executes the original code in an isolated Daytona cloud sandbox to confirm the exploit can be reproduced.

Generates a targeted patch using Google's Gemini models.

Verifies the patch by rerunning the unchanged tests, checking existing repository tests when available, and generating a Git diff.

Produces an evidence report displaying the vulnerability, original code, proposed fix, and before-and-after test outcomes.

Unlike a conventional AI code assistant, PatchProof does not declare success simply because an AI produced new code. A vulnerability must first be reproduced, and the proposed patch must subsequently pass the verification checks.

How we built it

We built PatchProof with Next.js, React, JavaScript, Python, Google Gemini, and Daytona.

The Next.js frontend provides a dashboard where users submit GitHub repository URLs and monitor the audit pipeline.

Gemini analyzes selected Python source files, identifies candidate vulnerabilities, generates exploit and regression tests, and proposes security patches.

Daytona provides disposable, isolated cloud sandboxes where untrusted repository code and generated tests can execute without running on the user's local machine.

Our backend coordinates the entire process, compares before-and-after test results, attempts the repository's existing Python unit tests, and generates a downloadable JSON report containing the evidence and Git diff.

Challenges we ran into

One major challenge was distinguishing between a potential vulnerability and a vulnerability that could actually be reproduced.

Initially, Gemini sometimes generated security tests that passed against vulnerable code, meaning the tests did not demonstrate the issue. We implemented a feedback-and-retry mechanism that lets Gemini refine unsuccessful test cases using their actual execution results.

We also encountered SDK integration issues, Python module naming conflicts, inconsistent AI output formatting, and temporary model overload. Resolving these issues helped us make the execution pipeline more reliable.

Accomplishments that we're proud of

We successfully demonstrated PatchProof on two different public test repositories containing intentionally introduced vulnerabilities:

SQL Injection: PatchProof identified an authentication bypass, reproduced the exploit, generated a parameterized SQL query as a fix, and verified that the original attack was blocked.

Path Traversal: PatchProof identified unauthorized file access through a vulnerable file-reading function, reproduced the issue, generated a path-containment fix, and verified that the exploit was blocked.

In both demonstrations, the generated security tests passed after patching, regression tests passed, existing repository tests continued passing, and the Daytona sandboxes were cleaned up successfully.

These results demonstrate an end-to-end workflow across two vulnerability categories rather than a single hardcoded example.

What we learned

The most important lesson was that AI-generated fixes should be evaluated through execution, not just explanation.

We also learned how to orchestrate AI models with isolated cloud execution environments, manage unreliable model responses, and build test-driven feedback loops for autonomous software engineering.

What's next for PatchProof

Our next goals are to support JavaScript, TypeScript, and additional programming languages; improve multi-file repository analysis; integrate broader testing frameworks; support private repositories securely; and enable developers to review proposed fixes through automatically generated GitHub pull requests.

We also plan to improve independent test validation, authentication, and scan history.

PatchProof is currently an experimental prototype. Verification is limited to the specific vulnerability and tests exercised during a scan; it does not certify an entire repository as secure.

Our vision is to make AI-generated security fixes more trustworthy by making reproducible evidence part of the development workflow.