ProofPatch

ProofPatch – Detect, Diagnose, and Automatically Repair Code Defects with AI.

Comment

About ProofPatch

Inspiration

Debugging is one of the most time-consuming parts of software development. Developers often spend hours identifying bugs, understanding error messages, implementing fixes, and repeatedly running tests to ensure their changes work correctly.

AI coding assistants can generate code fixes quickly, but a generated solution is not necessarily a correct solution. An AI-generated patch may introduce new bugs or fail to address the original issue.

This inspired us to build ProofPatch, an AI-powered autonomous code repair system that goes beyond simply generating code. Our goal was to create a system that can understand a reported bug, propose a repair, execute tests in an isolated environment, analyze failures, and iteratively improve its solution.

Our central idea was simple: A code fix should not be considered successful just because AI generated it. It should be considered successful only when the tests verify it.

How We Built It

We developed ProofPatch using a combination of AI-powered code generation, backend orchestration, and isolated test execution.

The system follows an evidence-driven repair workflow:

  1. Code Input: Developers provide their Python source code, describe the identified bug, and supply test cases.
  2. AI-Based Analysis: A locally hosted AI coding model analyzes the source code and bug description to generate a proposed replacement.
  3. Code Comparison: ProofPatch generates a unified diff to highlight the changes between the original and proposed code.
  4. Isolated Testing: The proposed code and test cases are executed inside a Docker container with restricted network access and resource limits.
  5. Failure Analysis: If tests fail, the system extracts relevant error messages, assertions, and traceback information.
  6. Autonomous Repair: The AI receives the failure evidence and generates an improved version of the code.
  7. Verification: The repair process stops when the tests pass or the maximum of three attempts is reached.

Technology Stack

  • Frontend: React, TypeScript, Vite, CSS
  • Backend: Python, FastAPI, Pydantic
  • AI Integration: Ollama with Qwen2.5-Coder
  • Execution Environment: Docker
  • Testing: Pytest
  • Version Comparison: Python unified diff functionality

We chose local AI inference through Ollama to make the system less dependent on external AI APIs. Docker provides an isolated environment for executing potentially faulty code.

What We Learned

Building ProofPatch helped us understand how individual technologies can work together to create an autonomous developer tool.

Some of our major learning outcomes were:

  • AI Integration: Connecting locally hosted language models to real-world software development workflows.
  • Agentic Workflows: Designing an iterative process in which an AI model uses feedback from previous attempts to improve its output.
  • Backend Engineering: Building REST APIs, handling asynchronous requests, and managing communication between frontend and backend components.
  • Containerization: Using Docker to execute untrusted code in a restricted environment.
  • Software Testing: Understanding how automated tests, assertions, exit codes, and tracebacks can provide objective evidence of correctness.
  • Error Handling: Designing reliable workflows for model failures, execution errors, and unsuccessful repair attempts.

One of our most important takeaways was understanding that AI-generated code needs independent validation rather than being trusted automatically.

Challenges We Faced

Developing ProofPatch involved several engineering challenges.

1. Unreliable AI-generated output

AI models sometimes return code surrounded by explanations or Markdown formatting. We implemented code extraction and normalization logic to obtain usable Python source code.

2. Building a genuine autonomous repair loop

Generating a patch once was relatively straightforward. The difficult part was making the system learn from failed test executions. We designed a bounded repair loop that sends actual failure logs and traceback evidence back to the model for subsequent attempts.

3. Safe code execution

Running potentially faulty code directly on the host machine creates security risks. We addressed this by executing tests inside Docker containers with network access disabled, dropped Linux capabilities, read-only container filesystems, and CPU, memory, and process limits.

4. Preventing false verification

A generated patch should never be marked successful merely because the AI returned code. We implemented a verification condition requiring both a successful test result and an exit code of zero.

5. Reliability and testing

We also had to handle model connectivity issues, Docker execution failures, timeouts, and exhausted repair attempts. We developed automated backend tests using mock model providers and test runners to validate these scenarios.

What Makes ProofPatch Different?

Traditional AI coding assistants primarily focus on generating or suggesting code. ProofPatch introduces an evidence-gated approach in which generated changes must pass automated tests before being marked as verified.

Instead of trusting an AI-generated answer, ProofPatch creates a feedback cycle:

Analyze → Generate → Execute → Evaluate → Repair → Verify

Our vision is to develop ProofPatch further into a reliable AI-powered software maintenance assistant that helps developers identify, repair, and validate code defects with greater confidence.

ProofPatch is built around one principle: AI can propose the fix, but evidence must prove it.

Built With

Share this project:

Updates

Submission history