DevPilot AI

An AI developer agent that uses MCP and AWS to investigate incidents, analyze context, and help engineers resolve production issues faster.

Comment

DevPilot AI

The Idea

Software repair agents are becoming increasingly capable, but giving an AI agent permission to modify a codebase creates an important problem: how do we know that the proposed fix is actually justified?

DevPilot was built around a simple principle: an AI should not get to call a change a fix just because the code looks plausible.

DevPilot is a safety-first software repair agent that investigates a failing repository, gathers repository-grounded evidence, forms and evaluates hypotheses, proposes a minimal change, reviews that proposal, and independently verifies the repair through a RED → PATCH → GREEN workflow.

The goal is not just to generate a patch. The goal is to produce a repair with a traceable chain of evidence showing why the change was proposed and whether it actually repaired the original failure.

How DevPilot Works

DevPilot follows a controlled investigation pipeline:

Incident → Evidence → Hypotheses → Proposal → Review → RED → PATCH → GREEN → Grounded Report

First, DevPilot reproduces the reported failure and collects evidence from the repository. Repository reads are converted into stamped, replayable evidence rather than allowing the planner to rely on arbitrary assumptions.

The rules-first planner attempts to explain the failure deterministically. When the available rules cannot sufficiently explain the incident, DevPilot can escalate to a model for hypothesis generation.

The model is not treated as an unquestioned authority. Hypotheses must be grounded in cited evidence, and competing hypotheses are evaluated using their supporting and contradicting evidence. If the evidence does not distinguish competing explanations, DevPilot refuses to force a selection.

Once a hypothesis is selected, DevPilot creates a proposal containing the affected path and patch. The proposal passes through review before it can reach the repair workflow.

The repair workflow then:

  1. Reproduces the original RED failure.
  2. Creates an isolated worktree.
  3. Applies the proposed patch.
  4. Runs the same verification command.
  5. Requires the expected failure to become GREEN.
  6. Checks that the original repository remains unchanged.
  7. Records the result as a grounded repair outcome.

This makes the final result more than an AI-generated code suggestion: it is a verified repair with an auditable execution trail.

Why We Built It

The important challenge with autonomous coding systems is not simply making them generate code. It is preventing plausible-looking but poorly grounded changes from being treated as successful repairs.

While building DevPilot, we focused heavily on failure modes around competing hypotheses, insufficient or contradictory evidence, model-generated reasoning, unsafe patches, incorrect verification, changes outside the intended worktree, and false claims that a repair succeeded.

We added automated tests for these failure modes rather than relying only on a successful happy-path demo.

Model Escalation

DevPilot uses a rules-first, model-escalation approach.

The model is used when deterministic reasoning cannot adequately explain the incident. This keeps straightforward repairs from unnecessarily depending on model calls while still allowing more difficult incidents to be escalated.

In our end-to-end host tests, a rule-solvable incident is repaired without a model call, while an incident that the rules cannot explain is escalated once and then independently verified.

This separation is intentional: the model proposes an explanation; the repair workflow determines whether the proposed change actually works.

MCP Integration

DevPilot is exposed through an MCP server using Streamable HTTP.

The host acts as the composition root, connecting the investigation service, planner, and MCP layer while keeping the MCP-facing surface separate from the underlying engine and repair implementation.

The MCP interface also enforces bearer authentication and an origin allowlist. The host is tested to ensure that sensitive implementation details such as model text, diffs, repository paths, and authentication tokens are not exposed through investigation responses.

What We Learned

The biggest lesson from building DevPilot was that autonomous repair needs to be designed around proof, not just generation.

It is easy to demonstrate an agent producing a patch. It is much harder to demonstrate that the original failure was reproduced, the proposed change came from grounded evidence, competing explanations were handled correctly, the patch was reviewed before execution, the patch was applied in isolation, the same failure disappeared after the patch, and the original repository was left untouched.

Designing and testing those boundaries became a major part of the project.

Validation

The repository has a comprehensive automated test suite covering investigation, evidence, hypothesis, proposal, review, repair, MCP host, and RED → PATCH → GREEN workflows.

Our final validation run completed with:

919 tests 919 passing 0 failing 0 skipped

The end-to-end repair tests specifically verify successful repairs as well as cases where DevPilot must refuse to proceed.

Built for Trustworthy Automation

DevPilot is ultimately an experiment in making autonomous software repair more controlled and explainable.

Instead of asking:

“Can the AI write a fix?”

DevPilot asks:

“Can the system justify a proposed repair and prove that the repair works?”

That distinction is the core of the project.

Built With

Share this project:

Updates

Submission history