DevPilot

Inspiration

AI coding assistants are increasingly good at generating code, explaining repositories, and implementing isolated changes.

But software improvement rarely depends on a single code edit.

A developer may want to:

Improve a benchmark score
Reduce inference latency
Fix an unreliable algorithm
Increase test coverage
Reduce memory consumption
Improve retrieval quality
Compare several architectural approaches
Discover which optimisation actually works

A normal coding assistant can propose a solution, make a change, and run a test. But when the first idea fails, the process often begins again with little memory of what was attempted, why it failed, or what was learned.

Long-horizon software research requires more than code generation.

It requires:

A clearly defined objective
Baseline measurements
Multiple competing hypotheses
Isolated experiments
Protected evaluation data
Persistent memory
Evidence-based merge decisions
A final explanation of what worked and what did not

DevPilot CLI was built for that gap.

It turns a software-improvement goal into a structured research process. Instead of making one speculative change, DevPilot creates an Idea Tree, proposes hypotheses, runs experiments in isolated Git worktrees, evaluates the results, learns from failures, and keeps only improvements that survive validation. The maintained CLI distribution also supports multiple model providers, OpenAI login, GitLab Orbit context, memory, compression, research tools, and an installable Codex skill suite.

What it does

DevPilot CLI is an autonomous research agent for codebases.

A developer gives it a goal such as:

“Improve this benchmark’s validation score without changing the evaluation harness or using the held-out test split during iteration.”

DevPilot then:

Inspects the target repository.
Clarifies the objective, metric, baseline, constraints, and budget.
Produces a Research Contract.
Creates an Idea Tree of possible improvements.
Selects the most promising hypothesis.
Launches an Executor inside an isolated Git worktree.
Implements and evaluates the experiment.
Records the score, evidence, failure mode, and reusable insight.
Merges, prunes, retries, or expands the Idea Tree.
Produces a final research report.

The output is not only modified code.

A DevPilot session leaves behind:

An experiment history
A persistent Idea Tree
Evaluation results
Branch and worktree records
Learned insights
Event logs
Run statistics
A final REPORT.md
Enough evidence to understand why a change should or should not be kept

The CLI is designed around cumulative research rather than one-shot generation. Each successful or failed experiment changes what the system knows and influences the next hypothesis.

Typical workflow

A developer enters a benchmark repository and starts DevPilot:

devpilot

During intake, DevPilot confirms:

The target directory
The objective
The evaluation command
The metric
Whether higher or lower is better
The current baseline
Protected files
The development and held-out test splits
Experiment budget
Human-review preferences

The Coordinator then begins the research cycle:

observe → ideate → select → dispatch → evaluate → backpropagate → decide

For example, DevPilot may create these branches in its Idea Tree:

Replace a slow nearest-neighbour loop with vectorised distance computation
Cache repeated feature normalisation
Reduce unnecessary data conversion
Introduce a more efficient search structure
Parallelise one safe portion of evaluation

Each hypothesis is sent to an Executor.

The Executor works in its own branch and Git worktree, makes a bounded change, runs the development evaluation, and returns structured evidence.

The Coordinator can then:

Merge a validated improvement
Prune a failed hypothesis
Refine a partially successful idea
Generate child hypotheses
Stop when the budget is exhausted
Run final held-out verification

DevPilot’s native workflow separates the Coordinator from the Executor so that the research director does not directly make every implementation change. The Coordinator maintains the search strategy, while Executors test individual ideas.

The Idea Tree

The Idea Tree is DevPilot’s central research memory.

Every experiment becomes a node containing information such as:

The hypothesis
Its parent idea
Expected mechanism
Relevant constraints
Development score
Validation outcome
Implementation branch
Evidence
Failure explanation
Lessons for future experiments
Merge or prune decision

This prevents the agent from repeatedly attempting the same failed optimisation.

A failed experiment can still be valuable. It may show that:

A suspected bottleneck is not significant
One protected file cannot be changed
A performance gain reduces correctness
A technique works only under one dataset
A dependency already performs the proposed optimisation
A more promising child hypothesis exists

Those insights propagate through the tree so later ideas begin with more context.

Experiment discipline

DevPilot treats software improvement as an evaluation problem.

Executors iterate on a development benchmark while the held-out test benchmark is protected for final verification.

The workflow is designed around:

B_dev for experimentation
B_test for merge or final verification
Configurable improvement thresholds
Protected evaluation files
Isolated branches
Reproducible commands
Recorded metrics
Evidence-based decisions

An experiment is not considered successful only because the code looks better.

It must produce a measurable improvement, preserve required behaviour, and satisfy the configured constraints.

Each experiment takes place in a dedicated Git worktree so the main project remains untouched until an improvement is deliberately accepted. This isolation and held-out evaluation discipline are core documented features of the CLI.

OpenAI Build Week extension

DevPilot CLI existed before OpenAI Build Week.

During the official submission period, it was meaningfully extended using Codex and GPT-5.6.

The major Build Week extension was adding an Agent Client Protocol runtime that makes DevPilot accessible as a structured agent rather than only through its native terminal interface.

Two dated Build Week commits added:

A DevPilot ACP standard-input/output runtime
Structured streaming of DevPilot events through ACP

These changes were committed on July 15, 2026, during the Build Week submission period.

The ACP extension allows a compatible client to:

Initialise DevPilot as an agent
Create sessions
List saved sessions
Load and resume sessions
Close or cancel sessions
Select an operating mode
Choose reasoning effort
Associate a model configuration with the session
Send research prompts
Receive structured progress events
Continue a previous research run

The supported modes include:

Research — run the complete Coordinator-driven hypothesis workflow
Plan — inspect a project and produce a plan without intentional code changes
Execute — perform one narrow experiment
Review — review code, branches, diffs, or issue context
Audit — perform operational and code-quality auditing
Memory — search previous DevPilot sessions and learned context

These modes are defined directly in the ACP runtime and mapped to bounded DevPilot execution behaviour.

Structured ACP event streaming

A long-running autonomous research agent should not appear as a silent process.

The Build Week extension therefore maps DevPilot’s durable runtime events into structured ACP updates.

The adapter can surface:

Model reasoning updates
Research-plan changes
Proposed hypotheses
Completed hypotheses
Pruned ideas
Merged ideas
Executor starts and completions
Tool calls
Tool results
Cycle progress
Checkpoints
Provider errors
Session completion

For example, when a hypothesis is proposed, the ACP client can receive an updated plan entry. When an Executor begins, it appears as an active execution operation. When a tool edits a file, runs a test, searches code, or fetches evidence, the event is mapped into the corresponding tool-call category.

Sensitive event fields—including API keys, tokens, passwords, secrets, and authorisation values—are redacted before events are exposed to the client.

The adapter follows the session’s append-only events.jsonl file and translates new records into ACP messages, thoughts, plans, and tool updates. It also handles incomplete final JSONL records without silently losing them.

How Codex and GPT-5.6 were used

Codex and GPT-5.6 were used as active engineering collaborators during the Build Week extension.

They helped with:

Auditing DevPilot’s existing CLI and event architecture
Understanding the Coordinator and Executor lifecycle
Designing the ACP session model
Mapping DevPilot runtime state into protocol capabilities
Implementing session creation, loading, resuming, listing, cancellation, and closing
Designing bounded Research, Plan, Execute, Review, Audit, and Memory modes
Integrating the official Agent Client Protocol SDK
Building the standard-input/output transport
Translating Idea Tree events into plan updates
Translating Executor activity into structured execution events
Translating runtime tools into read, edit, search, fetch, delete, and execute operations
Handling process cancellation across Windows, Linux, and macOS
Preserving session state between prompts
Reading append-only JSONL events safely
Redacting credentials and other sensitive values
Writing tests for the ACP agent, event mapper, runtime, session store, and stdio transport
Reviewing compatibility with the existing native CLI
Documenting the new testing and installation path

GPT-5.6 was especially valuable when reasoning across several layers simultaneously:

CLI process management
Asynchronous event forwarding
Session persistence
Protocol schemas
Idea Tree state
Cross-platform signals
Tool-call observability
Security boundaries

I remained responsible for the architecture and product decisions.

Codex accelerated implementation, code review, debugging, and validation, while I decided how much of the native DevPilot runtime should be exposed and which parts should remain internal.

ACP architecture

The Build Week architecture is:

ACP-compatible client → DevPilot ACP stdio adapter → native DevPilot CLI runtime → Coordinator and Executors → Git worktrees and evaluation commands

The ACP adapter does not replace the existing research engine.

It acts as a protocol bridge.

When a user sends a prompt:

The adapter resolves the saved DevPilot session.
It applies the selected mode’s safety prefix.
It launches the native DevPilot CLI as a subprocess.
It associates the run with a persistent session name.
It reads normal process output.
It follows the durable event log.
It maps events into ACP updates.
It sends progress to the connected client.
It saves the session for later continuation.

The adapter uses the native session directory under .devpilot/sessions/, preserving the same checkpoint, event, and reporting model used by the CLI.

Codex Research Agent Skill Suite

DevPilot also includes an installable skill suite for Codex.

The suite reconstructs the core DevPilot research workflow as 11 coordinated Agent Skills rather than one enormous prompt.

Its public entry point is:

$devpilot-research-agent <your research or optimisation request>

The suite includes specialised skills for:

Intake and Research Contract creation
Orchestration
Coordinator behaviour
Ideation
Executor management
Merge and evaluation
Related-work search
Human-in-the-loop and budget controls
Resume and reporting
Deterministic fallback tools

The skills preserve concepts such as:

Durable session state
Idea Tree memory
B_dev and B_test separation
Coordinator and Executor responsibilities
Worktree isolation
Protected paths
Checkpoint and resume
Final report generation

The repository documents how to install the entire suite into the Codex skills directory and invoke it directly.

The suite is intentionally complementary to the native CLI. The native runtime remains the preferred path for complete production research runs, concurrency, provider execution, event streaming, and the terminal dashboard.

OpenAI support

DevPilot supports OpenAI through multiple paths:

OpenAI Responses API
OpenAI-compatible endpoints
Experimental ChatGPT subscription login
Codex-compatible Agent Skills
Configurable models and reasoning effort

The experimental OpenAI OAuth provider uses a token obtained through devpilot login openai, refreshes that token when needed, and streams responses from the ChatGPT Codex backend while preserving the existing Responses-provider tool and reasoning interface.

This means developers can select the provider path appropriate for their environment rather than being locked to one model vendor.

Additional capabilities

Beyond the Build Week ACP work, DevPilot includes:

Flexible model providers

OpenAI
OpenAI-compatible APIs
Anthropic
Gemini
LiteLLM
Local or hosted gateways such as Ollama, vLLM, DeepSeek, and Qwen-compatible endpoints

DevPilot Learning Layer

The CLI can preserve project-local memories, compress trajectories, and mine reusable skills from previous research runs.

Long-term memory

Optional memory integrations allow previous sessions, evidence, and learned artifacts to be indexed and searched.

Context compression

Large logs, evidence, prompts, and session histories can be compressed to preserve relevant information during long research runs.

DevPilot Reach

Reach provides optional research tools for:

Web pages
Search
GitHub repositories
YouTube transcripts
RSS feeds

Evidence retrieved during a session can be stored with the cycle and task context that produced it.

GitLab Orbit

Optional Orbit integration gives the agent structured repository context, including dependencies, symbols, services, merge requests, pipelines, and impact relationships.

How we built it

DevPilot CLI is primarily built with:

Python 3.10+
Typer
Pydantic
AsyncIO
Git
Git worktrees
Agent Client Protocol SDK
OpenAI Responses API
OpenAI OAuth
GPT-5.6
Codex Agent Skills
Anthropic
Google Gemini
LiteLLM
YAML configuration
JSONL event logs
Pytest
Optional GitLab Orbit
Optional MemPalace
Optional Headroom

The maintained package is published as:

pip install miles-devpilot-cli

After installation, developers can validate the environment with:

devpilot doctor

The repository supports installation through either pip or pipx, as well as editable installation from source.

Installation and testing path for judges

Install from PyPI

pip install miles-devpilot-cli
devpilot doctor

Configure a provider

devpilot setup

Run the included CPU-only example

cp -r examples/algotune_knn /tmp/algotune_knn
cd /tmp/algotune_knn
git init -q
git add -A
git commit -qm baseline
devpilot

The included example asks DevPilot to improve a brute-force nearest-neighbour solver while preserving the reference result. It requires no GPU and is designed to complete quickly.

Start the Build Week ACP runtime

devpilot acp --stdio

The current ACP milestone communicates over JSON-RPC through standard input and output, while runtime logs and child-process diagnostics are kept on standard error.

Install the Codex skill suite

CODEX_SKILLS_DIR="${CODEX_HOME:-$HOME/.codex}/skills"
mkdir -p "$CODEX_SKILLS_DIR"
cp -R skills/devpilot-* "$CODEX_SKILLS_DIR"/

Restart Codex and run:

$devpilot-research-agent try a one-cycle smoke run in this repo.

Challenges we ran into

Turning a terminal runtime into a protocol agent

DevPilot was originally designed as a complete CLI with its own dashboard, process lifecycle, event bus, checkpoints, and interactive controls.

The Build Week extension needed to expose this functionality through ACP without rewriting or duplicating the native runtime.

The solution was to make ACP a bridge that launches and observes the existing CLI.

Mapping research into standard agent events

DevPilot contains concepts that do not map directly to a normal chat conversation:

Idea Tree nodes
Research cycles
Coordinator phases
Executors
Evaluation results
Merge decisions
Pruned hypotheses
Checkpoints

These had to be translated into plans, thoughts, tool calls, progress messages, and completion states that a protocol client could understand.

Streaming a durable append-only log

The event adapter may read a JSONL file while the CLI is still writing to it.

A reader can encounter a partially written final line.

The implementation therefore preserves the previous byte offset and retries incomplete records rather than dropping or corrupting them.

Process cancellation

Stopping a research process differs across operating systems.

The ACP adapter needed to support graceful interruption and escalation across Windows and Unix-like systems without leaving orphaned Executor processes.

Preserving existing behaviour

The new protocol runtime could not break:

The native CLI
Session resume
Research contracts
Git worktree isolation
Event logging
Provider support
Reports
Existing configuration

The extension was added around the existing architecture rather than replacing it.

Preventing information leakage

Runtime events may contain tool arguments or provider metadata.

Before exposing event payloads through ACP, sensitive keys are replaced with redacted values.

Accomplishments that we are proud of

Built a complete autonomous research CLI rather than a one-shot code generator
Implemented persistent Idea Tree exploration
Separated Coordinator and Executor responsibilities
Added isolated Git-worktree experiments
Added development and held-out evaluation discipline
Added checkpoints and resumable research sessions
Added multiple interaction modes
Added multiple model providers
Added an installable Codex Research Agent Skill Suite
Added OpenAI and experimental ChatGPT login support
Added a Build Week ACP stdio runtime
Added persistent ACP session creation, loading, listing, resuming, cancellation, and closing
Added six bounded ACP operating modes
Streamed Idea Tree and research events into structured ACP updates
Added tool-call and Executor observability
Added sensitive-value redaction
Preserved the existing native DevPilot runtime
Included a small CPU-only benchmark that judges can run without special hardware
Published the maintained distribution through PyPI

What we learned

A coding agent does not become autonomous only because it can edit files.

It needs a disciplined loop:

understand → hypothesise → isolate → execute → measure → learn → decide

Without measurement, an agent can produce convincing changes that do not improve the actual system.

Without memory, it repeats failed experiments.

Without isolation, one bad idea can contaminate the repository.

Without a held-out evaluation, it may optimise for the test it has already seen.

Without observability, users cannot understand what the agent is doing.

DevPilot’s Idea Tree, worktree isolation, evaluation discipline, event stream, and reports are all different parts of the same principle:

The value of an autonomous coding agent is not how much code it writes. It is whether it can prove which changes are worth keeping.

The Build Week extension also showed that mature CLI agents can become interoperable without discarding their native architecture.

A protocol adapter can expose sessions, plans, tools, reasoning, and progress while allowing the original runtime to remain responsible for execution.

What is next for DevPilot CLI

Expand ACP support beyond the current stdio milestone
Add richer protocol-native configuration controls
Improve visual presentation of the Idea Tree in ACP clients
Stream evaluation metrics as dedicated structured updates
Add approval gates for high-risk tools
Add stronger workspace and protected-path controls
Improve concurrent Executor observability
Add richer Codex skill installation commands
Add a simpler one-command judge sandbox
Add deeper GPT-5.6 configuration presets
Expand long-term research memory
Improve automatic skill mining from successful sessions
Add more CPU-only benchmark examples
Add CI integration for validated experiments
Generate pull requests automatically for improvements that pass held-out validation

Team

Miles — solo builder

Responsibilities:

Product direction
Autonomous-agent architecture
Idea Tree and research workflow design
CLI engineering
ACP architecture
Codex and GPT-5.6-assisted implementation
Event-stream mapping
Session persistence
OpenAI integration
Agent Skill design
Testing
Documentation
Demo preparation

Built With

codex
git-worktrees
llm-apis
openai
pypi
python
rich
typer