Reforge

• 

  Candidate Screen

• 

  Implement Screen

Inspiration

Rebuilding a software feature usually starts with documentation or source code. We wanted to explore a different starting point: what if an AI could learn a feature by using the application itself?

Reforge investigates an existing application, experiments with its behavior, reconstructs a feature in a separate editor, and tests whether the result matches the reference.

What it does

Our prototype focuses on image-editing features in miniPaint, starting with Brightness and Contrast.

The “Implement a feature” workflow prepares an isolated editor, then asks an AI agent to discover the reference controls, collect experimental evidence, and build an implementation. Independent evaluation compares reference and candidate outputs and checks that the editor’s existing functionality still works.

The dashboard exposes progress, experiments, candidate revisions, image differences, evaluation metrics, and rejection or acceptance decisions. Portable recordings preserve actual runs for inspection without a running backend.

How we built it

We built a Python learning and evaluation service, browser automation with Playwright, and a React and TypeScript dashboard. An isolated React image editor provides the reconstruction workspace.

The workflow separates investigation, implementation, and verification. Candidates must satisfy pixel comparisons and editor regression checks before acceptance. A visually convincing result alone is not enough.

Challenges we faced

Image-editing behavior involves more than finding a slider. The agent must identify the correct procedure, understand parameter behavior, preserve image handling, and integrate the new feature without breaking the editor.

We encountered candidates that matched pixel cases but failed interface regressions, planner requests outside execution bounds, and Windows filesystem races affecting dashboard snapshots. These failures pushed us to make diagnostics, evidence preservation, and rejection history central to the project.

What we learned

Observing an application can provide useful evidence for reconstructing its behavior—but reliable reconstruction requires controlled experiments and independent verification.

We also learned to distinguish a successful recorded run from repeatable end-to-end reliability. Reforge preserves accepted Brightness and Contrast recordings, while fresh-run reliability remains an active engineering challenge.

What’s next

The long term vision of this is, once web interaction and learning are perfected, to analyze developer tool data. That would allow it to get much more accurate results and potentially understand at a code level how certain features were implemented. We didn't start with this because it would weaken the training data for learning the systems and interacting with them autonomously.

Built With

• codex
• minipaint
• numpy
• openai
• playwright
• python
• react
• typescript
• vite