The problem
Improving an AI agent today means sitting in the loop yourself. Run it, watch it fail, tweak the prompt, run it again. You're the optimizer.
Forge closes that loop twice - once at runtime, once at the source level.
What it does
Forge triages incoming GitHub issues: given an issue, it decides the labels, the owner, and the priority.
That sounds mechanical, but every repository has unwritten conventions. Which words a team actually uses for labels. Who owns which subsystem. What they treat as urgent. None of it is in the API schema — it only exists in the data. That's the contextual logic Forge has to learn.
The agent pulls real context from the GitHub API: the issue body, the repo's label vocabulary, similar closed issues with their resolutions, and the reporter's history. Then it answers.
Two learning loops
The inner loop (runtime). After every issue it handles, the agent reflects on what happened and asks what it just learned — not the answer, the general rule. Lessons go into a durable, inspectable store with a trigger describing when they apply, evidence linking back to the tasks that produced them, and a confidence score. On later runs, relevant lessons are retrieved and placed in the prompt before the agent answers. Entries that help get reinforced; entries that hurt lose confidence and eventually retire.
Nothing about the code changes. Only what the agent knows.
The outer loop (source). When memory isn't enough, Forge analyzes its own failure traces, clusters them into patterns, and proposes concrete changes across four surfaces: the prompt, the tools, the memory policy, and the orchestration graph. Each candidate change is dispatched to its own AO worker on its own branch — several competing hypotheses tested in parallel — and only what measurably scores better gets merged.
Results
Two arms, identical code, 14 held-out issues the agent had never seen.
| pass | memory ON | memory OFF |
|---|---|---|
| 0 | 28.6% | 28.6% |
| 1 | 42.9% | 28.6% |
| 2 | 35.7% | 28.6% |
Both arms start identical, because the memory store starts empty. Once lessons accumulate, the memory arm moves to 42.9% — 4 correct to 6 correct, a 50% relative jump — while the control stays pinned at 28.6% across every single pass. Memory grew from 44 to 80 rules, all written by the agent about this specific repository.
How I built it
Every line was written through AO across 24 sessions and 24 merged pull requests. Its isolated worktrees are what let the outer loop test several competing fixes simultaneously without them colliding, and its PR and CI lifecycle is the gate that decides which candidate survives.
Neatlogs traces every run as a span tree so each GitHub tool call, each model call, each retry, with timing attached. It's not just observability here: the failure analyzer reads those traces to see what each intermediate step actually received, rather than guessing from a final output.
GitHub API responses are cached record/replay style, so evaluation scores are fully reproducible and never drift with live data.
What I learned
The bug that cost me the most was silent. Memory was being written and retrieved correctly, and the store was visibly growing, but the retrieved lessons never reached the model prompt. Every metric looked plausible. The tell was cost: passes after the first cost exactly $0.00, which meant every prompt was identical and replaying from cache.
What's next
- Adaptive retrieval depth instead of a fixed k=5 cap
- Larger holdout sets so results clear the noise floor
- Per-step model routing for cost and latency, which the graph schema already supports
Log in or sign up for Devpost to join the conversation.