Forge.

FORGE autonomously engineers AI agents by turning runtime failures into persistent playbooks and evolving better architectures with Pareto gating. Agents don't just run. They evolve.

Comment

🔥 FORGE — The Story Behind Autonomous Agent Evolution

"Agents don't just run. They evolve."

💡 Inspiration: The Fragility of Modern Agents

Every developer building LLM agents eventually hits the same wall: the real world is full of hidden constraints that prompts cannot predict.

When an AI agent interacts with enterprise APIs—Linear, Slack, CRM databases, or GitHub—it does not fail because the model lacks intelligence. It fails because of unwritten operational reality:

  • A Linear endpoint unexpectedly rejects a team slug like "CORE" and demands a 36-character UUID: "550e8400-e29b-41d4-a716-446655440001".
  • A Slack channel policy silently rejects customer alerts unless they are prefixed with [SLA-ALERT].
  • A database transaction fails because a priority field expects integer levels (1, 2, 3, 4) instead of the string "urgent".

The standard industry reaction is manual prompt patching: an engineer inspects logs, manually rewrites instructions, and redeploys. But prompt engineering does not scale across hundreds of tools and millions of execution edge cases.

We asked a fundamental question:

Why are we treating agent failures as bugs to hide, rather than the highest-density training signal we have?

That question gave birth to FORGE—an autonomous agent engineering platform that treats runtime failures as empirical evolutionary data, enabling agents to self-reflect, build persistent playbooks, and evolve their own architectures across generations.

🏗️ How We Built It: The Dual-Loop Engine

We architected FORGE around a biologically inspired Dual-Loop Evolutionary Framework, separating rapid experiential learning from multi-generation structural mutation:

                        Runtime Failure
                               │
            ┌──────────────────┴──────────────────┐
            ▼                                     ▼
       INNER LOOP                             OUTER LOOP
  Experiential Learning                  Structural Evolution
(Discover → Reflect → Store)           (Cluster → Mutate → Pareto Gate)
            │                                     │
            ▼                                     ▼
   Persistent Tool Playbook              Generation Spec (G1, G2...)
  (−66.7% Calls, −75% Latency)              (+62% Composite Score)
            │                                     │
            └──────────────────┬──────────────────┘
                               ▼
               SHA-256 Cryptographic Provenance

1. The Inner Loop: Experiential Tool Learning

Tool Failure ➔ Autonomous Reflection ➔ Persistent Playbook ➔ Zero-Waste Re-execution

When an agent triggers an API error (e.g., HTTP 422 Unprocessable Entity), the runtime captures the entire execution context and dispatches it to a dedicated reflection worker. The reflector extracts the underlying schema invariant and persists a generalized rule into ToolMemory.

ToolMemory = ToolMemory ∪ { trigger, learned_rule, confidence }

On subsequent runs, when the agent invokes that tool, the playbook is injected directly into the tool's working context.

In empirical testing, this reduced tool calls by 66.7% and slashed latency by 75.0%, turning 50% first-run accuracy into 100% second-run accuracy with zero exploratory errors.

2. The Outer Loop & The Pareto Acceptance Gate

Benchmark Traces ➔ Failure Clustering ➔ Architectural Mutation ➔ Pareto Gate ➔ Next Generation

Across complete benchmark suites, failure traces are clustered. FORGE mutates architectural levers such as:

  1. System Directives: Synthesizing precise boundary conditions and edge-case guards.
  2. Planner Specification: Transitioning between direct execution, ReAct, and subgoal-driven planning.
  3. Verifier Mechanics: Enforcing mandatory automated test suites before declaring task completion.
  4. Retry & Recovery Policies: Calibrating resilience based on observed error categories.

To prevent over-engineering, candidate generations must pass a strict Multi-Objective Pareto Dominance Gate.

Candidate generations are evaluated across four competing objectives:

Objectives = ⟨ +Accuracy, +Reliability, −Cost, −Latency ⟩

Candidate Dominates Parent when:

• Candidate is greater than or equal to Parent on ALL metrics, AND
• Candidate is strictly better on AT LEAST ONE metric.

If a candidate increases accuracy by +4% but increases cost by +74% and latency by +39%, the candidate is rejected.

This is intentional.

FORGE isn't programmed to always improve. It is programmed to select the better engineering tradeoff.

3. Cryptographic Provenance Ledger

Trust requires verifiability.

Every event in FORGE—including model invocations, tool results, reflection outcomes, mutation diffs, and acceptance decisions—is cryptographically chained using SHA-256:

event_hash[0] = 0000000000000000000000000000000000000000000000000000000000000000

event_hash[i] =
  SHA-256(
    event_hash[i-1]
    +
    CanonicalJSON(payload[i])
  )

The Command Center provides a real-time provenance inspector that verifies the chain and detects tampering.

The current verified campaign contains 189+ chained events.

4. The Meta-Agent Layer: Agent Orchestrator (AO)

We integrated Agent Orchestrator (AO) as our development harness.

AO coordinates coding-agent work across the FORGE codebase, allowing specialized development work to be delegated, tested, reviewed, and integrated through isolated workflows.

In the final project, AO is clearly separated from FORGE's runtime:

AO
↓
Development Agents
↓
FORGE

while the product itself operates through:

FORGE
↓
Agent Runtime
↓
Enterprise Environment
↓
Benchmark
↓
Evaluation
↓
Evolution

5. Technology Stack & Sponsor Integrations

  • Reasoning Engine: TensorMux (glm-4-7-flash) powers agent planning, execution, and tool interaction.
  • Reflection / Reasoning: OpenAI provides additional reasoning capabilities where configured.
  • Voice Debriefs: Smallest.ai (Waves Lightning v3.1) synthesizes natural spoken debriefs of generational breakthroughs directly in the browser.
  • Development Orchestration: AO coordinates coding-agent workflows used to build FORGE.
  • Backend: Python 3.11, FastAPI, SQLAlchemy 2.0.
  • Database: SQLite locally with PostgreSQL/Supabase support for deployment.
  • Frontend: Next.js 14 App Router and Tailwind CSS.
  • Transport: REST + Server-Sent Events (SSE).
  • Trust Layer: SHA-256 tamper-evident provenance chain.

🧗 Challenges We Faced

1. The "Improvement Trap" & Over-Engineering Bloat

Early mutation strategies suffered from metric myopia: agents could achieve marginally better task performance by adding more planning steps, more tool calls, and more reasoning—but at the cost of dramatically higher latency and token consumption.

We solved this by introducing a multi-objective Pareto acceptance gate.

FORGE now treats accuracy, reliability, cost, and latency as simultaneous optimization objectives.

The lesson was simple:

Sometimes the best evolution is rejection.

2. Dual-Database Parity

FORGE is designed to operate locally with SQLite while supporting PostgreSQL for deployment.

This required keeping the persistence layer database-agnostic and accounting for PostgreSQL connection-pooling behavior in hosted environments such as Supabase.

The result is a development workflow optimized for rapid local iteration without sacrificing production deployment flexibility.

3. Zero-Fabrication Live Telemetry

During development of the Live Execution Console, it was tempting to simulate progress during cold starts.

We deliberately avoided this.

Every visible execution event is backed by actual runtime state and persisted evidence.

The same principle applies to:

  • benchmark scores
  • tool calls
  • token usage
  • costs
  • mutations
  • provenance
  • AO status

If the system cannot verify it, the UI should not pretend it happened.

🧠 What We Learned

Failures Are High-Value Knowledge

An agent failure is not merely a bad outcome.

It often reveals an exact constraint that the agent did not previously know.

When that observation is structured and persisted, the failure becomes reusable knowledge.

Prompt Engineering Has a Ceiling

You cannot indefinitely prompt your way out of structural weaknesses.

Sometimes the correct fix is:

  • better tool selection
  • stronger verification
  • improved recovery
  • better memory
  • different orchestration

FORGE therefore evolves the architecture, not just the wording of the prompt.

Efficiency Is Part of Intelligence

A successful agent that takes ten times longer and costs ten times more is not necessarily a better agent.

Real systems care about:

correctness + reliability + cost + latency.

FORGE makes those tradeoffs explicit.

📊 Empirical Results

Inner Learning Loop — Cold → Warm

Metric Cold / Naive Warm / Memory-Guided Improvement
Tool Calls 6 2 −66.7%
Latency 5.2s 1.3s −75.0%
API Cost $0.0048 $0.0014 −70.8%
Accuracy 50% 100% +50 pts
Exploratory Errors 3 0 −100%

Outer Evolution Loop

Generation G0
Composite Score: 0.3332

        ↓
Failure Clustering
        ↓
Architecture Mutation
        ↓
Pareto Acceptance

        ↓

Generation G1
Composite Score: 0.5396

Improvement: +62.0%

These results come from the recorded FORGE experiments and benchmark runs.

🔮 What's Next for FORGE

FORGE's current architecture opens several future directions:

  • Cross-Experiment Tool Playbook Sharing: Allow verified tool knowledge to transfer between domains and experiments.
  • Autonomous Multi-Agent Specialization: Evolve specialized teams such as architect, debugger, researcher, and verification agents rather than a single monolithic agent.
  • Dynamic Benchmark Synthesis: Automatically generate adversarial tasks based on observed agent weaknesses.
  • Federated Agent Knowledge: Share verified operational playbooks across environments while retaining provenance and trust guarantees.
  • Continuous Agent Optimization: Treat deployed agents as continuously measurable systems rather than static prompts.

🚀 The Core Idea

FORGE turns this:

Agent fails
    ↓
Developer manually patches prompt
    ↓
Deploy again

into this:

Agent acts
    ↓
Failure occurs
    ↓
FORGE analyzes evidence
    ↓
Knowledge is stored
    ↓
Agent improves
    ↓
Benchmark measures the change
    ↓
Architecture evolves
    ↓
Provenance records the decision

Ultimately:

ACT
 ↓
FAIL
 ↓
LEARN
 ↓
REMEMBER
 ↓
ADAPT
 ↓
MEASURE
 ↓
EVOLVE
 ↓
PROVE

FORGE doesn't just run agents. It engineers better ones.

Built With

Share this project:

Updates

Submission history