SkillFoundry

Agents that learn the tools, not just the task.

Comment

Inspiration

AI agents can understand a task and still fail because they choose the wrong tool, use an incorrect parameter, or fail to reuse information discovered earlier.

That led us to a simple question:

Can an agent turn its execution failures into reusable skills and prove that those skills actually improve future runs?

That idea became SkillFoundry.

What We Built

SkillFoundry is a trajectory-based agent-learning system for tool-using agents.

Instead of simply storing conversation history, SkillFoundry learns from execution experience:

FAIL → UNDERSTAND → LEARN → REPLAY → MEASURE → PROMOTE

Given a goal, available tools, and an evaluator, the system:

  1. Executes the task.
  2. Records the complete execution trajectory.
  3. Evaluates the result.
  4. Analyzes the failure.
  5. Synthesizes a reusable skill.
  6. Replays that skill on a fresh trajectory.
  7. Measures the improvement.
  8. Promotes the skill only when the evidence shows improvement.

How It Works

GOAL
  ↓
AGENT
  ↓
TOOLS
  ↓
TRAJECTORY
  ↓
EVALUATOR
  ↓
FAILURE ANALYZER
  ↓
SKILL SYNTHESIZER
  ↓
SKILL
  ↓
REPLAY
  ↓
PROMOTION GATE
  ↓
VERIFIED SKILL
  ↓
BETTER NEXT RUN

The key idea is that a generated skill is not automatically trusted. It must be replayed and demonstrate measurable improvement before promotion.

Benchmark

We built a deterministic GitHub CI investigation benchmark.

The agent investigates a failed CI workflow using:

list_workflow_runs
        ↓
inspect_job_logs
        ↓
inspect_commit
Run 1 — Baseline

The baseline reaches the final commit inspection, but the commit lookup fails because it uses an invalid SHA.

Baseline score: 0.6250

Tool calls: 3

Failure type: tool_failure

Learning

SkillFoundry analyzes the recorded trajectory and synthesizes a reusable procedure:

list_workflow_runs
        ↓
inspect_job_logs
        ↓
inspect_commit

It also learns actionable parameters from successful evidence already present in the trajectory:

inspect_job_logs.run_id   = 2
inspect_job_logs.job_name = test
inspect_commit.sha        = def456ghi789

The learned information retains provenance to the source trajectory.

Run 2 — Guided Replay

The synthesized skill is replayed on a fresh trajectory using the learned procedure and parameters.

Replay score: 1.0000

Tool calls: 3

The measured improvement is:

$$ 0.6250 \rightarrow 1.0000 $$

with:

$$ \Delta score = +0.3750 $$

and:

$$ \Delta tool\ calls = 0 $$
Promotion

The Promotion Gate requires:

$$ ReplayScore > BaselineScore $$

and:

$$ ReplayToolCalls \leq BaselineToolCalls $$

For this benchmark:

Improved: True

Promoted: True

The skill is therefore promoted because it improved quality without increasing tool usage.

Technical Implementation

The project is split into a core execution layer and a learning layer.

Core Execution Layer
Task and tool models
Agent Executor
Deterministic GitHub Simulator
Trajectory Recorder
Trajectory Evaluator
Learning Layer
Failure Analyzer
Skill model
Skill Synthesizer
Replay Engine
Promotion Gate
Learning Loop

Together they implement:

trajectory
    ↓
failure analysis
    ↓
skill synthesis
    ↓
replay
    ↓
measurement
    ↓
promotion

The project uses AO as the agent orchestration environment with OpenCode and TensorMux / GLM-4.7-Flash.

The benchmark itself is deterministic and offline so the learning result is reproducible and auditable.

Challenges

One of the most important challenges was making sure the benchmark demonstrated genuine learning rather than a hidden replay shortcut.

Our initial implementation appeared to improve from Run 1 to Run 2, but investigation revealed that replay was benefiting from a demo-only fallback that supplied a valid parameter.

We removed that fallback and changed the learning path so that the replayed skill explicitly carries parameters extracted from actual trajectory evidence.

This made the improvement traceable:

failure
   ↓
trajectory evidence
   ↓
learned procedure + parameter
   ↓
fresh replay
   ↓
measured improvement

Another challenge was keeping the system deterministic enough to validate the learning loop reliably while still supporting an agent-oriented orchestration workflow.

What We Learned

The biggest lesson was that agent learning needs more than memory.

A useful learning system needs to capture:

WHAT FAILED
     ↓
WHY IT FAILED
     ↓
WHAT SHOULD CHANGE
     ↓
WHAT WAS LEARNED
     ↓
DID THE NEXT RUN IMPROVE?

Generating a skill is not enough. The skill should be replayed and measured before it is trusted.

Results

The repository currently contains:

113 passing tests

0 failing tests

Benchmark result:

Baseline:       0.6250
Replay:         1.0000
Improvement:   +0.3750
Tool calls:      3 → 3
Promoted:        True
Why It Matters

Traditional agent memory mainly asks:

"What happened before?"

SkillFoundry asks:

"What failed, why did it fail, what should change, and did that change actually improve the next run?"

The goal is to turn verified execution experience into reusable operational skills that can generalize beyond the current GitHub CI benchmark to engineering operations, project management, finance workflows, approvals, and other tool-driven environments.

Core Insight

Agents should learn how to use the environment, not just remember the conversation.

SkillFoundry turns execution failures into testable skills and asks the question that matters most:

Did the next run actually get better?

Built With

Share this project:

Updates

Submission history