Skill Forge

Your agent learns a skill only when it can prove it helps: Nemotron on Nebius Token Factory distills a SKILL.md, then a replay must beat the no-skill run before it is saved.

Comment

What it does

Skill Forge is a plugin for Hermes Agent. It lets your personal agent learn a reusable skill, but only after the skill proves it helps.

After a multi-step session, NVIDIA Nemotron-3-Super on Nebius Token Factory distills the session trace into a candidate SKILL.md. Skill Forge then replays the task in a scratch copy with the skill loaded. The skill is saved only if the replay passes its checker in fewer tool calls. Otherwise it is discarded and nothing is written. Skills stay as plain markdown files on your machine. Only the redacted task trace is sent to the model.

Track: Personal AI. Persistent memory and reusable skills for a private assistant, running on Hermes Agent with an NVIDIA open model.

Why

Agents that save skills automatically never check whether those skills help. Some add steps or encode a wrong shortcut, and the library fills with noise. Skill Forge adds a test: a skill has to beat the no-skill run before it is kept.

Results: live on Nebius Token Factory

10 tasks, each trained on one seed and tested on 3 held-out seeds (30 episodes per arm). Model nvidia/nemotron-3-super-120b-a12b at api.tokenfactory.nebius.com.

arm median tool calls mean success
A: no skill 4.0 5.07 73%
B: auto-saved, unverified 3.0 3.80 97%
C: Skill Forge (replay-verified) 2.0 3.43 87%
  • Median tool calls cut by 50% compared with no skills, while success rises from 73% to 87%.
  • The replay gate kept 7 skills and discarded 3 that did not reduce calls.
  • Honest caveat: B had higher success in this run. The gate's measured gain over B is in calls (mean 3.43 vs 3.80) and in a smaller library, not in success.
  • Whole run: 667 Token Factory calls, 852k input + 501k output tokens, about $0.91.

Reproduce in about 3 seconds with no key: make judge-demo replays the recorded Token Factory responses through the real harness. The sandbox, the tools and the checkers all run for real.

How it works

  1. A post_tool_call hook records every terminal call in the Hermes session.
  2. /forge <name> sends the redacted trace to Nemotron-3-Super on Token Factory, which writes SKILL.md.
  3. Skill Forge replays the task in a scratch copy with the skill loaded, and the task checker runs.
  4. Fewer calls and the check passes → save to ~/.hermes/skills/forged/<name>/. Otherwise → discard.
  5. /forge-status shows the forged skills and the verdict ledger.

How we used NVIDIA and Nebius

  • NVIDIA Nemotron-3-Super (120B-A12B) does both distillation and the agent loop. It handled tool-calling JSON well across 667 calls.
  • Nebius Token Factory serves every live call through its OpenAI-compatible API (SF_BASE_URL=https://api.tokenfactory.nebius.com/v1). Swapping endpoints took one environment variable, and running ten workers in parallel got the whole three-arm experiment done in about 15 minutes.

Feedback on Nebius Token Factory

  • Promo credits need a card on file before the code can be applied. For a hackathon, a card-free sandbox for promo credit would remove a real barrier.
  • The /v1/models catalog was accurate and listed every Nemotron variant, which made picking a model easy.
  • A per-request cost header would make cost reporting simpler. We had to compare the balance before and after the run.

What was built during the submission period

The whole project: the engine, the three-arm harness, the Hermes plugin and the Token Factory run.

Built with

Python, Hermes Agent, NVIDIA Nemotron-3-Super, Nebius Token Factory. MIT licensed.

Built With

Share this project:

Updates

Submission history