Scar

Agent that improves by retraining its own weights on its failures

Comment

Inspiration

Most "self-improving" AI agents only rewrite their own prompt; the model itself never changes. We wanted an agent whose mistakes change its weights, with a referee strict enough to refuse a model that only got better at the questions it practiced.

What it does

Scar Tissue is a bookkeeping agent that retrains itself from its own failures. You ask it a question about one company's books (total spend in an account, duplicate payments, an accrual entry), and it answers with tools that can only see that company's ledger. Code checks every answer against a value computed from the ledger with SQL, so no LLM ever grades another LLM.

Each generation of the loop:

  1. Run: the agent answers 200 practice questions. Sentry traces every run and groups the failures into issues.
  2. Triage: Jev reads each failure and decides why it happened and how likely a better model is to fix it.
  3. Practice: each fixable failure becomes 5 new questions from the same template at other companies, with answers computed by SQL. The model attempts each one several times, and we keep only its passing transcripts.
  4. Retrain: Veri fine-tunes a LoRA adapter on those transcripts and serves it next to the base model on one multi-adapter deployment.
  5. Gate: the candidate $c$ replaces its parent $p$ only if it does better on 300 questions from 30 companies it never trained on, and does not fall behind on the tasks the parent already solved:

$$\text{promote}(c) \iff \text{eval}(c) > \text{eval}(p) \;\wedge\; \text{reg}(c) \ge \text{reg}(p) - 0.05$$

Here $\text{reg}$ is the pass rate on up to 100 training tasks the parent passed, with the parent re-run on the same tasks in the same gate.

How we built it

  • Model: Qwen3-4B-Instruct-2507 with tool calling, served by Veri. It starts at $46.3\%$ on the held-out companies.
  • Ledger: the QuickBooks hackathon dataset loaded into Postgres (159,790 expense rows across 100 companies, reconciled to each company's profit-and-loss report to the cent). Row-level security lets the agent's SQL tool see one company at a time.
  • Tasks: 12 question templates in three families (aggregation, anomaly detection, bookkeeping entries). Every expected answer comes from SQL.
  • Services: three FastAPI services over one Postgres database: an environment (ledger, tools, verifier, task generation), an agent (agent loop, Sentry spans, Jev mining, judge UI backend) and a trainer (data builder, Veri jobs, promotion gate, loop runner). A Next.js app gives judges a dashboard, an "ask anything" page and a run viewer.
  • Sponsor tools:
    • Sentry: AI agent monitoring on every run (model and tool spans), failure issues grouped by template and failure mode, logs from all three services, and Session Replay on the judge UI.
    • Veri: every weight update is a Veri managed LoRA job, and every model version is served from one Veri multi-adapter deployment (about 26,000 requests during the event).
    • QuickBooks: the dataset, plus Jev, which triages each failed run in the loop's mining step and picks the template for judges' questions.

Challenges we ran into

  • With our first system prompt, the base model typed submit_answer(number=...) as plain text instead of calling the tool and passed 0 of 30 seed tasks. Describing the call as a JSON tool call took it to 12 of 30.
  • The SQL tool's row-level security could be escaped by calling set_config from inside a query, so we revoke that function from the tool's database role.
  • Veri's managed fine-tuning trains on whole conversations by default, so we export one training row per assistant turn. That keeps the loss on the model's own tokens and matches the exact text the inference server shows the model.
  • One GPU serves every agent run. When two gates ran at once, half the model calls timed out, so lineages now take turns on the replica one batch at a time, and runs lost to infrastructure are retried before they count against a model.
  • Our first "no forgetting" rule required $95\%$ on tasks the parent had already passed, but the base model itself re-passed only $81/90 = 90\%$ of them at temperature 0. The gate now re-runs the parent on the same tasks and compares against that.

Accomplishments that we're proud of

  • Weight updates nearly doubled accuracy on unseen companies: from $46.3\%$ to $72.7\%$ in two generations (seed 0).
  • Both prompt-only methods made the agent worse on the same failures: GEPA $43.0\%$, failure memory $8.3\%$.
  • The referee said no to us. Our first adapter scored $70.7\%$ but was rejected: on tasks the base had solved it scored $86.7\%$ against the base's $93.3\%$, breaking 9 and fixing only 3. The second adapter trained on both generations' data, beat the base there too ($92.7\%$ vs $88.5\%$), and shipped.
  • On the held-out set, the promoted model fixed 96 questions the base got wrong and broke 17.
  • The first adapter improved every task family: aggregation $29\% \to 79\%$, bookkeeping $61\% \to 79\%$, anomaly $49\% \to 55\%$.
  • No LLM labels anywhere: every score comes from answers checked against SQL.
  • About 590 automated tests across the five packages, including database tests of the gate and the loop runner.

What we learned

  • A 4B model picks up tool-calling habits from a few hundred of its own passing transcripts: the first adapter stopped typing the answer tool as text and gained 50 points on aggregation questions.
  • Prompt-only memory can backfire on a small model: lessons written by a larger model quoted submit_answer({"number": X}) as text, and the 4B model copied the text instead of calling the tool.
  • A referee only means something if it can reject your best-looking result.
  • Measure noise before you set a threshold: the same model at temperature 0 swung by 3 of 90 tasks between two runs.

What's next for Scar

  • More seeds and generations: run seeds 1 and 2 for error bars, plus generation 3, whose judge failures Jev has already triaged.
  • A fairer comparison: give failure memory and GEPA the same number of generations as the weight-update arm.
  • Keep long transcripts: raise the training context beyond 4,096 tokens; $15\%$ of per-turn training rows were dropped for length.
  • Tune Jev's threshold: our spot check found Jev agreed with us on the cause of 17 of 20 failures but skipped 6 fixable ones, so we want to tune its inclusion threshold.
  • Faster turnaround: cut a full generation from about 2 hours toward minutes, so a judge's failed question can come back fixed during the same demo.

Built With

Share this project:

Updates

Submission history