Dreaming

Dreaming learns every call your AI agent makes and skips the ones it can predict. Your agent gets cheaper and faster the more it runs.

Comment

Dreaming

Your agent's next call is usually one it has made before. Dreaming notices, learns it, proves it, and stops paying for it.

Live: dreaming-demo.fly.dev · Benchmark: /benchmark.md · Agent on ASI:One: dreaming-bookkeeper

Track: Model Learning. Company challenges: Fetch.ai · StepFun · Intuit (QuickBooks System One) · CREAO · veri.studio · Sentry · Cognition (Devin) · Paper. Each has its own section below, with evidence.

Inspiration

Production agents repeat themselves. A bookkeeping agent categorizes the 400th Notion charge the same way it categorized the first 399, and every time it pays a frontier model 2.6 seconds and real money to do it. It calls a vendor API whose answer has not changed in months. When that API goes down, the whole agent goes down with it.

We built for the people who run agents in production, the ops and platform teams who pay the bill, get paged when a vendor API dies, and have to explain a wrong decision after the fact.

The obvious fix is to cache or fine-tune. Neither is safe to ship on its own: a cache cannot generalize to a merchant it has never seen, and a fine-tuned model gives you an answer with no way to know how often it is wrong. Nobody wants to explain to an accountant that the books are "probably" right.

The name comes from the Dreamer line of reinforcement-learning research, where an agent learns a model of its world and plans inside it. We asked what happens if an agent learns a model of its own calls, and only answers from that model when it can prove the answer is good enough.

What it does

Dreaming is a runtime you wrap around an existing agent. It does not replace the agent's model or its tools. It learns them.

  1. Log. Every call (a frontier-model step, a tool call, a routing choice) writes one trace row: input, output, latency, cost, and who answered.
  2. Train. For each call site, Dreaming trains a tournament of small predictors on those rows, in parallel, and runs them all in shadow next to the real call.
  3. Certify. Each predictor is scored on shadow rows it never trained on. A Clopper-Pearson bound finds the confidence cutoff above which it is wrong at most N% of the time with 95% confidence, where N is a budget the operator sets.
  4. Promote. The candidate with the most certified coverage starts answering first. Below its cutoff, the real call still runs. One confident answer in twenty is still sent to the real call as a spot check, and the certificate is revoked the moment those audits break the budget.
  5. Replay. Every answer a hosted predictor gave can be re-run later, byte for byte, because the certificate names the exact weights, environment and teacher it was proven against.

The pitch in one sentence: every saving comes with a certified error bound, and every decision can be replayed.

The demo agent is a nightly bookkeeper: bank-feed line → categorize → vendor lookup → match invoice → anything odd? → post to the books, or wake a human. On the live dashboard you can:

  • Run 2,000 transactions and watch each step move from the big model to a "dream": cost per run, seconds per run, failed runs and the replayable share are charted as the agent learns.
  • Drag the error budget and watch every step re-certify in milliseconds; each step shows its promise ("wrong at most 5%"), the evidence behind it, and which candidate is serving.
  • Unplug the vendor API and watch the agent keep posting: certified lookups are answered by the learned tool model, the rest fall back to a snapshot.
  • Replay any past run: steps a hosted dream answered come back with the same bytes; frontier steps are read from the log.
  • Chat with it from ASI:One: send NOTION *WEB 96.00 and get back a receipt that names, per step, whether the real call or a dream answered, with the dream's weights hash and certificate.

The numbers

Measured on held-out data, from the live benchmark table. The teacher is StepFun step-5-preview.

  • categorize (a frontier-model step): 89.2% of calls answered by a dream on merchants it never saw, certified at a 5% budget, 0.0% realized error on that slice. Projected from the logged StepFun calls: cost per 1,000 calls $0.59 → $0.09 and median latency 2,578 ms → ~5 ms. The winning candidate is a memory seeded from LLM-written examples, so it certified before a single real row was logged.
  • match_invoice: 78.3% of repeat traffic answered by a dream at 2.3% realized error. Projected cost per 1,000 calls $1.75 → $0.72; median latency 5,020 ms → 5.4 ms.
  • QuickBooks bill → account, on companies held out entirely: 94.3% coverage at a 5% budget, 1.8% realized error (3 seeds, Intuit's 100-company dataset).
  • The live site after a 2,000-transaction batch on Fly.io: cost per run 0.68¢ → 0.55¢, replayable decisions 0% → 28%, 0% failed runs, 10,000 trace rows. Replayed runs come back byte-identical ("3 of 4 decisions re-ran with the same bytes, the rest came back from the log").
  • Honest about latency: on Fly's shared CPU, seconds per run rose from 15.8 s to 17.4 s, because every dream embeds its input on a shared core (about 300 ms). A dedicated core or the embedding cache brings that to the ~5 ms above.
  • What it cost to build: the 31-cell training campaign ran for $0.06 on Modal. Labelling 8,219 StepFun answers cost $6.88. The whole build's compute and API spend was about $25.

We publish where it breaks too. On merchants never seen, the match_invoice dream's realized error was 15.6%, three times its budget. That is the distribution-shift case the 1-in-20 audits exist for: once the audits exceed the budget, the certificate is revoked and the site goes back to the real call. Our runtime tests cover that revocation and failover.

How we built it

We tested before we built. Before writing the product we ran 32 feasibility experiments ("smokes") on rented GPUs and real APIs, each with a written result and a critic's pass. They decided the architecture:

  • A 33M-parameter encoder (bge-small) beat every 0.5B decoder we tried as a student, at every data size, for a fraction of the training time.
  • Naive "empirical error ≤ budget" rules break the budget about 45% of the time in simulation. We switched to a fixed-grid Clopper-Pearson certificate, which held on held-out data in every experiment (0.4-3.3% realized error under a 5% budget).
  • The teacher disagrees with itself. 200 identical calls to StepFun step-5-preview disagreed 3.5% of the time. No certificate can promise less than that, so the API refuses any budget below the teacher's measured floor.
  • Hypernetworks lost. Three weight-generator designs lost to a plain label-embedding baseline on unseen tasks; a day-zero student from 10 LLM-written examples per label won instead ($0.005 per 122 labels).
  • Learned tool models keep agents alive. In a synthetic outage, 89% of runs completed using the learned vendor tool versus 0% with a circuit breaker.
  • CPU inference at batch 1 with pinned threads is deterministic: 1,000 concurrent calls gave 1 distinct output. That is what makes byte-exact replay a promise rather than a hope.

The library (packages/dreaming) is pure Python with no network or I/O except through three interfaces (Store, Teacher, Trainer). An agent opts in with decorators and keeps its own code:

@step("categorize", labels=list(CATEGORIES), descriptions=CATEGORIES, teacher=TEACHER, thresholds=T)
def categorize(txn):
    return f"Transaction: {txn['descriptor']}, ${txn['amount']:.2f}"

@tool("vendor_lookup", fallback=tools.vendor_snapshot, price_call=0.004)
def vendor_lookup(merchant):
    return tools.vendor_api(merchant)

Agents that cannot change code get the same loop through connectors: an OpenAI-compatible proxy (a one-line base-URL change), an OpenAI client wrapper, MCP middleware, and a uAgents wrapper.

Each call site runs its own state machine: awake (the real call answers) → watching (dreams run in shadow) → testing (certifying) → dreaming (promoted). The router decides, per call, between called, predicted, audited and fallback.

The candidate tournament. Every site trains all of these in parallel and the certificate picks the winner; the rest stay as hot spares:

  • memory: a softmax-weighted vote over stored (embedding, label) rows, seeded with LLM-written examples. Learning is appending a row, forgetting is deleting one, and every answer can show the rows that voted for it.
  • head: frozen bge-small (384-d) plus a linear head, fit per site in seconds on CPU.
  • finetune: the same encoder unfrozen, trained on a T4 (Modal) or an L4 (veri.studio).
  • hdc: hyperdimensional computing with one integer bundle per class over 10,000-bit codes. After the encoder it is pure integer arithmetic, so the same input gives the same bytes on any machine.
  • exact: memoized tool returns. config: routes a step to a cheaper configuration of the same model (thinking off) when the two agree.

Making the certificate honest took more work than training:

  • Calibration rows are shadow rows the weights never trained on, collected before promotion, and split by group (merchant or company) so repeat inputs cannot leak.
  • With N candidates on one calibration slice, each is certified at δ/N, so picking the winner does not spend the same confidence twice.
  • After a revocation, a spare is promoted only if its own shadow predictions on the audited rows pass the gate. Its old certificate came from the slice the traffic just drifted away from.
  • Confidence is quantized to 4 decimals before it is thresholded or hashed, so replay compares exact bytes.
  • Outages are detected, never declared: three consecutive failures mark a tool down, one call in twenty keeps probing, and only tools are short-circuited. A model step always reaches its teacher.

Certification is a pure function over stored shadow rows, not a training job. That is why dragging the budget slider re-certifies every site in milliseconds.

Infrastructure.

  • Fly.io runs the FastAPI service and the Fetch.ai uAgent: control routes, an OpenAI-compatible proxy, /v1/systemone, /benchmark, and the dashboard.
  • Supabase Postgres is the single source of truth (trace rows, policies, certificates, weights, training runs). Supabase Realtime pushes changes to the page.
  • Modal trains candidates on demand and runs the offline campaign: a matrix of (site × candidate × seed × rows seen) cells on CPU or T4, with a monitor cron that writes checkpoints back to Postgres.
  • Sentry receives tracing with hand-made gen_ai.invoke_agent, gen_ai.chat and gen_ai.execute_tool spans per run and per routed call, plus Logs.
  • Every swappable part (candidate, teacher, encoder, trainer, store, telemetry sink, serving format) sits behind one Protocol and one registry. Adding a new candidate means one module with two functions, not a refactor.
  • One Pydantic contracts file is the vocabulary. The SQL migrations mirror it column for column and it generates the page's TypeScript types. Pyright runs in strict mode; 195 tests pass.

The dashboard is a hand-drawn "Lucid Doodle" design: dot-grid paper, wobbly ink borders, washi tape and a sleeping cloud logo. It is vanilla JS served by the API, and the dreams tab was redesigned in Paper. Every element on it answers one question an ops team would ask: what is dreaming, what did it save, what did it promise, what can be replayed?

Challenges we ran into

  • Teachers are noisy. Even at temperature 0, a frontier model disagrees with itself (3.5% for StepFun, up to 11% for other models we measured). An error budget below that number means nothing, so every certificate carries the teacher's measured floor.
  • Generalization looked great until we held out the right thing. Random row splits let repeat merchants leak into calibration; in our tool experiment, leaving out the real rate of novel inputs inflated coverage by about 30 points and produced confident wrong answers. Holding out whole merchants and companies made our numbers smaller and true.
  • Determinism is fragile. Dynamic padding gave 8 different byte strings for the same text, and changing the thread count changed the bytes. We pinned batch 1, one thread and CPU, and quantize confidence before hashing.
  • Frontier rate limits shaped the schedule. StepFun voucher accounts allow 100 requests per minute, so a 2,000-run labelling pass took about 2.5 hours per seed. We recorded answers once and replayed them, labelled as recorded on the page.
  • Infrastructure surprises at 4 AM: our laptop network blocked Postgres ports, so all database work went through Modal and Fly. An idle-in-transaction connection silently dropped 2,900 runs until we serialized the store's connection.

Accomplishments that we're proud of

  • A frontier step answered 89% of the time on unseen merchants, under a certified 5% error bound, at a projected 15% of the cost and a fraction of the latency.
  • Byte-exact replay of learned decisions on the deployed stack.
  • A certificate that can say no. It refuses budgets it cannot measure, revokes itself under drift, and the benchmark publishes the cells where it failed.
  • Six candidate families, from a Hopfield-style memory to 10,000-bit hypervectors, competing per call site under one statistical gate.
  • Eight sponsor integrations that each do real work in one coherent system, for about $25 of compute and API spend.

What we learned

  • The binding cost of distillation is teacher labels for calibration, not compute. About 200 calibration rows per call site certify most of a good student's coverage; training takes seconds.
  • Small encoders beat small decoders for typed agent steps, by a lot.
  • A certificate holds only on traffic like the traffic it was calibrated on. Audits are not optional.
  • Agents are not just model calls. Tools, routing choices and human reviews are all call sites, and the same log → shadow → certify → promote loop works on every one.

What's next

  • Human-review deferral: certify against reviewer corrections so the agent wakes a human only when it should. Our experiment caught 80% of reviewer changes while reviewing 24% of items.
  • Context pruning for typed steps: our learned span scorer removed 82% of prompt tokens at 92.5% agreement.
  • Running Dreaming on real customer traffic, where repeats are the norm and the savings compound.

Built With

  • agentverse
  • asi-one
  • creao
  • devin
  • fastapi
  • fetch.ai
  • fly.io
  • hugging-face
  • javascript
  • modal
  • numpy
  • paper
  • postgresql
  • pydantic
  • python
  • pytorch
  • quickbooks
  • sentence-transformers
  • sentry
  • stepfun
  • supabase
  • typesafe
  • uagents
  • uv
  • veri.studio
Share this project:

Updates

Submission history