Synapse

Your LLM bill, cut 30x. Describe a task and an AI agent builds a tiny specialist that beats Nemotron Ultra at its own job, for pennies per million calls.

Comment

Inspiration

Most of what companies send to large language models isn't hard. It's the same narrow question asked millions of times: which spending category is this purchase?, which team should get this ticket?, is this log window normal? Teams pay frontier-model prices for every one of those calls, because building something smaller means choosing a method, finding data, testing it honestly, and maintaining it.

We kept coming back to one analogy: a senior doctor doesn't run every screening test personally. They train a nurse to do that one test, and they check the nurse's work. We wanted an agent that does the same with models: the big model (NVIDIA Nemotron Ultra) plans, teaches and grades, and a small specialist does the repetitive work.

What it does

You describe a repetitive task in a chat. Nemotron Ultra asks up to seven clarifying questions (which mistake is worse, how many records a month, latency budget, whether it may call the cloud), and every answer visibly changes the plan. Then Synapse:

  1. Searches a ladder of candidates, cheapest first: keyword rules, TF-IDF with linear models, LightGBM, embedding models, prompted Nemotron Nano, Lightning and Super, and cascades, where a cheap model answers the confident cases and escalates the rest to Nemotron.
  2. Tests honestly. A sealed test set is split off before anything else happens; only the Evaluator agent can read it, once per version.
  3. Ships the cheapest specialist that meets your bar, as its own versioned API with its own key.
  4. Improves it safely. Your corrections retrain it, and the new version goes live only if it beats the old one on the same sealed test set. Rollback is one click.
Task Specialist vs Nemotron Ultra (same sealed records) Yearly cost at the task's volume
Procurement categories (19 classes, 2M records/month) macro-F1 0.627 vs 0.456 \$621 vs \$19,363
Support routing (77 intents, 500k/month) macro-F1 0.913 vs 0.738 \$0.72 vs \$6,175
Log anomalies (5M windows/month) recall at 1% FPR 1.000 vs 0.000 \$0.21 vs \$104,780

Each whole build, including the Nemotron Ultra baseline, cost between \$0.07 and \$0.31 of Token Factory credit.

How we built it

Four agents, one plain-Python orchestrator. The Data, Planner, Trainer and Evaluator agents never talk to each other directly. The orchestrator runs them in a fixed order, and every handoff is a JSON file you can open from the UI. Nemotron Ultra plans the search with strict JSON-schema output, writes targeted training examples during repair rounds, analyzes errors, and is the zero-shot baseline every specialist is measured against.

Search is successive halving by data fraction: every candidate trains on 10% of the data, the top quarter on 40%, and the top two or three on 100%.

Cost is measured, never assumed. For a cascade whose cheap stage answers a share $s$ of items:

$$ C_{\text{1M}} = s \cdot C_{\text{cheap}} + (1 - s)\left(\bar{t}{\text{in}}\, p{\text{in}} + \bar{t}{\text{out}}\, p{\text{out}}\right) \times 10^{6} $$

where $\bar{t}$ are the measured mean tokens per escalated call and $p$ the Token Factory list prices per token. The yearly projection is $C_{\text{year}} = C_{\text{1M}} \cdot \frac{12V}{10^{6}}$ for the monthly volume $V$ on the task card.

Stack: Nemotron-3 Ultra, Nano, Lightning and Super plus Qwen3-Embedding-8B on Nebius Token Factory; the shared specialist server on a Nebius Serverless CPU endpoint; model bundles and run folders in Nebius Object Storage; FastAPI, scikit-learn and LightGBM on the backend; React, Tailwind and Recharts on the frontend, with a live Pareto chart streamed over server-sent events; Supabase for Postgres and sign-in. We used Claude Code as a coding agent, working from a written spec.

Challenges we ran into

  • Hidden reasoning tokens. Nemotron Lightning spent 196 billed tokens to answer "ok", and with reasoning on it leaked its thinking into the visible answer. Setting enable_thinking: false cut a one-word classification on Nano from 85 tokens to 3. That one flag is what makes prompted Nemotron affordable as a cascade stage.
  • Sandboxes weren't available. Our account had no spawn permission for Token Factory Sandboxes, so candidate training runs as parallel local processes instead: same scripts, different launcher.
  • Dirty public data. Our first 28,900 USAspending rows held only 6,127 unique awards, because long-running contracts matched every month's date filter. Filtering on the signing date fixed it: 42,000 unique awards.
  • Saturating metrics. On the logs task almost every candidate hit recall 1.000 at 1% FPR, so search rounds were eliminating by tie order. We added PR-AUC as a tie-breaker.
  • Calibration can backfire. Temperature scaling made an embedding kNN model's confidence error worse (0.060 → 0.241), so calibration is now kept only when it measurably helps.
  • Rate limits. Nano allows 100 requests a minute on our account, so one improve run's final exam took 12 minutes.
  • Keeping the benchmark fixed. Our first "improve an already-improved version" couldn't find the sealed test set. Every version now traces back to the original build's sealed set, so v3 is graded on exactly the same records as v1.

Accomplishments that we're proud of

  • The specialist beat Nemotron Ultra's zero-shot score on all three tasks, on the same sealed records, at a small fraction of the cost.
  • Two real validated updates from held-back corrections, both through the version gate: procurement v2 → v3 (0.655 vs 0.647 on 3,368 sealed records) and support v1 → v2 (0.935 vs 0.933). Worse versions are rejected automatically.
  • Every number in the UI and README traces back to a file in the run folder.

What we learned

A frontier model is a great teacher and an expensive worker. Nemotron Ultra was excellent at asking the right questions, planning the search and explaining errors, but a TF-IDF model trained on the same data out-scored its zero-shot answers on every task. The best systems we built use both: the cheap model handles the confident majority, and Nemotron handles only the hard cases.

We also learned how easy it is to fool yourself with evaluation. Choosing among many candidates inflated validation

Built With

Share this project:

Updates

Submission history