DownShift

Cheaper model, similar results! A SDK that helps you test different models with a specially tailored harness for a specific task to find a cheaper replacement while keeping similar or better results.

Comment

Inspiration

Most agents I see in production run on the biggest model available, because nobody has time to prove a smaller one would do. Teams pay for Ultra on every step, including steps that only write a search query. I wanted a tool that answers one question with evidence: can a cheaper Nemotron model run my agent, and what do I have to change to make it work? The answer should come as a pull request, not a hunch.

What it does

Downshift is a thin Python SDK plus an optimizer service:

  1. Record. You add the SDK to your agent and it records real runs: every model call, tool call, token count and final answer.
  2. Split. Runs are split 60/40. The optimizer only ever sees the 60%; every number we report comes from the held-out 40%.
  3. Analyze. Nemotron 3 Nano labels each step (plan, search, synthesize...) by kind and difficulty.
  4. Rewrite. Nemotron 3 Ultra rewrites the agent's harness (system prompt, tool descriptions, few-shot examples, per-step model and reasoning settings) for each cheaper target: Super, Nano, and a per-step mix.
  5. Replay. The agent runs again on the target model. Matching tool calls come from the recording, so replays are cheap and reproducible.
  6. Score. Against the task's correct answer when there is one, otherwise with automatic checks plus a blind, randomly ordered pairwise Ultra judge.
  7. Ship. You get a cost vs quality table and a pull request that changes one file: harness.yaml.

The key design choice is that the harness lives in YAML, not in Python. The SDK loads it at runtime, the optimizer writes new versions of it, and the PR stays small and reviewable.

How I built it

  • SDK (Python): harness schema, model table with prices, one provider layer (Nebius Token Factory, Ollama, OpenAI-compatible), tracing and replay.
  • Server (FastAPI + SQLite): storage, the optimizer pipeline, the judge, GitHub PR creation.
  • Web (Next.js): onboarding (pick a folder → generate a harness → record baseline runs), a live Run screen, results, harness diffs and an Open PR button.
  • Bring your own agent: an AgentAdapter interface lets Downshift drive agents it didn't build, through a subprocess in the agent's own environment. I used it on the open-source text2sql-framework, graded against Spider gold SQL.

Cost is always computed from token counts at Token Factory list prices, so the numbers mean the same thing whichever provider served the call:

$$\text{cost} = \frac{t_{in}\cdot p_{in} + t_{out}\cdot p_{out}}{10^6}$$

A cheaper harness is accepted only if it passes a non-inferiority test: the lower 90% bound of the paired success difference against the baseline must stay above $-10$ points. Otherwise it's marked "inconclusive", not "good enough".

Results

  • Demo research agent: Nano kept 94% task success against 100% for Ultra at 94% lower cost. The per-step mix kept 100% at 51% lower cost.
  • An agent I didn't write (text2sql, Spider): on 136 questions no run had ever seen, the Nano harness Downshift wrote matched Ultra (+2.9 pp, 90% CI −1.5 to +7.4) and beat the plain model swap (+7.4 pp, 95% CI +0.7 to +14.7) at 87% lower cost.

Challenges

  • Fooling myself. Early results looked great on 17 held-out runs, where one run is 5.9 points. I added repeated trials, paired comparisons against the plain model swap, racing on unseen tasks, and fresh re-runs of the real agent rather than replays. Some claims didn't hold up, and the README says which.
  • The judge. I hand-checked 108 of the judge's verdicts against the sources: 87% agreement on correctness. But its "supported" score tracked citation formatting, and it passed 3 of 4 clearly wrong answers. That's why I now prefer tasks with correct answers.
  • Small-model quirks. The worst Nano failure on text2sql came from a few-shot example that wrote a tool call as plain text. Nano copied it faithfully. Turning thinking off alone halved Nano's pass rate.
  • The optimizer isn't always reliable. One run on a larger sample kept the plain swap instead of finding the better harness. After two fixes, three repeat runs each found a non-inferior Nano harness on their own.

What I learned

Smaller models rarely fail on intelligence. They fail on format. Most of the gap between Nano and Ultra was output format and tool-call shape, and a well-written harness closes it. I also learned that evaluating an optimizer honestly is harder than building one.

Built With

Share this project:

Updates

Submission history