Pacer

• 

  dashboard

Inspiration

Coding agents make you choose a model, and most people pick the most expensive one (Opus) for everything. Existing routers either pick a model per request without knowing anything about your work, or just cap spending. We wanted an agent that asks: what is the cheapest model that actually passes the tests?

What it does

Pacer is a budget-aware model planner for coding tasks:

1. Plan: a cheap model sizes up the task. Pacer splits it into steps and assigns Haiku, Sonnet or Opus to each step within your budget, keeping 30% in reserve.
2. Implement: the chosen model writes the code. Steps pass short handoff notes instead of full chat histories.
3. Test: hidden tests run in an isolated Daytona sandbox. If they pass, Pacer moves on. If they fail, that step upgrades to the next model, along with the failing test output.
4. Learn: every step is traced to Arize. Before each plan, Pacer reads those traces back to find the cheapest model that passes this kind of step at least half the time. Arize is Pacer's memory.

You can also describe your own task. Pacer splits it into steps, writes hidden tests for each step and a reference solution, and checks in Daytona that the solution passes and the starter fails. When a run finishes, you get a downloadable, runnable project with a demo script that Pacer has verified.

How we built it

• AWS Bedrock runs Claude Haiku 4.5, Sonnet 5.5 and Opus 5.5, accessed through the Vercel AI SDK.
• Daytona sandboxes are the quality gate: every attempt is tested there, and the result decides whether to upgrade.
• Arize AX stores OpenInference traces of every step (model, tokens, cost, test result) and LLM-as-judge code-quality evals. Pacer queries the traced test results through the Arize export API to decide where to start next time.
• Next.js UI: a live run timeline, a diff of what the models changed, Daytona terminal output, and a benchmark dashboard.

Starting tier for a task of type k: the cheapest model t (Haiku → Sonnet → Opus) that is trusted for that type — i.e.

• pass rate p̂[k,t] ≥ 0.8, and
• tried at least twice n[k,t] ≥ 2

No model trusted yet → fall back by difficulty (≤3 Haiku, 4 Sonnet, 5 Opus). Any failure → upgrade one tier.

Results

13 tasks × 4 strategies, all tested in Daytona:

An honest finding: Haiku alone was even cheaper on these tasks. But you only learn that by measuring. Pacer measures automatically, while still catching the steps that need a stronger model.

Challenges

• Model access: Haiku 5.5 was blocked on our Bedrock account, so we used Haiku 4.5 and priced each model from its ID.
• Sandbox limits: Daytona's 10-CPU org limit and slow sandbox starts broke parallel benchmarks, so we added a concurrency cap, retries and cleanup.
• Module formats: the Arize instrumentation package ships only as an ES module, while our scripts run as CommonJS, so we load it asynchronously.
• Benchmark tasks were too easy: Haiku passed them all, which led us to build user-generated, multi-step tasks with tests for each step.

What we learned

Tests make the best router. A real test in a sandbox is a stronger signal than any guess about difficulty. And observability becomes far more valuable once the agent reads it back to make decisions, not just to fill dashboards.

Built With

• amazon-bedrock
• amazon-web-services
• anthropic
• arize
• arize-ax
• claude
• claude-code
• daytona
• mcp
• model-context-protocol
• next.js
• node.js
• openinference
• opentelemetry
• otlp
• pytest
• python
• react
• recharts
• server-sent-events
• tailwindcss
• typescript
• vercel-ai-sdk
• zod