Agent Cost Down

Inspiration

Before paying for a bigger AI model, find out what actually failed.

Developers building agents face a practical question: which tasks can a cheaper model handle, and when is a larger model worth the cost? A price table cannot answer that. Neither can a dashboard that treats every completed request as a success.

I built Agent Cost Down to turn that question into a working process: compare execution policies, inspect the failures, and export the exact configuration you tested.

What it does

Agent Cost Down compares five ways to run an agent using NVIDIA Nemotron on Nebius Token Factory:

• Nano alone
• Super alone
• Ultra alone
• Nano with one retry
• Nano followed by Ultra when application checks reject the first attempt

The interface connects three steps:

Compare: See estimated cost, correctness, speed and failure causes across all five policies.

Inspect: Follow real recorded attempts, tool calls and state changes. Find out whether a failure came from the model, the tools or the scoring rule.

Apply: Download the measured policy and run it through the same execution engine, locally or through a bounded live sample.

The exported file controls execution. It is more than a recommendation on a chart.

A concrete example

In one recorded run, a customer asks for seven additional shirts. Nano checks inventory, finds twenty-five available, and incorrectly refuses the request.

The application rejects that refusal. Under the configured Nano → Ultra policy, Ultra starts again from the original state and completes the change: the order quantity increases from two to nine, and available stock decreases from twenty-five to eighteen.

A separate evaluator then confirms the outcome.

That separation matters. Runtime checks decide whether to retry; independent evaluation decides whether the completed result was correct. Expected answers never guide the agent or trigger escalation. Another recorded example shows why this is necessary: an analysis answer passes the runtime checks but is independently scored wrong.

What I measured

The completed v4 study contains 100 development executions and 300 evaluation executions. Evaluation covers twenty synthetic store tasks, repeated three times across five policies.

Result across 60 executions per policy	Ultra	Nano → Ultra
Correct under declared scoring	57/60	57/60
Total estimated inference cost	$0.4727	$0.1207
Median execution time	5.45 seconds	18.91 seconds

The cascade used 74.5% less estimated inference cost on this workload. Seven runs escalated to Ultra, and all seven recovered to correct outcomes. The trade-off was slower execution.

The quality comparison depends on scoring. Ultra’s three failures asked for extra clarification fields that my frozen rule rejects. Under a broader clarification rule, Ultra scores 60/60, while the cascade remains at 57/60. Both interpretations are visible in the explorer. Equal observed counts under one rule do not establish equivalent model quality.

What I learned and the challenges I faced

My first study exposed a defect in my own tools. Offline replay traced 33 of 52 final failures to a runtime rule rejecting fields that merely repeated their current values. Using a bigger model did not fix an inconsistent tool contract.

I repaired redundant-field handling and error feedback together, committed the amendment before rerunning, and preserved the original evidence.

Controlled rerun on the same evaluation suite. These tasks and their v3 results were inspected before the v4 amendment. This is not fresh independent validation.

The second discovery concerned measurement itself: a seemingly simple clarification rule changed the comparison between policies. I kept the frozen scores and added the alternative interpretation rather than hiding the sensitivity.

The lesson is the product’s value: before changing models, inspect the specific runs and understand why they failed.

How I built it

NVIDIA Nemotron 3 Nano, Super and Ultra run through Nebius Token Factory, using pinned model identifiers, settings and prices.

I built the backend with Python, FastAPI, Pydantic and HTTPX. Each attempt receives an isolated SQLite snapshot. Runtime validation and independent evaluation are separate components, and retries restart from the original state.

The React and TypeScript explorer runs on Vercel. Render hosts the API, and Neon stores durable jobs and spending records. Live samples use predefined tasks, rate limits, separate judge and public budgets, and conservative cost reservations.

What comes next

This is a small synthetic workload with seven distinct evaluation scenarios, not a universal routing benchmark. Costs are estimates from reported usage and pinned prices.

My next steps are fresh independently authored tasks, additional agent adapters and separately evaluated routing by task family.

Agent Cost Down shows what you save, what goes wrong, and what you can actually run.

Built With

• fastapi
• httpx
• nebius-token-factory
• neon
• nvidia-nemotron
• postgresql
• pydantic
• python
• react
• render
• sqlalchemy
• sqlite
• typescript
• vercel
• vite