Inspiration
Not every AI request needs the most powerful—and most expensive—model. But choosing a cheaper model can feel risky when a wrong answer becomes bad data downstream.
We built RouteWise to make that decision measurable. Instead of guessing which model is good enough, developers can test configurations against their own accuracy and latency requirements and see the cost of each option.
What it does
RouteWise benchmarks NVIDIA Nemotron configurations on Nebius Token Factory and identifies the least expensive setup supported by the workload evidence.
Our current workflow focuses on extracting refund information from support conversations. Developers can set quality and latency requirements, compare model configurations, inspect recorded examples, run a fresh request, and download a deployment decision.
On our 312-conversation representative dataset, the adopted RouteWise profile achieved:
- 77.1% lower inference cost than grounded Nemotron Super: about $0.080 versus $0.349 per 1,000 requests.
- 98.4% correct completions without human review.
- Zero observed wrong answers released.
At one million requests per month, the recorded usage rates project to approximately $80 versus $349, or $269 in monthly inference savings. This projection excludes hosting, development, labeling, and human review.
The dataset is synthetic, and independent human validation of this representative dataset has been completed. These are workload-specific measurements, not a production guarantee.
How we built it
We used NVIDIA Nemotron 3 Nano, Super, and Ultra through Nebius Token Factory. Python and Pydantic handle model calls, structured outputs, validation, scoring, routing, and cost accounting. Streamlit and Altair power the dashboard.
The adopted profile starts with grounded Nano and escalates to Super when a deterministic check fails. It also normalizes supported refund issue dates from the store’s message timestamp. Ambiguous dates are held for human review.
Each trace preserves the model output, validation checks, final answer, latency, token usage, and the cost of every model call. Developers can export the selected configuration, requirements, evidence run, and source hashes.
Challenges we ran into
Measuring reliability was harder than measuring cost. A valid JSON response can still contain the wrong date or order ID. We needed field-level checks and held-out comparisons, not just a format validator.
Refund dates are easy to confuse. “The refund will be issued in four days” and “the money will arrive in four days” describe different events. We built a date analyzer that resolves supported issue dates and sends ambiguous expressions to review.
Latency varies across serving conditions. The adopted profile passed its four-second deadline gate in the representative confirmation, but the later writing-noise study missed that gate on a slower serving day. We report both results rather than treating a single favorable latency measurement as a guarantee.
Accomplishments that we're proud of
We built a complete workflow from workload requirements to a measured model choice, a live trace, and an exportable deployment decision.
We’re especially proud that RouteWise can recommend a cheaper route, select a fixed model when the requirements favor it, or make no recommendation when the benchmark cannot support one.
We also preserved the experiments that did not work. For example, a learned predictor looked promising on random splits but failed to transfer to unseen scenario families, so we did not adopt it.
What we learned
- Model agreement is not ground truth. Strong models can agree and still be wrong; the source and field contract matter.
- Review must be measured separately from accuracy. Sending more requests to a person should not make an automated system appear more effective.
- The right comparison matters. A matched, grounded fixed-model baseline makes the cost and quality tradeoff more meaningful.
- Reliability includes latency. A system needs evidence about both answer quality and response time.
- Simple rules can be useful. Our retrospective rules-only baseline completed 70.5% of the representative conversations correctly without review, compared with 98.4% for the recorded adopted profile. This is a comparison on previously exposed synthetic data, not a claim about every possible rules-based system.
What's next for RouteWise
Our next step is to evaluate the workflow on a separate set of independently labeled, de-identified customer conversations and measure latency across more serving days and realistic request volumes.
We also want to expand beyond the current six-field refund-extraction contract, while keeping the same principle: measure cost, quality, latency, and review burden together before recommending a configuration.
Log in or sign up for Devpost to join the conversation.