NemoRoute

Inspiration

Junior consultants spend hours turning a strategy question into a sourced memo. LLMs can draft that memo, but sending every question to the biggest model is expensive, and most teams never measure whether a smaller model would have done the job. We wanted an agent that picks the cheapest model that is good enough, and evidence for when that works and when it doesn't.

What it does

You ask a business-strategy question. NemoRoute plans web searches (Tavily), extracts numbered facts from the sources, and writes a cited memo: bottom line, key findings, analysis, risks and next steps. A fast NVIDIA Nemotron 3 Nano router reads the question first and sends the memo-writing step to Nano, Super or Ultra. The app shows which tier was chosen and why, the cost and latency of every step, and an all-Ultra cost estimate for comparison. An Evidence page shows all the benchmark results.

How we built it

• Models: Nemotron 3 Nano (router, planning, extraction, tier 1), Super (tier 2) and Ultra (tier 3), all served by Nebius Token Factory through its OpenAI-compatible API.
• Pipeline: route → plan → search → extract → synthesize, with schema-constrained JSON, retries, and every call logged with tokens, latency and cost.
• Evaluation: we built a 600-query consulting dataset with Ultra. Nano, Super and Ultra all answered every query, and a blind Ultra judge graded them. Thresholds were calibrated against human labels on a separate set before the test split was frozen.
• Benchmarks: a held-out routing test set (121 queries), MMLU-Pro business/economics (300 questions, exact match, no judge), and a 40-question memo benchmark where all three strategies see identical frozen sources.
• App: Streamlit on Community Cloud, with per-session, daily and lifetime spend caps.

Results

• Memos: routed 4.13 vs all-Ultra 4.41 (out of 5) at 28% of the all-Ultra cost ($0.0058 vs $0.0204 per memo).
• Held-out test set: the router's pass rate (79.8%) was on par with all-Ultra (77.7%) at 58% of the cost.
• MMLU-Pro: routing saved 85% but lost 20.7 points of accuracy. The router sent 289 of 300 questions to Nano, which is too aggressive for multiple-choice reasoning.
• Judge vs human: 70% agreement on 30 blind memos (Cohen's κ = 0.33). The judge was more lenient than the human reviewer.

Challenges we ran into

• Our first memo benchmark was unfair: each strategy planned its own searches, so they saw different sources. We rebuilt it so every strategy shares identical frozen evidence.
• Generated queries invented regulations and statistics and stated them as fact. We hardened the generation, answer and judge prompts against false premises.
• Answers and judgments were cut off at the token limits; Ultra's reasoning used up the judge's budget before it wrote its JSON.
• We had to choose a pass rule for multi-claim memos without moving the locked labeling thresholds, so we report both.

Accomplishments that we're proud of

• An honest evaluation: frozen test sets, calibration k unresolved queries counted rather than hidden,bootstrap confidence intervals, and a human-agreement check on the judge.
• Finding where the quality goes: on questions routed t traces to the Nano extraction step (accuracy 3.04 vs3.60, grounding 2.68 vs 3.24), not to the router's choice.
• A live, cost-capped demo that shows the cost of every

What we learned

• Routing is cheap to deploy but easy to over- or under-tune. The same zero-shot router was well balanced on consulting queries and far too Nano-heavy on MMLU-Pro.
• In a multi-step agent, the cheapest step can cap quality. Here extraction mattered more than synthesis. - LLM judges need checking against humans; ours was mea

What's next

• Run extraction on a larger tier and re-measure the quality gap. - Train the router on our labels. Token Factory can't fo the router is prompted rather than trained.
• A stronger, multi-judge evaluation and a larger memo benchmark.
  

Built With

• matplotlib
• nebius-token-factory
• nvidia-nemotron
• openai
• pandas
• pydantic
• python
• scikit-learn
• streamlit
• tavily