Waypoint

Large-model accuracy at small-model prices: a support router that lets NVIDIA Nemotron Nano handle most messages, calls in Super or Ultra only when needed, and records every decision.

Comment

Inspiration

Every support team trying AI hits the same wall. The big models give good answers, but you pay big-model prices on every "where's my order?". Small models are cheap, but in support a confident mistake means a wrong refund or an angry customer. We kept thinking about how a real support desk works: a junior agent handles most tickets and only pulls in a senior when they're stuck. We wanted an AI system that works the same way, and that can show its working.

What it does

Waypoint is an OpenAI-compatible router for customer support, running NVIDIA Nemotron on Nebius Token Factory. You change the base_url in your app, and every email or chat goes through a pipeline:

  • Recalls similar past cases: finds the 3 most similar solved conversations and shows them to Nemotron Nano as worked examples.
  • Understands the request: Nano classifies it; if its confidence is low, Nemotron Super takes over.
  • Pulls out details exactly: order numbers and emails are matched by plain code, not a model, then checked against the store's records. If a customer says "I'm gold" and the store says silver, the store wins.
  • Checks photos: Nemotron Super-VL confirms the photo matches the customer's story before anything is approved.
  • Applies your rules: refunds and returns follow the store's rulebook, written as plain code, and the reply can only promise actions that actually happened.
  • Writes and checks the reply: the smallest model that writes well enough, followed by a safety check, with sensitive topics handed to a person.

Every reply keeps a decision record: what was understood, where each detail came from, which rule applied and which model did the work. The product includes sign-up with your own Token Factory key, onboarding that tunes each connection on your examples, a dashboard, live chat with a step-by-step Decision Flow, a Shopify-style demo store, docs, and a Docker download for running it in your own network.

Results on 600 real support messages from 4 public datasets (scored on a held-out half):

Setup Accuracy Cost per 1,000 messages
Nemotron Super alone 68% $0.177
Nemotron Ultra alone 64% $0.739
Waypoint (Nano with memory, Super when unsure) 81% $0.042

That's 77% cheaper than Super and 94% cheaper than Ultra, with 95% of messages handled by Nano. To be fair: Super given the same memory scores 82%, so Waypoint is one point behind it at about a sixth of the cost.

How we built it

  • Backend: Python, FastAPI and httpx, with SQLite for projects, requests and decision records, and Fernet encryption for each user's Token Factory key.
  • Models, all through Token Factory's OpenAI-compatible API:
    • Nemotron 3 Nano is the first responder, with thinking switched off so its token log-probabilities measure confidence in the answer itself.
    • Nemotron 3 Super is the backup and the default writer for complex replies.
    • Nemotron 3 Ultra handles high-stakes replies and grades answers during tuning.
    • Nemotron 3.5 Super-VL checks photos.
    • Qwen3-Embedding-8B powers the memory of past cases.
  • Confidence cascade: the threshold is tuned per connection on the customer's own examples, holding Nano to the quality bar of the model it would hand off to.
  • Forced choices: structured outputs restrict classification to the valid labels, so there are no invented categories.
  • Gateway features: pacing based on Token Factory's rate-limit headers (live traffic goes before background jobs), fallback to a similar model on overload, cost tracking per customer with budgets, prompt-cache hints, and LangSmith tracing of every model call.
  • Frontend: static HTML with Tailwind and Chart.js; no build step.

Challenges we ran into

  • Confidence was meaningless at first. Nemotron 3 thinks by default, so the log-probabilities scored the reasoning, not the answer. Turning thinking off for the scoring call fixed it.
  • Bigger wasn't better. Ultra scored below Super on short classification. And without memory, our cascade traded 3 points of accuracy for a 61% saving, which isn't worth it in support. Adding memory of past cases turned that trade-off into a win on both accuracy and cost.
  • Noisy benchmarks. We found wrong answer keys in public datasets and reviewed them by hand before trusting any score.
  • Dropping fine-tuning. We first planned to distil Ultra into a smaller model, but that would need hourly dedicated GPUs, which can't compete with Nano's per-token price. Retrieval gave us the gains without any training.
  • Trust. Customers state things that aren't true. We had to make the store record the source of truth for account facts and stop replies from promising anything that hadn't happened.
  • Messy conversations. On 348 deliberately difficult conversations, we raised the pass rate from 49% to 69%. Two requests in one message is still an open problem.

Accomplishments that we're proud of

  • Beating Nemotron Ultra's accuracy at 6% of its cost, measured on held-out data.
  • A system that shows its working: every decision is visible, from the past cases it recalled to the rule that fired.
  • A complete product, not a notebook: sign-up, onboarding with automatic tuning, a dashboard, live chat, docs and a self-hosted Docker version.

What we learned

  • Before switching to a bigger model, give the smaller one better context. Three good examples did more than 18 times the parameters.
  • Measure, don't assume: on our tasks, the largest model was neither the most accurate nor worth its price.
  • In support, exact things belong in code. Models should read and write; rules and records should decide.

What's next for Waypoint

  • Link "Teach your router" uploads to a specific connection, so a company running two businesses keeps them separate, and feed each upload into memory, instructions and tests.
  • Handle two requests in one message reliably.
  • Connectors for real helpdesks and stores (Zendesk, Shopify), beyond our demo store.
  • Grow each company's memory nightly from conversations it marks as good (opt-in), so accuracy keeps improving.

Built With

  • chart.js
  • cryptography-(fernet)
  • docker
  • docker-compose
  • fastapi
  • html
  • httpx
  • javascript
  • langsmith
  • nebius-token-factory
  • numpy
  • nvidia-nemotron-3-nano
  • nvidia-nemotron-3-super
  • nvidia-nemotron-3-ultra
  • nvidia-nemotron-3.5-super-vl
  • openai-api-(compatible)
  • python
  • qwen3-embedding
  • sqlite
  • tailwind-css
  • tavily
  • uvicorn
Share this project:

Updates

Submission history