Screenshot of Reflex showing various types of answers

Inspiration

Every AI agent starts with the same decision: which tool does this message need? Many agents send every message to an LLM to decide. That is a paid call and about half a second, every time, even for the easy ones.

Small classifiers are fast, but they fail in two ways we kept running into. They can change their answer when the same tools are listed in a different order. And they must pick a tool even when none fits, so a request to set an alarm gets sent to the timer.

What it does

Reflex picks the right tool from any list of tools written as name: description, or it answers NONE_OF_THESE.

Fast and local: about 50–120 ms per request on a laptop GPU, including fresh embeddings of the question and every tool.
Order-free: shuffling the tool list never changes the answer.
Open tool lists: tools it never saw in training work without retraining.
Asks for help only when needed: when Reflex answers NONE_OF_THESE, or its best tool is below a confidence threshold, the request goes to NVIDIA Nemotron 3.5 Lightning on Nebius Token Factory. Nemotron either picks one of the tools or proposes the missing one, for example alarm: Set an alarm for a specific time.

The live demo has four domains with fixed tools: customer service, banking, smart assistant, and travel & dining. Each domain is tagged as trained or unseen; customer service is entirely outside the training data. Every message is routed on its own, and the page shows each step as it happens: the question embedding, the tool embeddings and their similarity to the question, Reflex's probabilities, the routing decision, and the Nemotron call with its token counts.

How we built it

Embeddings: NVIDIA Nemotron Embed 1B, frozen, running locally on the GPU. Questions use encode_query; tools use encode_document on "name: description". Nothing is cached: every request embeds the question and all tools again.
Reflex head (2.3 M parameters): the same small network scores every tool against the question and the average of the whole list, so the order of the list cannot matter. The NONE_OF_THESE score comes from a 129-parameter head that reads six numbers describing the scores (the best score, the gap to the second, how spread out they are, and so on) rather than the question itself. That is what lets it reject lists of tools it has never seen.
Training data: CLINC150 and BANKING77, 227 intents. 45 labels are held out of training entirely so we can measure unseen tools. Training runs in three steps; in one of them, some training labels play the part of unseen tools.
LLM fallback: a runtime call to the Token Factory chat completions API with nvidia/Nemotron-3_5-Lightning, with thinking turned off. A call takes about 0.5–0.7 s and costs roughly $0.00003.
App: a FastAPI server streams each step to a single-page UI. All model work runs on one worker thread, so the server stays responsive while the model loads or embeds.

Results

Validation split, mean of five training runs with different seeds. The test split has not been used.

	Reflex	Cosine similarity	Order-sensitive head
Accuracy (seen labels and out-of-scope messages)	90.2%	76.4%	81.9%
Answer changes when the list is shuffled	never	never	26.8% of sets
`NONE_OF_THESE` F1	92.7%
Unseen label: correct tool ranked first	93.6%
Unseen label, only near-miss tools offered: says none fits	74.0%

In the demo, all nine example questions that escalate came back right from Nemotron: three resolved to an existing tool, and six were answered with a sensible new tool.

Challenges we ran into

Saying "none of these" for tools the model never saw. Our earlier heads read the question vector directly and learned what the training labels look like. Deciding from evidence in the scores instead raised correct rejection of near-miss lists for unseen labels from 65.2% to 74.0%, with 129 parameters for that decision instead of 656,129.
Reasoning on by default. On Token Factory, Nemotron 3.5 Lightning reasons before answering. Replies took 3–7 s and ran out of tokens before the JSON. Setting chat_template_kwargs: {"enable_thinking": false} brought calls to about 0.6 s with clean JSON.
Where to run embeddings. We tried moving embeddings to Token Factory to simplify hosting, but its embedding endpoint took 2.4–6.7 s per call. The embedder stays local, where it takes tens of milliseconds.

Accomplishments that we're proud of

Order invariance by construction: zero answer changes under shuffling, compared with 26.8% of sets for an order-sensitive head on the same embeddings.
It works in a domain it never saw. On customer-service tools written for the demo, an unrecognized $49.99 charge goes to dispute_charge with p = 0.99.
The LLM is only called when it is needed, so a confident request never leaves the machine.

What we learned

A small model that knows when it doesn't know is what makes an LLM fallback cheap.
Tool descriptions matter. Switching from bare names to name: description raised accuracy on unseen labels from 77.1% to 80.8%.

What's next

Choose the routing threshold from the measured trade-off curve, for example to meet a false-accept budget, instead of a fixed 0.70.
Close the main gap: when the right tool is new, Reflex picks it about 81% of the time, even though it ranks it first about 94% of the time.
Evaluate once on the held-back test split.
Package Reflex as a drop-in router for agent frameworks.