JOULE The AI Compute Governor

JOULE learns how much AI compute each task needs, routing requests to the most efficient NVIDIA model on Nebius that can succeed—and escalating only when needed.

Comment

Inspiration

AI systems are becoming more capable — but we are increasingly using the most powerful model for tasks that often do not need it.

That creates a simple question:

Why spend frontier-level compute on every request if a smaller model could produce the same successful outcome?

Most AI routing systems are based on static rules, model availability, or manually configured thresholds. We wanted to explore something different:

Can a system learn how much intelligence a task actually needs before spending the compute?

That question became JOULE — The AI Compute Governor.

JOULE is built around a simple principle:

Use the least amount of AI compute required to achieve a successful outcome — and escalate only when necessary.

The goal is not to make AI cheaper by accepting worse answers.

The goal is to make AI more efficient without sacrificing outcome quality.

What it does

JOULE learns which class of model is likely to succeed on a task and routes the request accordingly.

Instead of sending every task to the most capable model, JOULE estimates:

[ P(\text{efficient model succeeds} \mid \text{task}) ]

and uses that probability to decide whether the request should go directly to an efficient model or be routed to a more capable model.

The basic flow is:

[ \text{Task} \rightarrow \text{JOULE Policy} \rightarrow \begin{cases} \text{Efficient Model} \ \text{Frontier Model} \end{cases} \rightarrow \text{Outcome} ]

JOULE then compares its routing policy against an Always Frontier baseline.

But cost reduction alone is not considered a success.

We created a benchmark called J25.

J25 passes only when:

  1. routed output quality remains within a configured equivalence band compared with the frontier baseline, and
  2. cost per successful outcome improves by at least 25%.

In other words:

cheap but bad does not count.

For the Nebius × NVIDIA hackathon, the architecture is being adapted around Nebius infrastructure and NVIDIA open models, allowing JOULE to govern compute across different levels of model capability instead of being tied to one provider.

How we built it

JOULE evolved through several prototypes.

JOULE 0.4 — Adaptive Router

The first routing version used an efficient-model-first strategy:

[ \text{Efficient Model} \rightarrow \text{Verify} \rightarrow \text{Escalate if needed} ]

This proved that selective escalation could be measured against an Always Frontier baseline.

It also taught us something important:

even when the efficient model is unnecessary, calling it first still consumes compute.

That led to JOULE 0.5.

JOULE 0.5 — Experience Map

Instead of always trying the efficient model first, JOULE 0.5 attempts to predict the correct route before the first production inference call.

The prototype learns from historical task outcomes using:

  • TF-IDF task representations
  • logistic regression
  • predict_proba
  • observed success labels
  • configurable routing thresholds

We deliberately started with a transparent, inexpensive policy model rather than another large language model.

The important part is the learning loop:

[ \text{history} \rightarrow \text{policy} \rightarrow \text{route} \rightarrow \text{outcome} \rightarrow \text{new history} ]

This creates what we call the Experience Map: a growing dataset describing which kinds of tasks require which level of intelligence.

Shadow exploration

A learning router creates a difficult problem.

If JOULE decides that a task is too difficult for the efficient model and sends it directly to the larger model, we never observe whether the efficient model could actually have succeeded.

That creates a selective-label problem.

To address it, JOULE supports shadow exploration.

A small percentage of tasks routed to the stronger model can also be evaluated with the efficient model in the background.

The user still receives the production answer from the selected route, while the shadow result becomes new training evidence for the Experience Map.

This allows JOULE to continue learning instead of becoming trapped by its own previous decisions.

Challenges we ran into

The hardest challenge was defining what “efficient” actually means.

Reducing token cost is easy if quality is allowed to collapse.

That is why we moved from measuring raw inference cost to measuring:

cost per successful outcome.

Another challenge was verification.

A router cannot know the correct answer in advance in a real production environment. Early benchmark gates can use known reference answers, but those mechanisms are useful for evaluation rather than deployment.

This forced us to separate:

  • online routing
  • online verification
  • offline evaluation
  • production economics

We also encountered the selective-label problem: once the router starts making decisions, its own decisions influence the data available for future learning. Shadow exploration became our first solution to that problem.

Finally, we had to be careful about energy claims.

JOULE is called JOULE because compute efficiency ultimately has an energy dimension, but API token usage is not the same thing as measured physical energy.

We therefore do not claim that token savings equal watt-hour savings.

True energy optimization will require GPU telemetry, job duration, GPU allocation, and datacenter overhead measurements.

Accomplishments that we're proud of

We are proud that JOULE has evolved from an idea about measuring AI efficiency into a working routing experiment with a measurable success criterion.

The project now includes:

  • adaptive model routing
  • an Always Frontier comparison baseline
  • quality-equivalence testing
  • cost-per-successful-outcome measurement
  • the J25 benchmark
  • a learned Experience Map
  • probabilistic pre-inference routing
  • shadow exploration
  • explicit accounting for learning overhead

We are particularly proud of one design decision:

JOULE refuses to call an optimization successful simply because it is cheaper.

The output still has to work.

That changes the optimization target from:

[ \text{minimize cost} ]

to something much closer to:

[ \text{maximize successful outcomes per constrained resource} ]

We think that is a more useful way to think about AI infrastructure.

What we learned

Our biggest lesson is that model routing is not really a model-selection problem.

It is a decision problem under uncertainty.

The optimal model depends not only on task difficulty, but potentially on:

  • expected quality
  • inference cost
  • latency
  • risk
  • output length
  • tool requirements
  • customer constraints
  • available compute
  • model pricing
  • eventually, measured energy

We also learned that routing data can become an asset of its own.

Every completed task can teach the system something about how much intelligence similar tasks require.

That means JOULE can improve not only because models improve, but because the governor itself gains experience.

The deeper insight is:

Not every task deserves the same amount of intelligence.

The system should learn the difference.

What's next for JOULE — The AI Compute Governor

The next step is JOULE 0.6.

Instead of routing with a fixed probability threshold, JOULE will evolve toward an economic and resource-aware policy.

The decision becomes something closer to:

[

a^*

\arg\max_a \; \mathbb{E} [ \text{utility} \mid \text{task}, a ] ]

subject to constraints such as:

[ \text{quality}, \text{cost}, \text{latency}, \text{risk}, \text{compute}, \text{energy} ]

For this hackathon, we are adapting JOULE to Nebius infrastructure and NVIDIA open models so that different levels of NVIDIA model capability can become actions inside the governor.

The longer-term vision is broader than routing between two models.

JOULE could eventually decide between:

  • different model sizes
  • different reasoning budgets
  • different inference providers
  • local vs. cloud inference
  • different GPU configurations
  • different latency targets
  • different energy constraints

Instead of asking:

“Which AI model should we use?”

JOULE asks:

“How much intelligence does this task actually need?”

That is the system we want to build.

Built With

Share this project:

Updates

Submission history