EcoCompute AI

Stop using your biggest AI model for every task. EcoCompute AI routes requests across NVIDIA Nemotron models to reduce cost and token use, with quality checks that keep the trade-offs visible.

Comment

Inspiration

Why send every AI task to your biggest model?

For teams running recurring jobs such as ticket classification, structured extraction, and data enrichment, that default can become expensive. But switching to a smaller model raises another question: how do you know the savings did not come at the expense of correctness?

I built EcoCompute AI to make that decision measurable. My interest in sustainability shaped the goal: identify unnecessary computation, test a bounded change, and keep the evidence visible.

What it does

EcoCompute AI is an evidence-driven optimization agent for recurring AI workloads.

It measures a baseline, asks NVIDIA Nemotron to diagnose where resources are being wasted, and turns that diagnosis into constrained optimization proposals. A human chooses which proposals may be tested. EcoCompute then runs the candidate configuration on the same tasks and checks whether it preserves every previously passing result while reducing cost or token use.

The workflow is:

  1. Observe: Measure the workload’s quality, tokens, cost, and latency.
  2. Reason: Nemotron analyzes the measured profile and proposes changes.
  3. Review: A human selects which proposals to test.
  4. Execute: Run the candidate configuration on the same tasks.
  5. Verify: Independent validators check correctness and regressions.
  6. Decide: A human accepts, rejects, or exports the recommendation and evidence.

Nemotron proposes. Deterministic checks verify. A human decides.

How I built it

The frontend uses Next.js, React, TypeScript, and Tailwind CSS, with light and dark themes and a guided optimization workflow. A Python FastAPI backend handles workload execution, routing, validation, measurement, and evidence storage.

Nebius Token Factory provides access to NVIDIA Nemotron Nano, Super, and Ultra. Nemotron Super serves as the default analyst, interpreting workload measurements and returning structured proposals.

The analyst can propose only three kinds of changes:

  • Change the model tier assigned to a task category.
  • Adjust output-token limits.
  • Enable escalation when validation fails.

An allowlist rejects unsupported actions before execution. The analyst cannot run shell commands or edit the live router.

Quality checks use JSON Schema, exact fields, labels, numeric tolerances, formatting constraints, and sandboxed Python tests. Failed outputs can trigger bounded repair or escalation to a larger model.

Each run preserves a trace with model choices, validator verdicts, provider request IDs, token usage, latency, and calculated inference cost.

Results backed by committed evidence

A real optimization run

On an eight-task support-operations workload, Nemotron’s proposed configuration:

  • Preserved 7/8 passing tasks, with no per-task regression.
  • Reduced inference cost from $0.005453 to $0.001980, a 63.7% reduction.
  • Reduced tokens from 2,389 to 2,112, an 11.6% reduction.
  • Increased summed model latency by approximately 33%.

The one-time analysis cost was approximately $0.003, giving an estimated break-even point of about two subsequent workload runs.

A separate held-out routing benchmark

EcoBench v1 contains 49 original synthetic tasks across seven categories: 14 calibration tasks and 35 held-out test tasks.

In the primary held-out run, calibrated adaptive routing achieved:

  • 80.8% lower inference cost than always using Nemotron Ultra.
  • 34/35 passing tasks, compared with 33/35 for Ultra.
  • Zero observed losses on tasks Ultra passed.
  • A paired bootstrap 95% interval corresponding to approximately 66%–93% lower cost.

The cost reduction is calculated as:

$$ \text{Cost reduction} = \left(1-\frac{C_{\mathrm{adaptive}}}{C_{\mathrm{baseline}}}\right) \times 100\% $$

The tradeoff matters: median latency increased from approximately 1.1 seconds to 1.6 seconds, and tokens per task increased by approximately 8%. In this benchmark, savings came from cheaper model tiers, not fewer tokens.

A separate cold-start replicate also showed lower cost with zero observed losses on baseline-passing tasks. These are small benchmark results, not a guarantee for every workload or evidence of statistically superior quality.

Explore the benchmark results and limitations.

Challenges I ran into

Making optimization verifiable. An aggregate pass rate can hide individual regressions. The acceptance gate therefore checks each task: a candidate cannot sacrifice a previously passing task to improve the overall cost.

Keeping calibration separate from evaluation. Routing evidence comes from the calibration split. Held-out tasks do not inform their own routing decisions.

Handling strict validators honestly. Some semantically reasonable answers fail exact checks, such as “Dr. Priya Raman” instead of “Priya Raman.” I retained and documented those failures rather than loosening the checks after seeing the results.

Separating sustainability estimates from measurements. Serverless inference does not expose actual GPU power or utilization. Energy, carbon, and water figures are explicitly labeled as scenario estimates with low, central, and high assumptions.

Accomplishments I am proud of

  • Giving Nemotron a substantive role in diagnosing and improving a measured workload.
  • Building a complete proposal, approval, execution, verification, and export workflow.
  • Recording real Nebius inference results with inspectable evidence.
  • Demonstrating safety checks that reject an unauthorized command proposal and a deliberately inadequate token cap.
  • Publishing an offline verifier that checks split separation, token-based cost calculations, and provider request IDs.
  • Designing a clear interface that shows both savings and tradeoffs.

What I learned

Smaller does not automatically mean faster, and cheaper does not automatically mean fewer tokens.

Reliable optimization requires measuring the workload, checking individual outcomes, and accounting for failed attempts and escalation. I also learned that an agent’s usefulness depends on the boundaries and verification around its recommendations.

What is next

I want to expand evaluation to larger, representative workloads, repeat measurements across more runs, and investigate routing that balances cost with latency requirements.

For environmental validation, the next step is testing with infrastructure that exposes hardware power telemetry, so modeled estimates can be compared against direct measurements.

Try EcoCompute AI · View the source code

Built With

Share this project:

Updates

Submission history