EcoRoute

Inspiration

Every AI chat costs about 10x the electricity of a web search, and most of that cost is spent asking a frontier-scale model to answer "can I return this?" We realized our laziness costs thousands in energy lost to large data centers. AI is cool, but the ecological impact cannot be ignored. We thought: what if the internet's AI traffic had a bouncer? Something that looks at a request, decides "this is easy, send it to the intern" or "this one's dangerous, get the expert," and does it before you burn a gigawatt on a return-policy question. EcoRoute is that bouncer.

What it does

EcoRoute is an OpenAI Chat Completions compatible AI gateway with a separate operator control center and a demo e-commerce support app (Northstar Outfitters). A customer points their existing OpenAI SDK client at EcoRoute (base_url="http://localhost:8000/v1"), sends normal chat requests, and gets normal responses back, while EcoRoute makes energy- and cost-aware decisions underneath.

For every request it:

Reuses responses safely through a two-tier cache (exact plus semantic) so identical and near-identical prompts never hit a model at all.
Classifies the request (complexity, risk, task type, SLM-eligibility) using a deterministic classifier with an optional FreeSOLO-trained router on top.
Routes carbon- and complexity-aware: clean grid plus easy prompt gets downgraded to a cheaper general model; dirty grid plus easy prompt is handed to a specialized self-hosted SLM; high-risk or high-complexity prompts stay on the frontier model, always.
Validates small-model output against a schema/quality gate and automatically falls back to the frontier model if it fails.
Accounts for the impact of every decision (baseline vs. actual energy, carbon, and cost) and shows it in the control center with honest evidence labels (measured / estimated / stale / simulated).

There's also a self-hosted node agent that applies reversible OS- and GPU-level optimization profiles to inference hosts, and an SLM Studio wizard that takes you from "describe your business policies" all the way to a deployed specialized model.

How we built it

The stack is a pnpm plus Python monorepo, fully runnable from one docker compose up:

Gateway: FastAPI plus Pydantic v2, SQLAlchemy 2 async on PostgreSQL, pgvector for embeddings, Redis for the hot cache and job streams, and LiteLLM for upstream provider normalization (never for policy decisions, those are ours). The request pipeline lives in services/gateway/ecoroute and flows: normalize_request, cache lookup, classify, select_candidate, provider dispatch, quality gate, impact accounting.
Caching layer: exact matches are a SHA-256 fingerprint over normalized messages/params stored in Redis; semantic reuse embeds the prompt with a local Sentence-Transformers model (384-dim) and does an HNSW cosine search in pgvector. We only reuse on a high similarity threshold (0.94) and a minimum margin over the runner-up (0.02) to avoid confidently returning the wrong cached answer. Cache TTLs are even scaled by grid state (shorter when clean, longer when dirty).
Routing engine: a weighted multi-objective scorer (carbon, cost, latency, quality penalty, evidence penalty) with hard rules layered on top for the grid/complexity/risk cases. Grid state is bucketed into clean / moderate / dirty via configurable gCO2/kWh thresholds.
FreeSOLO training: two locked recipes, a Qwen 2B router (SFT then GRPO) as the classifier and a Qwen 4B support SLM (SFT then OPD, distilling from a teacher model) for in-domain policy Q&A. A worker service drives the whole lifecycle (env push, dry-run, cost quote, launch, poll, evaluate, deploy) through the FreeSOLO flash CLI, gated behind explicit operator confirmation.
Node agent: a Linux agent that reads real telemetry (NVML for GPU, Intel RAPL for CPU package energy, psutil for the rest) and applies reversible profiles: NVML GPU power caps, cgroup v2 CPU weighting, nice/ionice, an experimental sched_ext kernel scheduler, NAPI network tuning, and gateway concurrency limits, all transactional with verify-and-rollback. A deterministic simulator stands in when no real hardware is present.
Frontend: two Next.js App Router apps, the operator control center (dashboards, routing policies, endpoints, nodes, request audit, SLM Studio, impact reports) and the customer-facing support demo. Recharts for live series, TanStack Query for server state, Playwright for E2E.
Reporting: Prometheus metrics, CSV export, and an Impact Framework manifest export, all under a documented ecoroute-v1 methodology that preserves evidence provenance.

The features we're most excited about

FreeSOLO for the SLM and the classifying layer. This is the heart of EcoRoute. We use FreeSOLO to train both brains of the system: the router/classifier (Qwen 2B) that decides how hard a request is, and the support SLM (Qwen 4B) that actually answers the easy in-domain questions. The classifier emits a structured verdict (complexity, risk, task_type, slm_eligible, cache_eligible, predicted_output_tokens, confidence), and the SLM answers with a strict JSON contract (answer, confidence, policy_ids, needs_human) so its output can be machine-verified before we trust it. Crucially, the classifier is deterministic-first: hard safety rules (tools, PII, secrets, legal, safety-critical) are authoritative and can never be overruled by the learned model, and everything fails closed to the frontier model when confidence is low or a call errors.

The caching layer and prompt reuse. Before we ever think about which model to use, we ask "have we already answered this?" Exact hits (Redis) short-circuit instantly; semantic hits (pgvector) let us reuse a stored answer for a differently-worded but equivalent policy question, with the conservative similarity plus margin guardrails above so we never reuse across ambiguous matches or high-risk requests. A cache hit means zero model inference, which is the cheapest possible energy: none.

Carbon- and complexity-aware model selection. This is the routing story, and we confirmed each branch in the code:

Easy prompt plus clean grid means downgrade. On a non-dirty grid with low/medium complexity, we exclude the specialized SLM (it's reserved for dirty grids) and let the weighted scorer pick the leanest eligible general model, since carbon and cost weights push it toward a smaller/cheaper endpoint.
Dirty grid means use the SLM. When the grid is dirty and the prompt is low/medium complexity, there's an explicit dirty_grid_specialized_preference rule that routes to the self-hosted specialized SLM.
High risk or very complex means stay frontier. Any request classified high-risk or high-complexity forces frontier_required, no downgrade, ever.
All of it is measured. Each decision writes an impact record (baseline vs. actual energy/carbon/cost) that surfaces in the control center's overview, per-request audit, and cache stats, so the user can actually see the savings each choice bought.

Kernel-level optimization on self-hosted nodes. For self-hosted inference we went below the application layer to make each call cheaper in wall-clock and watts. The node agent can install a reversible eco profile that combines an experimental sched_ext BPF scheduler, NAPI network-stack tuning, cgroup v2 CPU weighting that starves background work in favor of inference, and NVML GPU power caps (down to 80%). Less wasted scheduling, less idle power draw, and tighter power envelopes mean the same inference finishes using less electricity, and every profile is applied transactionally with guardrails (latency/throughput/error checks, GPU temperature limits, heartbeat loss) that auto-roll-back if the optimization ever hurts.

SLM Studio plus honest reporting. SLM Studio is a full wizard (Define, Policies, Generate, Review, Train, Evaluate, Deploy) that turns "here are my business policies" into a deployed specialized model, with Gemini-backed synthetic dataset generation, human review, and evaluation gates (for example schema validity at or above 0.99, zero prohibited promises) that block deploy unless they pass. And the whole platform refuses to lie about its numbers: every impact value carries an evidence label, simulated fixtures are never dressed up as measured, and exports (CSV, Impact Framework YAML, Prometheus) preserve that provenance.

Challenges we ran into

Making routing feel smart without hard-coding brittle if-ladders. The tension between "just apply a rule" (dirty grid means SLM, high risk means frontier) and "optimize continuously" (weighted carbon/cost/latency scoring) took real iteration. We landed on hard rules for the cases where correctness/safety is non-negotiable and a scorer for the fuzzy middle.
Semantic caching that doesn't quietly return the wrong answer. A naive nearest-neighbor cache is a great way to confidently hand a customer someone else's answer. Getting the similarity threshold and the runner-up margin and the per-task eligibility gates right (only policy_qa, never PII/tools/personalized/high-risk) was the difference between "clever" and "dangerous."
Trusting a small model's output. Small models are cheap until they hallucinate a refund promise. Building the JSON output contract plus verify_output quality gate plus automatic single frontier fallback was essential to make SLM routing safe rather than reckless.
Kernel-level controls that can't brick the host. GPU power caps, cgroups, and a sched_ext scheduler are powerful and dangerous. Every control had to be reversible, transactional, and guarded by heartbeat/latency/temperature rollbacks, and default to observe only until an operator explicitly approves each capability.
Being honest about evidence. It's tempting to show one big "carbon saved" number. We forced ourselves to distinguish measured / estimated / stale / simulated everywhere, which is less flashy but the only version we'd actually defend.
Credential-free by default. We wanted the whole thing runnable with zero API keys, so we built deterministic fake endpoints, fixture grid readings, and a node simulator, while keeping the real Gemini/FreeSOLO paths fully wired and inactive until an operator opts in.

Accomplishments that we're proud of

A drop-in gateway that's genuinely OpenAI-compatible, adopt it by changing two lines.
A complete FreeSOLO pipeline for two different models (router plus support SLM) with dry-run cost quotes, evaluation gates, and gated deploys, so there are no accidental training runs and no surprise spend.
A conservative two-tier cache that reuses prompts safely, plus routing that provably downgrades on clean grids, reaches for the SLM on dirty grids, and refuses to downgrade risky work.
Real kernel/OS/GPU optimization on self-hosted nodes with transactional apply-verify-rollback, not a mock.
An impact-accounting layer with intellectual honesty: evidence labels, methodology ecoroute-v1, and Impact Framework exports.
The entire stack runs from one command with no credentials, and still demonstrates every core capability end to end.

What we learned

The cheapest inference is the one you don't run. Caching plus classification did more for cost/carbon than any single model swap; the biggest wins came from not calling a model at all.
Sustainability claims live or die on evidence. The moment you blur measured and simulated numbers, the whole story loses credibility. Provenance had to be a first-class citizen, not a footnote.
Safety has to be the floor, not a feature. Deterministic safety rules that a learned model cannot override, fail-closed defaults, and output verification made us comfortable letting cheaper models handle real traffic.
"Kernel-level efficiency" is real and reversible. You can meaningfully cut inference energy with power caps, scheduler and network tuning, and cgroup priorities, as long as everything can roll back the instant it degrades service.
Great DX is a sustainability strategy. If adoption requires a rewrite, it won't happen. Matching the OpenAI API exactly was what made every downstream saving possible.

What's next for EcoRoute

Wire live node telemetry into per-request attribution. Today per-request impact uses endpoint energy coefficients; we want measured NVML/RAPL energy attributed straight to individual requests.
Ship trained router plus SLM deployments so the learned FreeSOLO classifier and specialized model run out of the box, not just deterministic routing with fake endpoints.
Expand the optimization catalog: quantization, model load/unload and sleep/wake, and inference-server batching to complement the existing power/scheduler controls.
Broaden routing intelligence: geographic/region-aware routing where providers expose regional endpoints, and richer multi-objective policies.
Surface savings directly in the response path (beyond debug headers) so developers see per-request carbon/cost impact inline, and grow the cross-request/cross-tenant reporting.
Go beyond text: extend the same route-classify-cache-optimize loop to more modalities once the core loop is battle-tested.