ModelRouter

Inspiration

We kept running into the same problem: cloud LLMs are powerful but expensive, and most prompts don’t need a frontier model. Simple questions, basic code, or light creative tasks were burning tokens (and money) on overpowered models. We wanted a smarter middle layer that could look at every prompt, understand how complex it really is, and send it to the cheapest model that can still do a great job.

What it does

ModelRouter is an intelligent AI middleware gateway that sits between your application and remote LLM providers. It analyzes every incoming prompt with an NLP classifier, scores its complexity and task type (Open QA, Code Generation, Reasoning, Creative Writing, etc.), then automatically routes it to the most cost-effective and capable model.

• Tier 1 (Cheap & Fast): Gemini 1.5 Flash, GPT-4o-mini, Claude 3 Haiku
• Tier 2 (Balanced): GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro
• Tier 3 (Frontier Reasoning): o3-mini, o1, DeepSeek-R1

It also handles API-key management, usage tracking, and offers both a custom routing endpoint and a fully OpenAI-compatible /v1/chat/completions interface.

How we built it

We built a modular FastAPI gateway with three core services:

Model Service – Uses the NVIDIA prompt-task-and-complexity-classifier (DeBERTa-v3) to predict task category and multi-factor scores (creativity, reasoning, domain knowledge, constraints). A fast heuristic fallback keeps it running even when the model isn’t available. Auth Service – Generates, validates, tracks, and revokes API keys with persistent SQLite storage. Routing Service – Maps classifier metrics to the right model tier and returns full routing rationale plus simulated responses.

Everything is mounted under a single unified gateway with interactive Swagger docs.

Challenges we ran into

There was no public dataset tailored for prompt complexity and task classification at the granularity we needed, so we had to work with limited training signals and heavy reliance on the NVIDIA model. We were unfamiliar with NVIDIA NeMo Curator, which slowed down data preparation and experimentation when we tried to improve the classifier.

Accomplishments that we're proud of

We shipped a fully working end-to-end system: authentication, real-time classification, intelligent routing across three capability tiers, OpenAI-compatible endpoints, and a live demo that correctly routes simple prompts to cheap models and complex ones to frontier models—all in a clean, production-ready architecture.

What we learned

Prompt complexity is multi-dimensional; a single “hard/easy” score isn’t enough—you need separate signals for reasoning, creativity, and constraints. A solid fallback heuristic is just as important as the fancy classifier for real-world reliability. Building a unified gateway with clear service boundaries makes the whole system much easier to extend and debug.

What's next for ModelRouter

Ship it as a real subscription-based service, just like any other LLM provider. Developers will get an API key, drop in our OpenAI-compatible endpoint, and automatically start saving money while still getting the right model for every request. Future plans include live cost analytics, custom routing policies, and support for more providers.

Built With

• css3
• fastapi
• html5
• huggingface
• javascript
• python