CascadeRouter — Autonomous FinOps & Latency Compiler
Inspiration
Modern AI applications increasingly rely on multi-agent architectures, where multiple LLM calls work together to complete complex tasks. However, developers often use expensive, heavyweight models even for simple operations such as intent classification, data extraction, and JSON formatting.
We saw an opportunity to address this inefficiency by bringing compiler-style intelligence to AI application development.
Inspired by the NVIDIA Nemotron 3 model family and Nebius Token Factory, we envisioned CascadeRouter as an autonomous optimization layer that helps developers understand their AI workflows, evaluate model choices, and make evidence-based routing decisions without manually refactoring their entire codebase.
What it does
CascadeRouter is an autonomous FinOps and latency optimization workbench for Python-based AI applications.
Its core capabilities include:
- Static AST Analysis: Scans Python repositories to discover LLM calls without importing or executing untrusted source code.
- Agent Dependency Mapping: Identifies relationships between LLM calls and constructs a visual Agent DAG.
- Benchmarking Engine: Supports controlled local and recorded-data benchmarking, with infrastructure designed around model performance evaluation.
- Evidence-Based Optimization: Evaluates available benchmark evidence to identify cost-efficient model configurations.
- Routing Policy Generation: Produces structured model-routing policies based on optimization decisions.
- Verified Code Proposals: Generates source-level patch proposals with exact change ranges, backups, and static validation safeguards.
- Interactive Developer Workbench: Provides a unified interface for repository analysis, agent visualization, benchmarking, optimization, and patch review.
The intended model-cascade architecture uses NVIDIA Nemotron 3 Nano for lightweight tasks, Super for intermediate reasoning, and Ultra for complex reasoning.
How we built it
We built CascadeRouter using a modular architecture combining a Python backend and a modern React-based frontend.
Backend:
- Python and FastAPI for API orchestration.
- Python AST for static repository analysis.
- Dedicated compiler modules for dependency graph construction, evaluation, metrics, optimization, and policy generation.
- SQLite-backed storage for analysis snapshots, benchmark jobs, and immutable evidence records.
- A separate runtime routing module with explicitly registered local executors.
Frontend:
- React 19, TypeScript, and Vite.
- Interactive SVG-based Agent DAG visualization.
- Repository scanning and benchmark workbench.
- Evidence-based optimization interface.
- Code patch review and validation interface.
Infrastructure and ecosystem alignment:
- Nebius Token Factory and NVIDIA Nemotron models form the target model-serving and benchmarking ecosystem.
- The architecture is designed to support cost, latency, and quality-aware model selection.
We also developed a structured testing and validation system covering compiler modules, API contracts, repository access, benchmark evidence, optimization, and patch safety.
Challenges we ran into
One of our biggest challenges was analyzing complex AI workflows without executing potentially untrusted repository code. We addressed this through static AST analysis and controlled source inspection.
Another challenge was reconstructing meaningful dependencies between LLM calls. Identifying individual calls was not enough; we needed to understand how outputs from one operation could influence subsequent operations.
We also faced the challenge of making optimization decisions trustworthy. A cheaper model should not automatically be considered better. Decisions need to account for available quality evidence, token costs, latency, and benchmark provenance.
Finally, generating automated code changes introduced security and reliability concerns. We addressed these through exact source-range tracking, immutable review artifacts, candidate validation, and explicit separation between patch proposals and source application.
Accomplishments that we're proud of
- Built a modular compiler architecture for analyzing Python-based AI repositories.
- Implemented static AST scanning without executing target repository code.
- Developed data-dependency graph construction for visualizing agent workflows.
- Created structured benchmark job management and evidence evaluation.
- Implemented an optimization engine that uses retained evidence to generate routing policies.
- Developed source-level patch generation with static validation and reviewable diffs.
- Built an interactive developer workbench connecting the analysis, benchmarking, optimization, and patch-review workflows.
- Established automated unit, integration, and security test coverage.
We are particularly proud of approaching AI optimization as an engineering and verification problem rather than simply replacing one model name with another.
What we learned
We learned that effective AI optimization requires more than selecting the smallest or cheapest model.
Reliable model routing requires understanding task complexity, preserving output quality, measuring performance, and maintaining traceable evidence behind every optimization decision.
We also learned the importance of secure static analysis, reproducible benchmarking, strict API contracts, and separating generated code proposals from actual source modifications.
Most importantly, we learned that agentic engineering tools should make developers more confident in their decisions, not introduce another layer of unpredictable automation.
What's next for CascadeRouter
Our next steps include:
- Expanding live benchmarking against Nebius Token Factory and NVIDIA Nemotron models.
- Integrating broader model-provider support and real-time pricing intelligence.
- Improving quality-parity evaluation using diverse and adversarial test cases.
- Advancing automatic model-cascade generation and runtime integration.
- Strengthening regression testing and end-to-end verification of generated patches.
- Introducing richer repository insights and developer workflow integrations.
Our long-term vision is to make CascadeRouter an intelligent compiler for AI applications: one that continuously analyzes, evaluates, and optimizes AI architectures while keeping developers in control.
Log in or sign up for Devpost to join the conversation.