Cascade-Router

Inspiration

For Track 1 of the Syndicate Hackathon, we wanted to tackle one of the biggest bottlenecks in modern AI development. As AI agents become more autonomous, they execute increasingly complex sequences of operations. Defaulting to a massive frontier LLM for every minor step is financially unviable and heavily degrades latency. Inspired by speculative execution in modern CPUs, we wanted to build a routing engine that safely speculates using low-cost student models, featuring deterministic sandboxing to seamlessly fall back to an expensive teacher model the moment a hallucination or failure is detected.

What it does

Cascade is a high-performance speculative execution routing layer for AI agents. When an application sends an LLM request, Cascade intercepts it and performs a cosine-distance vector search using Ollama embeddings to instantly fetch the exact Standard Operating Procedure (SOP) for that user's intent.

It then deploys a cheap, fast student model (like DeepSeek Flash) to generate a response. Instead of blindly trusting the output, Cascade locks the response inside an isolated Node VM sandbox. It is passed through a strict two-layer deterministic gate that rigorously validates the JSON schema structure and executes invariant logic. If the student passes, the fast path succeeds. If the student hallucinates or fails, the router catches the error and autonomously falls back to a highly capable teacher model (like DeepSeek Pro) to repair the output.

How we built it

Architecture: Utilized Agent Orchestrator (AO) extensively from start to finish to design and write the underlying architecture.
Backend: High-performance TypeScript backend (Express.js) connected to a Neon serverless Postgres database, using pgvector for instant intent classification.
AI & Routing: Integrated Ollama for local request embeddings and utilized the TensorMux API to execute our DeepSeek student and teacher models.
Observability: Instrumenting the entire pipeline with Neatlogs to capture nested trace spans for workflow latency, token costs, and automated routing fallback decisions in real time.

Challenges we ran into

Building a secure, deterministic sandbox in TypeScript that could execute arbitrary, AI-generated invariant logic without crashing the main router thread was incredibly challenging. We had to utilize the node:vm module with strict execution timeouts (100ms) to ensure runaway AI code would not cause infinite loops. Additionally, handling asynchronous race conditions during Postgres vector matching and ensuring clean state rollbacks when the student model failed required precise transaction management.

Accomplishments that we're proud of

We built a fully functioning, dual-model speculative execution pipeline in under 24 hours. Seeing the system autonomously catch a student hallucination in a secure sandbox, reject it, failover to a teacher model, and successfully return the repaired output while live-streaming the entire telemetry trace to Neatlogs was an incredible feeling. We are incredibly proud of how smoothly AO allowed us to architect such a massive undertaking in a compressed timeframe.

What we learned

We learned exactly how powerful Agent Orchestrator can be when designing large-scale system architectures. We also mastered the intricacies of Postgres vector similarity search operations, specifically navigating pgvector cosine distance ceilings, and the security nuances of isolated execution environments in Node.js.

What's next for Cascade-Router

Currently, a student failure immediately triggers the expensive teacher model. Our next step is to introduce an automated self-reflection loop, giving the cheap student model one chance to repair its own sandbox errors before we pay for the teacher model. We also plan to integrate an async learning loop where the teacher model periodically analyzes failure traces in Neatlogs to rewrite and optimize the SOPs stored in the database.

Tags: artificial-intelligence, machine-learning, llm, autonomous-agents, agent-orchestrator, routing-engine, speculative-execution, typescript, node.js, postgresql, vector-search, ollama, deepseek, neatlogs, tensormux

Built with: agent-orchestrator, typescript, node.js, express.js, postgresql, pgvector, neon, ollama, deepseek, tensormux, neatlogs