💡 Inspiration
Most AI applications are fragile wrappers that blindly generate text. We wanted to build an Enterprise Inference OS that guarantees security, verifies execution, and optimizes API costs using NVIDIA Nemotron and Nebius infrastructure. We realized that the true bottleneck in managed APIs is not just generating code, but ensuring it actually runs without crashing.
⚙️ What it does
SpeculativeStream is an AI Inference Operating System that acts as an intelligent, fault-tolerant controller for NVIDIA Nemotron models:
- Enterprise Guardrails: Intercepts and blocks malicious prompts before they hit the API, ensuring safe deployment.
- Semantic Speculative Decoding: Runs parallel draft models (Code vs. Logic) and uses vector divergence to abort requests if drafts conflict, saving API costs.
- Cloud Execution Sandbox: Uses Nebius ConTree Sandboxes to run generated code in secure microVMs. If the code crashes, the system captures the logs and triggers a Self-Healing Loop with the Nemotron 550B model.
- X-Ray Telemetry: Color-codes the verifier's output (Green = Accepted, Red = Corrected) and tracks exact token costs and Trace IDs in a Flight Recorder.
- RL Routing: Uses reinforcement learning based on user feedback to dynamically shift traffic between Fast Paths (30B) and Deep Swarms (550B).
- Multimodal Vision: Uses NVIDIA's Vision-Language models to generate code directly from uploaded screenshots.
🛠️ How we built it
- Backend: Python, FastAPI, asyncio for parallel drafting, and the Nebius Token Factory (Nemotron 550B, Nano 30B, Qwen 30B).
- Infrastructure: Nebius ConTree Sandboxes for secure cloud code execution.
- Frontend: Streamlit with a custom Cyberpunk/Glassmorphism UI.
- MLOps & Security: AST-based Quality Gates, scikit-learn for Vector Divergence, and a Circuit Breaker for API timeouts.
🚧 Challenges we ran into
Handling the probabilistic and often hallucinated nature of LLMs. We solved this by building a Circuit Breaker for API timeouts, an AST Quality Gate for syntax verification, and a Cloud Sandbox for runtime verification. We also had to solve the challenge of running untrusted AI code securely, which is why we integrated ConTree Sandboxes.
🏆 Accomplishments that we're proud of
Building a true multi-agent orchestration engine that doesn't just wrap an API, but actively measures, optimizes, and heals its own output. The X-Ray stream makes the invisible mathematics of speculative decoding completely visceral and easy to understand.
📚 What we learned
How to manage parallel async API calls, AST code verification, building an intelligent semantic caching layer, and how to execute untrusted code safely in the cloud using Nebius infrastructure.
🚀 What's next for SpeculativeStream
Integrating Nebius's Custom Speculator Training API to fine-tune workload-specific drafters based on the Flight Recorder data, creating a continuous data flywheel that makes the system smarter and cheaper with every request.
Built With
- asyncio
- nebius
- nvidia-nemotron
- plotly
- python
- scikit-learn
- streamlit
