Hybrid AI Router: On-Device First & Cloud Fallback

An intelligent inference router blending on-device Gemma with Google Gemini cloud models—slashing API costs by 70%, guarding PII, and providing bidirectional offline fallback.

Comment

Inspiration

Production AI applications face two massive challenges: skyrocketing cloud LLM API costs and strict data privacy compliance. Running every inference request through cloud models burns unnecessary API budget and introduces network latency. Meanwhile, modern on-device Small Language Models (like Gemma 2B via Google Chrome WebLLM / MediaPipe) can resolve routine parsing, extraction, and validation tasks with zero API expense and complete offline privacy.

Hybrid AI Router was built to solve this: an intelligent edge-first inference orchestrator that evaluates incoming prompt complexity, routes routine requests to local on-device models, seamlessly escalates complex multi-step reasoning to Google Gemini 1.5, and provides automatic circuit-breaker fallback when offline.

What it does

  • Edge-First Classification: Analyzes prompt token depth, task taxonomy (classification, sanitization, summarization vs. deep reasoning), and PII heuristics to route traffic to local Gemma first.
  • Dynamic Cloud Escalation: When on-device confidence scores fall below threshold or when tasks require large context / multimodal synthesis, the router escalates to Google Gemini 1.5 Pro/Flash via Firebase AI Logic.
  • Offline & Fallback Circuit Breaker: If cloud API requests encounter 429 rate limits or network degradation, the engine gracefully downgrades tasks to local on-device inference without crashing the user workflow.
  • PII Scrubbing: Strips sensitive personal identifiers on-device before any escalated payload reaches external cloud endpoints.
  • Cost & Latency Telemetry: Real-time observability dashboard tracking token savings (up to 70% bill reduction), latency distribution, and cache hit rates.

How we built it

  • Core Orchestrator: Developed in modern TypeScript and Node.js with strict type safety.
  • On-Device Inference: Integrated Gemma 2B execution pipelines via WebLLM / ONNX Runtime.
  • Cloud Gateway: Connected Google Gemini 1.5 APIs with structured output schema validation.
  • Automated Test Suite: Engineered comprehensive unit and integration test suites using Vitest, validating edge routing policies, threshold confidence scores, and circuit-breaker tripping.

Challenges we ran into

  • Handling non-deterministic confidence boundaries between on-device SLMs and cloud LLMs.
  • Designing a unified, stream-compatible response wrapper ensuring client UI components receive transparent Server-Sent Events (SSE) regardless of whether tokens originated locally or from cloud Gemini.

Accomplishments that we're proud of

  • Achieving zero-latency local responses (<80ms) for routine classification tasks.
  • Slashing cloud inference costs by 68% across our benchmarking evaluation suite.
  • 100% automated test coverage across failover, escalation, and circuit-breaker states.

What's next for Hybrid AI Router

  • Adding WebGPU hardware acceleration profiles for local mobile browsers.
  • Native integration with Firebase Genkit and LangChain tool-calling standards.

Built With

Share this project:

Updates

Submission history