Lantern Edge

Inspiration

Every AI agent today is a thin wrapper around a cloud API. When the network drops — on a factory floor, in a rural clinic, on a plane, in a subway — the agent dies. Users get nothing. We asked: what if the agent never stopped working?

What it does

Lantern Edge is an offline-first edge AI agent that runs real Qwen models directly on-device and degrades gracefully when connectivity drops. It uses a model-free classifier — pure functions, no LLM call needed — to route every task to the cheapest brain that can handle it:

⚡ Local Tools (time, calculator, notes) — zero network, zero model, instant
🧠 Edge Models (Qwen2.5 0.5B–3B via WebGPU) — on-device inference for privacy-sensitive or offline tasks
☁️ Qwen Cloud (DashScope) — streaming completions for complex reasoning when online
⏳ Deferred Queue — complex tasks queued offline, refined automatically on reconnect

The core innovation: offline never means failure. Simple tasks use local tools instantly. Complex tasks run on the edge model best-effort. Tasks too hard for the edge get queued and refined by Qwen Cloud the moment connectivity returns. The agent never breaks.

How we built it

Three-layer architecture:

Edge Layer (Browser): React + Vite frontend with WebLLM/WebGPU for on-device Qwen model inference. Includes a tool execution engine (time, calculator, notes) and a deferred-queue manager that drains on reconnect.
Routing Layer (Model-Free): The heart of the system. A pure-function classifier analyzes task complexity, capability requirements, privacy sensitivity, and reasoning depth — then a degradation policy (decideRoute()) picks edge / cloud / defer. No model needed for the decision. Fully unit-tested.
Server Layer (Express): Qwen Cloud streaming proxy via DashScope API (international endpoint) with SSE token streaming. Alibaba Cloud OSS for persistent cross-device state sync with local-disk fallback.

Qwen Cloud integration: Server-side proxy to DashScope (dashscope-intl.aliyuncs.com) supporting qwen-turbo, qwen-plus, and qwen-max. API key stays server-side — never reaches the browser. The agent loop is tool-augmented (maxSteps: 4), so Qwen can call local tools between reasoning steps.

Alibaba Cloud integration: OSS (Object Storage Service) in oss-cn-hangzhou for cross-device conversation sync. Full conversation history persists across devices. Graceful fallback to local disk when OSS is unavailable.

Challenges we ran into

WebGPU hardware compatibility: On-device inference only works on machines with WebGPU support. We designed the entire degradation flow so that even without WebGPU, the agent still works — it defers to Qwen Cloud or uses local tools.
Model-free routing accuracy: Getting the classifier to make good routing decisions without any LLM call required careful tuning of complexity scoring, capability coverage analysis, and reasoning depth estimation. Every path is unit-tested (35 tests passing).
Seamless degradation: The hardest part was making the offline → online transition smooth. The deferred queue needed to drain automatically on reconnect, update responses in place, and never lose data.

Accomplishments that we're proud of

35 unit tests covering classifier, degradation policy, agent loop, and calculator — all passing
Real on-device inference via WebGPU with Qwen2.5 models, not a mock or simulation
The degradation flow works end-to-end: online → load edge model → go offline → edge answers + complex task deferred → reconnect → Qwen Cloud refines automatically
Zero models needed for routing — pure functions make every decision

What we learned

Designing for graceful degradation fundamentally changes how you architect an AI agent. Every component needs an offline story.
Model-free routing is surprisingly effective — you don't need an LLM to decide which LLM to use.
WebGPU + WebLLM makes real on-device inference practical for small models, but hardware support is still uneven.