-
-
Overview
-
Architecture
-
v1 and v2
-
Results
Inspiration
Every LLM response actually does two different jobs: reading the prompt (fast, parallel, compute-bound) and writing the answer one token at a time (slower, memory-bandwidth-bound). Production systems like NVIDIA's Dynamo architecture already split these two phases across separate GPU pools at data-center scale. Prefill needs your laptop's GPU, decode doesn't, so the moment prefill finishes, that compute is free for other work while your phone handles decode. I wanted to know if the same idea held up at the smallest possible scale, using hardware already sitting on a desk instead of a second GPU: a laptop for prefill, an Android phone for decode.
What it does
EdgeSplit splits LLM inference across two ordinary devices connected over WiFi. A CUDA-equipped laptop runs prefill, then hands the live model state to an Android phone, which finishes generating the response. Two handoff mechanisms are implemented and benchmarked: V1 saves state to a file and uploads it over HTTP, V2 patches llama.cpp to export the raw sequence state directly and streams it over a custom TCP protocol, skipping the file entirely. Across two model families (Qwen3-0.6B, Llama-3.2-1B-Instruct) and three quantizations each, V2 cuts time-to-first-token by roughly 60 to 80% on most configurations, measured with warmed, repeated, statistically tracked benchmarks, not single samples.
How I built it
The laptop side runs CUDA llama.cpp inside WSL2 Ubuntu, a FastAPI router orchestrates requests, and the phone side runs a matching, commit-pinned llama.cpp build inside Termux. V2 required patching llama.cpp itself to expose two new endpoints for raw sequence-state access, then writing a binary framing protocol (shape, dtype, sequence length, SHA-256 integrity check) over a raw TCP socket. Codex, running on GPT-5.6, built and debugged the implementation continuously, at high/xhigh effort for the hard problems (the KV-cache protocol, the phone crash investigation, the kernel optimization). Every benchmark used a warmed repetition methodology (excluded warm-up pair, five retained pairs, mean/median/standard deviation) rather than single-run numbers.
Challenges I ran into
The original phone's from-source build segfaulted reliably on model load. I ruled out CPU dispatch flags, OpenSSL linking, context size, threading, and an API-level GWP-ASan hypothesis, each with real evidence, before finally switching to a second, rooted device where the identical build ran cleanly, confirming it was device-specific, not a code bug. Profiling the phone's real decode bottleneck took three separate attempts (an app-permission restriction, a silent failure, a wrong CLI flag) before it worked. Power instrumentation was its own saga: the phone's onboard fuel-gauge sensor, on a device roughly a decade old, produced physically impossible readings (over 100 amps) across every charge state I tested, eventually resolved by switching to Android's official BatteryManager API instead of the raw hardware sensor. Throughout, several "PASS" results turned out to be false positives (empty files diffed against each other, wrong binaries compared), caught only by checking the actual underlying data rather than trusting a green checkmark.
Accomplishments that I'm proud of
A working, cross-device, commit-matched V1 and V2 implementation with real, reproducible, statistically meaningful benchmark data, not a demo that only works once. A hand-profiled and hand-optimized NEON kernel, verified for byte-identical correctness before a single speed number was trusted. And an honest project: every limitation (the first phone's incompatibility, the power sensor's unreliability, the one benchmark result that fell within noise) is documented plainly instead of hidden.
What I learned
Disaggregating inference is a real systems tradeoff, not just an architecture diagram, memory bandwidth, network handoff overhead, and device-specific quirks all matter more than raw compute in practice. I also relearned a more general lesson: verify before trusting, a passing check that hasn't been interrogated isn't evidence, it's an assumption waiting to be wrong.
What's next for EdgeSplit
Prefix-reuse caching for repeated prompt prefixes, testing across more phone models to see how broadly the V2 speedup generalizes, and real hardware power measurement via an inline USB-C power meter now that the phone's onboard sensor has proven unreliable on aged hardware.
Log in or sign up for Devpost to join the conversation.