Inspiration
Running an LLM is easy to demonstrate: start a server, send a prompt, and show the generated text.
Operating inference reliably is much harder.
I wanted to understand what happens when inference must serve concurrent requests, preserve streamed responses during failure, move models between workers, expose useful metrics, and run across different hardware—including an Arm64 cloud worker.
That question became inferMesh: an OpenAI-compatible inference control plane and benchmarking toolkit built around llama.cpp.
The project combines a local Windows/NVIDIA worker with a real Google Cloud C4A Axion Arm64 worker. It measures both workers, routes requests between them, manages their model processes, and validates failure recovery without pretending that a second local process is a remote machine.
What it does
inferMesh provides:
- An OpenAI-compatible
/v1/chat/completionsgateway - A streaming benchmark for TTFT, latency, throughput, RAM, and GPU utilization
- Request admission with bounded concurrency and queue timeouts
- Round-robin, least-inflight, prefix-aware, hybrid, and multi-SLO routing
- Stream-stage-aware retries that avoid duplicating visible tokens
- Process-backed model loading, draining, replacement, and eviction
- Authenticated lifecycle APIs using logical allowlisted model IDs
- Prometheus metrics and OpenTelemetry traces
- Privacy-safe JSONL evidence without prompt or completion bodies
- Private inference and administration through separate Google Cloud IAP tunnels
- Real worker-kill, failover, recovery, and cleanup validation
Why Arm64 and Google Axion?
The remote worker runs CPU inference on a Google Cloud C4A Axion VM.
The worker agent is cross-compiled from Go for Linux Arm64 and deployed to the Axion machine. llama.cpp and quantized GGUF models remain self-hosted, so inferMesh does not hide inference behind a third-party hosted LLM API.
Axion adds something that two local processes cannot provide: an independent machine, architecture, network path, and failure domain.
This made it possible to test:
- x86/RTX and Arm64/CPU workers in one mesh
- Heterogeneous latency and capacity
- Private Arm64 model serving
- Local failure while the Axion worker remains available
- Per-worker model placement and recovery
- Reusable deployment and validation workflows for Arm cloud inference
Architecture
A request enters the Go gateway, acquires admission capacity and a lifecycle lease, and is assigned by the configured routing policy.
The selected worker streams an SSE completion through the gateway. The local worker runs llama.cpp on Windows, while the remote worker runs llama.cpp on Axion Arm64 through a private IAP tunnel.
Lifecycle operations use a separate authenticated IAP tunnel. The remote agent owns its llama-server process and maps logical model IDs to administrator-controlled GGUF files.
This separation prevents API clients from supplying arbitrary filesystem, drive, traversal, device, or UNC paths.
How I built it
The project evolved through several measured stages.
1. Establish trustworthy measurements
I first built a Go streaming client that measures:
- Time to first visible token
- Full request latency
- Tokens per second
- SSE correctness and
[DONE] - Stable error classes
- Process RAM
- GPU memory and utilization
Raw evidence stores prompt IDs and SHA-256 hashes, not prompt or completion text.
2. Compare local execution choices
I compared CPU and GPU execution, seven quantizations from the same Qwen2.5 model family, context sizes, thread counts, parallel slots, and concurrency levels.
On the documented Windows machine, GPU execution improved mean decode rate from approximately 69.3 to 143.5 tokens/second and reduced mean latency from approximately 847.5 ms to 412.2 ms.
These are local RTX measurements and are not presented as Axion speedup numbers.
3. Separate admission from continuous batching
One important lesson was that delaying HTTP requests in Go is not model batching.
The Go layer controls admission and queueing. Actual continuous token batching happens inside llama-server through its continuous-batching and parallel-slot configuration.
In the measured local matrix, enabling llama-server continuous batching increased throughput from 8.10 to 12.13 requests/second, while median TTFT improved from 131 ms to 89 ms.
4. Add routing and safe retry
Prefix affinity improved repeated-prefix TTFT in one workload, but strict affinity created a hotspot when one prefix dominated traffic.
That result led to hybrid policies that combine cache affinity with worker load.
Retry also became request-stage-aware. A request may be retried before visible output, but a broken stream is not replayed after tokens have already reached the client.
5. Introduce a real Arm64 worker
I deployed a self-hosted llama.cpp worker on Google Axion and connected it through private IAP tunnels.
The acceptance workflow verifies distinct worker URLs, health endpoints, model identities, and complete SSE responses. Missing or malformed Axion configuration fails closed; inferMesh never substitutes another local worker and labels it “remote.”
6. Make lifecycle state physical
The original lifecycle prototype could change an in-memory model label without changing the process.
I replaced it with process-backed lifecycle control. A selected worker drains active leases, terminates its owned model process, starts the allowlisted replacement GGUF, and verifies model identity before returning to ready.
7. Harden the release workflow
The final hardening pass added:
- Bearer authentication for model mutation
- Logical model allowlists
- Filesystem and traversal rejection
- BOM-safe generated configuration
- Rebuilt and hashed demo binaries
- Exact worker-targeted recovery
- Consecutive complete SSE recovery warmups
- Fresh-checkout validation
- Verified process, tunnel, firewall, disk, and VM cleanup
Results
The final real local-plus-Axion acceptance run recorded:
- Normal load: 12/12 successful requests
- Overload scenario: 20/20 successful requests
- Real local process-kill failover: 7/8 completed
- Explicit killed-worker recovery: 6/6 completed
- Recovery warmup: 3 consecutive complete streams
- Lifecycle transition: Q4 → Q2 → Q4
- Round-robin distribution: 8 local and 8 Axion requests
- Critical acceptance gates: 9/9 passed
The release was also repeated from a clean checkout at an exact Git revision. The demo rebuilt its binary, recorded source and binary hashes, and left the tracked working tree clean.
After validation, the temporary environment was removed. The final audit found zero remaining InferMesh VMs, firewall rules, disks, llama-server processes, listeners, or IAP tunnels.
Challenges and lessons
Several attractive ideas did not become defaults.
A short adaptive-admission experiment improved one traffic spike, but a longer fair-start soak produced worse tail latency and excessive policy switching.
Speculative decoding with the measured small Qwen draft/main pair consumed more memory and reduced throughput.
Strict prefix affinity improved cache reuse but overloaded one worker under a hot-prefix workload.
These negative results were valuable because they changed the production defaults instead of being hidden.
The largest remaining reproducibility limitation is clean-VM bootstrap. The current public workflow can redeploy an already provisioned Axion worker, but installing and verifying llama.cpp from a completely new Ubuntu C4A VM still needs to become an end-to-end automated step.
Why it should win
inferMesh is not only an inference demo. It is a reusable, evidence-driven foundation for developers migrating self-hosted AI inference to Arm64 cloud infrastructure.
It combines:
- Real Arm64 inference
- Production-minded routing and lifecycle control
- Secure private transport
- Measurable optimization work
- Honest negative results
- Reproducible validation
- Detailed engineering documentation
The project shows both how to adopt Arm cloud inference and how to prove that the resulting system is real, observable, recoverable, and safe to operate.
Log in or sign up for Devpost to join the conversation.