UniEdge

System Architecture
Workflow
Comparison between original and Optimized model

UniEdge: Gemma-3 on Arm Inspiration

Every "edge AI" demo I've seen runs on the latest hardware a Pi 5, a Jetson, an M-series Mac. But the real world is full of older, cheaper devices: Cortex-A72 boards, aging industrial gateways, budget smartphones. Those devices aren't getting a hardware upgrade. The main focus on the domain Physical AI.

That question became UniEdge- a universal ML runtime designed so the same optimization pipeline adapts automatically per device class, from wearables to drones to cameras. This hackathon submission is the first concrete experiment: run a modern 270M-parameter language model on a Raspberry Pi 4, measure what we actually gain, and build something genuinely useful on top of it.

How I Built It Quantized model benchmarking: I took Gemma-3 270M-IT in two GGUF quantization levels Q8_0 (279 MB) as the baseline and the Gemma-3-270M-UniEdge variant (242 MB) as the optimized target. I built llama.cpp from source on the Pi 4 with Arm's KleidiAI library explicitly enabled (DGGML_CPU_KLEIDIAI=ON), ran three identical inference passes per model, and averaged the results.

Hardware-level profiling: Instead of just measuring wall-clock time, I used Linux perf stat to capture CPU cycles, instructions executed, and cache behaviour for every run. This let me confirm whether the optimized model was actually doing less computational work, not just appearing faster due to memory effects.

Local memory database: To give the model persistent memory without any fine-tuning (impossible on a Pi 4), I built a SQLite-backed system. It extracts keywords from each user message, scores past interactions by keyword overlap, and injects the top two most relevant memories directly into the system prompt before inference. Every new exchange is saved back to the database. No embeddings, no external APIs — just Python and SQLite, running fully offline.

What I Learned

Quantization tradeoffs are non-linear. The UniEdge model is 13% smaller, but prompt evaluation was 11% slower while generation was 15% faster, and end-to-end time dropped by 17%. The two inference phases behave very differently under quantization, and understanding why how GGUF handles KV-cache vs. compute paths differently was the biggest modelling insight.

Arm-optimized is a spectrum, not a binary. KleidiAI's fastest kernels require dotprod, i8mm, SVE, or SME instruction extensions. The Pi 4's Cortex-A72 has none of them — confirmed directly in the CMake output. So inference fell back to NEON-level SIMD, which is still Arm-specific acceleration, just not KleidiAI's newest path. On a Cortex-A76+ or Pi 5, those kernels would activate automatically with no code changes. This is exactly the device-class-aware routing UniEdge is built around.

perf stat tells a richer story than a stopwatch. The UniEdge model completed the same task with 16% fewer CPU cycles, 14% fewer instructions, and 9% fewer cache references. That confirms it's doing genuinely less work — not just exploiting a clock-speed artifact. IPC improved slightly too, meaning the CPU was busier per cycle.

SQLite is a surprisingly capable memory backend for personal assistant workloads. When you can't afford embeddings or GPU inference, keyword overlap scoring works well because users tend to reuse consistent vocabulary. The constraint of the Pi 4 forced a lean, effective design.

Challenges

Building on the Pi 4 is slow. A full llama.cpp compile takes over 20 minutes with only two parallel jobs. Any misconfiguration meant starting over. I ended up scripting the entire build pipeline to make it reliably reproducible.

Debugging KleidiAI non-activation. When the accelerated kernels didn't fire, I wasn't sure if it was a misconfiguration or a genuine hardware ceiling. Tracing through the CMake output, the llama.cpp runtime logs, and the KleidiAI source to confirm that kernel selection is a runtime CPU-feature dispatch not a compile-time choice — required real digging. It became the most instructive finding of the whole project.

Streaming subprocess output.Getting character-level real-time streaming from subprocess.Popen in the memory chat shell required careful buffering configuration, stderr routing to DEVNULL, and correct process lifecycle management to avoid deadlocks.

Measuring quality retention rigorously. I assessed output coherence qualitatively — both models produced consistent, on-topic answers across all runs — but a proper quantitative eval (perplexity, task accuracy) remains future work. Defining "good enough" for a compressed model without a proper eval harness is harder than it sounds.