Juno

Load-time compiler and runtime for efficient low-bit SME2 inference

Comment

Inspiration

I was looking into a local inference setup for Hermes, so I asked it to help me compare the models, runtimes, and hardware I could use. I expected the work to stay at that level. Instead, the research introduced me to SME2, an Arm extension for matrix processing that I had never encountered before.

I then learned that my M4 already had SME2, which made me wonder how much of my stack was actually using it. Hermes also surfaced the paper Intelligence per Watt: Measuring Intelligence Efficiency of Local AI, which asks how much useful intelligence local hardware produces per unit of power. It changed what I wanted to measure from throughput alone to throughput and energy per token.

The search led me into llama.cpp's Arm CPU path and the way low-bit GGUF models store their weights. The models were compact, but their encoded weights were not arranged in the form an SME2 kernel wants to consume.

Juno closes that gap by rearranging eligible weights once when the model loads, then reusing the SME2-native representation for the lifetime of the model.

What it does

Juno is a load-time weight compiler and CPU inference runtime for efficient low-bit inference on Arm SME2.

Juno was built during the Arm AI Optimization Challenge for the Mobile AI track. It improves fully local, energy-efficient inference on Arm-powered client devices.

When a model loads, Juno inspects each tensor and determines whether its format and dimensions are supported. Eligible tensors are rearranged into layouts designed for Juno's SME2 kernels. Other tensors continue through the existing llama.cpp CPU path.

Juno currently accelerates Q2_K, IQ2_S, IQ2_XS, IQ3_XXS, IQ3_S, and IQ4_XS tensors. GGUF models commonly contain several tensor formats, so Juno makes this decision tensor by tensor rather than from the model filename.

The runtime is available through a command-line interface, an OpenAI-compatible local server, and a native macOS app. The app provides model inspection, one-turn inference, isolated Native-versus-Juno benchmarking, hardware details, and local server controls. Each interface uses the same underlying runtime and records which Juno kernels actually executed.

Results

Juno was measured on an M4 Pro using the same CPU-only llama-server binary with Juno disabled and enabled. Two threads is the default configuration for Juno's command-line tools.

Juno reduced energy per generated token across all nine measured configurations

Model and GGUF Threads End-to-end generation throughput Idle-adjusted CPU + GPU energy per generated token
Granite 4.1 3B, IQ4_XS 1 11.70 → 21.54 t/s (+84.2%) 0.5237 → 0.1895 J (-63.8%)
Granite 4.1 3B, IQ4_XS 2 21.37 → 34.48 t/s (+61.3%) 0.5368 → 0.2135 J (-60.2%)
Granite 4.1 3B, IQ4_XS 4 39.59 → 34.16 t/s (-13.7%) 0.5555 → 0.3114 J (-43.9%)
Gemma 4 E4B, UD-IQ3_XXS 1 3.34 → 5.41 t/s (+61.9%) 1.8692 → 0.6609 J (-64.6%)
Gemma 4 E4B, UD-IQ3_XXS 2 6.36 → 9.68 t/s (+52.1%) 1.8494 → 0.6866 J (-62.9%)
Gemma 4 E4B, UD-IQ3_XXS 4 11.95 → 8.23 t/s (-31.2%) 1.8949 → 1.1798 J (-37.7%)
Qwen3.5 0.8B, UD-Q2_K_XL 1 34.86 → 44.48 t/s (+27.6%) 0.1907 → 0.1114 J (-41.6%)
Qwen3.5 0.8B, UD-Q2_K_XL 2 62.48 → 64.45 t/s (+3.2%) 0.1938 → 0.1302 J (-32.8%)
Qwen3.5 0.8B, UD-Q2_K_XL 4 110.27 → 74.00 t/s (-32.9%) 0.2050 → 0.1626 J (-20.7%)

Higher throughput and lower energy per token are better. Each value is shown as Native → Juno at the listed thread count.

The three mixed-format models collectively exercised all six Juno kernels. Across all nine comparisons, Juno reduced measured CPU and GPU energy per generated token by 20.7% to 64.6%. At one and two threads, it also improved throughput in all six comparisons, by 3.2% to 84.2%.

Native was faster at four threads on all three models, while Juno still used 20.7% to 43.9% less energy per generated token. The current implementation is most effective for low-thread, energy-constrained inference.

Native and Juno produced exactly matching output at every measured thread count. Duplicate-run relative ranges remained within 3.16% for throughput and idle-adjusted energy.

Each comparison used the same warmed server binary, one discarded warm-up, deterministic 512-token requests, and Native, Juno, Juno, Native ordering. Apple powermetrics sampled CPU and GPU energy during request windows longer than 30 seconds. Sixty-second idle baselines were collected before and after each model. Model loading, Juno weight compilation, warm-up, and cooldown were outside the measured request windows.

The published evidence contains one uninterrupted sweep with 36 measured runs, six idle baselines, and 114 hashed compressed artifacts. It includes raw powermetrics output, server traces, response bodies, model hashes, and the measurements used to derive the results.

The evidence can be verified without SME2 hardware:

python3 scripts/verify-evidence.py

The verifier checks every published artifact, locates all 36 measurement windows inside the raw powermetrics output, validates response and dispatch records, and recomputes the aggregate Native and Juno comparisons.

How I built it

GGUF blocks store packed weight codes together with format-specific scales, signs, codebook indexes, and corrections. This representation keeps models compact, but values needed together by an SME2 calculation are not always adjacent or arranged in the order the kernel consumes them.

Juno rearranges those fields at model load. The current layouts group output rows into 64-row tiles and store encoded values and metadata in streams that the corresponding SME2 kernel can load directly. The original quantized values and scales are preserved.

IQ4_XS weights before and after Juno arranges them into an SME2-native layout

Each supported format is implemented through five connected parts:

  1. A shared packed-layout definition
  2. A weight packer that writes the layout
  3. An SME2 calculation that reads the layout
  4. Boundary and numerical tests
  5. llama.cpp integration that allocates, packs, and dispatches the tensor

The shared layout defines the offsets, sizes, and alignment used by every producer and consumer of the packed buffer. The packer, calculation, tests, and integration use the same layout rather than maintaining separate copies of the buffer arithmetic.

At runtime, Juno checks for Arm64, SME2, a 512-bit streaming vector length, a supported tensor format, and compatible tensor dimensions. If any requirement is not met, the tensor stays on the existing llama.cpp CPU path.

The backend is written in C and C++. CMake builds the runtime, tests, command-line tool, and Juno-enabled llama-server. The macOS app is written in Swift and SwiftUI and connects to the same runtime used by the command-line tools.

Challenges I ran into

The first challenge was understanding the low-bit formats themselves. Q2_K stores values across bitplanes, while the IQ formats use codebook indexes, packed signs, scales, and other format-specific metadata. Each path needed a different packed layout while preserving the same calculation.

Correctness had to be checked at several levels. Individual calculations are compared with their reference implementations across boundary cases and numerical tolerances. End-to-end benchmarks also compare the deterministic Native and Juno output sequences.

Runtime dispatch required separate verification. Hardware detection and successful compilation do not establish that a model executed an SME2 kernel. Juno records dispatch receipts from the kernels that ran, and the app, CLI, tests, and published server traces use those receipts when reporting SME2 execution.

Energy measurement introduced a different set of problems. Laptop power changes with background activity, temperature, and run order. The published evidence process uses repeated measurements, counterbalanced ordering, idle baselines, minimum measurement windows, retained raw output, and dispersion checks.

The thread sweep exposed the current scaling boundary. Juno improved throughput at one and two threads across the measured models, while Native scaled better at four threads. Publishing the complete grid made it possible to describe both the current efficiency advantage and the remaining scaling work.

Accomplishments I am proud of

Juno now has six custom low-bit SME2 paths running inside real llama.cpp model execution. The same backend supports the native app, command-line tools, benchmark, and OpenAI-compatible local server.

Every published Native and Juno output matched exactly. Juno used less measured energy per generated token in every cell of the three-model, three-thread sweep, and the repository retains enough evidence for those results to be recomputed without access to the original machine.

The project is licensed under Apache 2.0. Its shared layouts, packers, calculations, tests, and tracked llama.cpp integration patch provide a worked implementation that developers can inspect when extending the approach to another low-bit format.

What I learned

Portable model formats and hardware-native execution layouts solve different problems. A representation can be well suited to model storage and distribution without being the best form for a particular processor to execute.

Moving the translation to model load makes it possible to reuse a representation designed around the target hardware. The change does not require retraining or requantizing the model.

I also learned why throughput, power, and energy need to be considered separately. Lower power does not always mean lower energy if the work takes longer. Higher throughput can require enough additional power to reduce efficiency. Energy per generated token accounts for both the energy consumed and the work completed.

Mixed-format models were another important part of the project. A model filename describes its overall quantization preset, but the file can contain several tensor formats internally. Juno therefore inspects and routes individual tensors rather than deciding support from the filename.

How to run it

Juno currently requires an Arm64 system with SME2 and a 512-bit streaming vector length, a C and C++ compiler with SME2 support, CMake 3.20 or later, and Git. It is verified and measured on an M4 Pro. SME2 detection is implemented for macOS and Linux.

After cloning the Juno repository, fetch the pinned llama.cpp revision and apply the tracked integration patch:

git clone https://github.com/ggml-org/llama.cpp vendor/llama.cpp

git -C vendor/llama.cpp checkout \
  6ea215d171fd31df943bf1ac8227129f2b963160

git -C vendor/llama.cpp apply \
  ../../patches/llama-juno.patch

Configure and build the runtime:

cmake -S . -B build-juno-runtime \
  -DCMAKE_BUILD_TYPE=Release \
  -DJUNO_BUILD_RUNTIME=ON \
  -DBUILD_SHARED_LIBS=OFF \
  -DGGML_METAL=OFF \
  -DGGML_ACCELERATE=OFF \
  -DGGML_BLAS=OFF \
  -DGGML_CPU_KLEIDIAI=ON

cmake --build build-juno-runtime -j \
  --target juno llama-server

Install the command-line tool:

sudo cmake --install build-juno-runtime \
  --prefix /usr/local

Inspect the tensors Juno can accelerate:

juno inspect /path/to/model.gguf

Run a prompt:

juno run /path/to/model.gguf \
  --prompt "Explain why local inference is useful."

Compare Native and Juno using isolated repeated runs:

juno benchmark /path/to/model.gguf

Start the OpenAI-compatible local server:

juno serve /path/to/model.gguf

Open the native macOS app:

open app/Juno.xcodeproj

What’s next

The next technical priority is improving throughput scaling beyond two threads while preserving Juno's energy advantage. Further work includes supporting additional low-bit formats and caching validated compiled layouts across model loads.

The M4 Pro is the first verified target. The runtime detects SME2 on macOS and Linux, but other SME2 devices still need their own correctness, performance, and energy measurements. Future work includes validating Juno on more Arm laptops and phones and packaging the runtime for direct integration into native applications.

Built With

Share this project:

Updates

Submission history