Challenge track: Mobile AI — Kleidoscope targets on-device inference on Arm consumer silicon: measured end-to-end on Apple M4 (SME2), with ISA fingerprinting verified against a real Android phone over adb.
Inspiration
KleidiAI's promise is beautiful: no code changes required. Build llama.cpp, run it on an Arm CPU with SME2, and the optimized microkernels get dispatched automatically.
That promise is also the problem. Because nothing changes in your code, nothing tells you whether it happened. We went looking for the officially documented way to verify it and found this: grep your startup log for the string CPU_KLEIDIAI, and flip an environment variable to see if the number moves.
That is not a measurement. That is a rumor with a log line.
So the question we set out to answer wasn't "how do I make inference faster on Arm." It was the question you have to answer before that one is meaningful: did the optimization I think I'm running actually run — and what did it cost?
What it does
Kleidoscope is a measurement instrument for on-device AI on Arm. For any inference workload it answers three questions:
1. What can this silicon actually do?
It reads the real ISA capabilities — SME, SME2, SVL, I8MM, DotProd, BF16, FP16 — from the platform itself: sysctl hw.optional.arm.* on macOS, getauxval(AT_HWCAP/AT_HWCAP2) via ctypes plus /proc/cpuinfo on Arm64 Linux, and adb shell for Android.
2. Which path did the workload actually take?
Three independent lines of evidence, never just one: a symbol/string scan of the binary and every library it links (this matters — on Apple Silicon the KleidiAI markers live in libggml-cpu.dylib, not in llama-cli itself), the runtime log signature, and the plausibility of the A/B difference. When the three disagree, the report says so instead of smoothing it over.
3. What was it worth — in time and in energy?
Interleaved A/B runs with warmup discard, reported as median plus min–max range, never as a single point estimate. Energy via powermetrics on macOS and battery sysfs integration over adb on Android, expressed as joules per token.
The output is kleidoscope-report.md and .json — deterministic field order, so two runs on the same device produce a diff that shows only the measurements. Optimization becomes reviewable, like a test.
The verdict line is the product:
## Verdikt
**verfuegbar, aber ungenutzt**
**Behebung:** llama.cpp mit `-DGGML_CPU_KLEIDIAI=ON` neu bauen (cmake) oder eine
Distribution installieren, die KleidiAI-Microkernel einkompiliert.
A machine that has the capability. A binary that never touches it. And the one line that fixes it.
How we built it
Python 3.9+, standard library only. No pip install, no build step, no toolchain.
That constraint was a product decision, not a technical one. The distance between git clone and a juror's first report is the entire developer experience. ctypes reaches getauxval, subprocess drives sysctl/powermetrics/adb, and string templates render the report. Nothing has to be compiled on your machine before Kleidoscope tells you something true.
Architecture: probe/ (per-platform fingerprint), runner/ (env matrix, interleaved timing), energy/ (per-platform sampler behind one interface with an explicit available: bool), verdict/ (three-evidence logic), report/ (Markdown + JSON). Each platform is a swappable module behind a shared interface.
One rule governed everything: measure, never claim. Every number in the output comes from a run on real hardware, or the field reads "not available" with the reason. There is no estimated value anywhere in a Kleidoscope report — not a placeholder, not an interpolation, not a rounded guess.
Challenges we ran into
Four of them were the actual work of this hackathon, and all four were measurement bugs rather than feature bugs.
Our A/B design was confounded by run order. The first version ran all iterations of variant A, then all of variant B. That silently entangles "which variant" with "when it was measured" — thermal state, scheduler behavior, and background load drift across a run. It produced a phantom ~10% decode difference that we nearly reported as a finding. Interleaving the arms and randomizing round order made it vanish. It was never real.
powermetrics -n 0 silently loses every sample. The obvious way to sample energy is to start powermetrics in run-until-killed mode and kill it when the work finishes. It buffers internally and only flushes on a clean self-initiated exit, so a killed sampler yields a 97-byte header and nothing else. We tested SIGTERM and SIGINT, sent to the sudo wrapper and to the powermetrics child directly — all four produced byte-identical empty output. The fix was finite, self-terminating chunks on a background thread, each one exiting and flushing on its own. The sampling gap that introduces is reported in the report's sanity row rather than absorbed into the numbers.
Our first "KleidiAI vs stock" comparison compared the wrong thing. We benchmarked Homebrew's llama-cli against our own CMake build with -DGGML_CPU_KLEIDIAI=ON and measured a clean 11.5% decode improvement. Then we checked the versions: different llama.cpp revisions, different compiler patch levels. The KleidiAI flag was one of at least three variables. We rebuilt the same source tree twice, changing only the flag, and re-ran. That is the only comparison in this repo we consider valid.
Our own report contradicted its own table. The verdict text claimed "ranges overlap" for every below-threshold result — including cases where the ranges were genuinely separate but the delta sat under our 8% significance threshold. Two sentences below a table that disproved them. For a tool whose only product is credibility, that is the most expensive kind of bug, and it survived a full review round before anyone read the sentence against the numbers.
Accomplishments that we're proud of
We built something that told us an inconvenient truth and we published it anyway.
On our Apple M4 — FEAT_SME2=1, SVL 64 bytes — we compiled llama.cpp with -DGGML_CPU_KLEIDIAI=ON and compared it against the identical source tree built with the flag off:
| KleidiAI OFF | KleidiAI ON | |
|---|---|---|
| Decode ms/token | 49.987 (48.644–50.396, n=3) | 52.649 (52.096–52.865, n=3) |
| Joules/token | 0.454 (0.440–0.480, n=3) | 0.538 (0.511–0.564, n=3) |
Compiling the path in did not make this workload faster on this machine. It made it cost more energy.
We want to be precise about what that is and isn't. This is not a verdict on KleidiAI. It is one integration — llama.cpp's KleidiAI path — on one device class, with one model, measured three times with interleaved runs and honest ranges. Apple Silicon's SME implementation and llama.cpp's dispatch conditions are a specific combination, and a result there says nothing about Arm Linux, about Android, or about KleidiAI used directly.
What it does show is exactly why the tool needs to exist. Before Kleidoscope, a developer in this situation would have shipped the KleidiAI build, assumed a win, and never known. The honest answer to "should I enable this?" is measure it on your target — and now there is something that does.
We are also proud of what the reports refuse to say. Where energy could not be sampled, the field says "not available (no sudo)" instead of a plausible number. Where only one run exists, the spread column says "not computable" instead of implying precision. Where the evidence disagrees, the contradiction is printed.
What we learned
That the hard part of a measurement tool is not measuring. It is resisting the pull toward the tidier number.
Every one of our four big bugs made the output look better than the truth: a phantom speedup from run order, a real-looking speedup from a version mismatch, a confident sentence about overlapping ranges that weren't. None of them would have been caught by a test suite, because the code did exactly what it was written to do. They were caught by reading the report against its own table.
We also learned how much of Arm's optimization story is currently unverifiable in practice, and how little tooling stands between a developer and a wrong assumption.
What's next for Kleidoscope
On-device inference measurement on Android. The ISA fingerprint works over adb today — we verified it against a real midrange phone during the hackathon. Running the actual workload on-device, with battery-sysfs energy integration, needs an arm64 build of the runtime pushed to the device, which didn't fit in the remaining window. The energy sampler for it is already written.
A Neoverse rung. Ampere Altra / Neoverse N1 on a cloud instance, to extend the ladder from phone to laptop to server.
More runtimes. ExecuTorch and ONNX Runtime presets — the dispatch question is identical, only the log signatures and env switches differ.
Reports in CI. The report is already deterministic and diffable. The natural next step is a check that fails a pull request when a device's optimization path silently regresses.
Try it out
git clone https://github.com/fiya-chris-and-AI/kleidoscope
cd kleidoscope
python3 -m kleidoscope probe # ISA fingerprint, <1 second
python3 -m kleidoscope verdict -- llama-cli # fingerprint + binary scan -> verdict, no model needed
No dependencies. No build step. Python 3.9 standard library only. Apache-2.0.
Log in or sign up for Devpost to join the conversation.