PrivLens

On-device PII redaction for Arm: a from-scratch int8 transformer in C with five hand-written AArch64 kernel tiers and runtime dispatch that never picks an instruction your silicon can't execute.

Comment

Inspiration

PII redaction is the one case where "on-device" isn't a nice-to-have. The whole value is that the sensitive text never travels — and a cloud redaction API has to receive the unredacted document in order to redact it, which defeats the purpose.

That constraint sets everything else. The model has to be small enough to sit in a phone's cache, quantised enough to run on a CPU with no GPU or NPU, and fast enough to feel instant while you type. That's an optimisation problem, and it's an Arm optimisation problem.

What it does

Paste text, get it back with names, emails, phone numbers, addresses, card numbers and IDs stripped out. No network calls, no cloud, no telemetry, no API key.

$ privlens redact leak.txt
Hi, my name is [PERSON] and you can reach me at [EMAIL] or on [PHONE].
My account [ACCOUNT_NUM] was charged twice on [DATE_OF_BIRTH].
Reference 2019-CX-4 and build 3.2.1 are not personal data.

-- 5 span(s) redacted in 3.1 ms (i8mm kernel, 71.4% of time in GEMM)

Underneath is a from-scratch int8 transformer inference engine in C — no dependencies beyond C11 and a libc — with five hand-written AArch64 kernel tiers and runtime dispatch that picks the fastest one the silicon can actually execute, and never selects one it cannot.

How I built it

A byte-level transformer encoder, 1.47M parameters, tagging every input byte with one of 12 classes; contiguous runs become a redactable span. Byte-level is a modelling choice, not a shortcut: PII is overwhelmingly character-patterned — the @ and dot structure of an email, the digit grouping of a card number, the NN Streetname St shape of an address. It also means the C engine needs no tokenizer, no vocabulary file and no Unicode tables. The whole on-device artefact is one binary plus one 1.5 MB blob.

Quantisation is W8A8 — int8 weights, symmetric per output channel, baked at export; int8 activations, symmetric per row, computed each forward pass. The inner GEMM is a dot product over contiguous int8, which is exactly the shape Arm's SDOT and SMMLA instructions exist to accelerate.

Each tier is its own translation unit compiled with its own -march, so one portable binary contains every path and chooses at run time — the same structure llama.cpp uses for GGML_CPU_ALL_VARIANTS. All five produce bit-identical output; integer GEMM has no reassociation error, so "close enough" isn't the bar.

The optimisation that mattered

The first SMMLA kernel lost to SDOT. That wasn't the expected result.

SMMLA needs its two operands as interleaved row pairs inside one 16-byte register. Building that with vcombine_s8 costs two 8-byte loads plus a combine per operand, every iteration — and at that ratio the extra arithmetic SMMLA gives you is entirely consumed by data movement.

The fix was to store the weights already interleaved, once, at model load. The inner loop collapses to ldr / ldr / ldr / smmla / smmla. Measured on an identical GEMM (M=8, N=32, K=192), counting dynamic AArch64 instructions actually retired:

Tier Instructions int8 MACs/instr vs scalar
scalar 37,998 1.29 1.00x
dotprod 11,414 4.31 3.33x
i8mm, naive 16,961 2.90 2.24x
i8mm, packed 8,233 5.97 4.62x

Packing is a 2.06x improvement over the naive SMMLA kernel, and it takes it from losing to SDOT to beating it by 1.39x. This is the same insight behind llama.cpp's CPU_REPACK buffer type — arriving at it independently, from a measurement that contradicted my expectation, is the most useful thing this project did.

Where it got interesting: instructions are not cycles

Two independent bodies of evidence, and they disagree. Modelled with llvm-mca against LLVM's real per-core scheduling models, packed SMMLA retires 28% fewer instructions than SDOT — and on Neoverse-V1 it is still slower in cycles, while on Neoverse-V2 it wins by 1.6x.

SMMLA's throughput relative to SDOT is a microarchitectural property, and no compiled-in rule gets it right on every core. So PrivLens doesn't guess: privlens tune runs the real workload through every kernel the CPU can execute, on the device that will run it, and caches the winner under a hardware signature. Copy that file to a different machine and the signature won't match, so it's ignored — and the loader re-verifies executability against the live CPU before honouring it, because a config file must never be able to talk the engine into an illegal instruction.

Never SIGILL: the CPU probe

src/arm_features.c is standalone and dependency-free, and it's the piece most likely to be useful on its own. AArch64 feature detection is harder than it looks, and getting it wrong doesn't degrade gracefully — it traps.

  • Naming. Linux /proc/cpuinfo calls FEAT_DotProd asimddp. Arm docs, GCC -march modifiers and Apple's sysctl all call it dotprod. Scripts that grep for dotprod silently fall back to baseline Armv8 on hardware that has it — this is Stockfish #4931, observed on a Raspberry Pi 5.
  • Android under-reports. Many shipped kernels fail to advertise HWCAP_ASIMDDP on cores that implement it, so PrivLens falls back to a MIDR allowlist, mirroring pytorch/cpuinfo. Additive only — never used to disable something HWCAP reported.
  • SME is not SME2. llama.cpp #22182 selects SME2 kernels on a base-SME Snapdragon X2 and dies with STATUS_ILLEGAL_INSTRUCTION before emitting a log line.
  • Vector length isn't in HWCAP. prctl(PR_SVE_GET_VL) is the only way to learn it, and it's decisive.

The contract: pl_cpu_detect() never executes a feature-gated instruction in order to test for that feature. Everything comes from the OS — auxv, sysctl, prctl, Win32.

Verified across seven Arm CPUs

The same statically-linked binary, run under qemu-aarch64 against seven emulated cores, asserting bit-identical GEMM output on every one:

cortex-a72   Raspberry Pi 4                selects neon
cortex-a76   Raspberry Pi 5                selects dotprod
neoverse-n1  AWS Graviton2 / Ampere Altra  selects dotprod
neoverse-v1  AWS Graviton3                 selects sve
neoverse-n2  Azure Cobalt 100              selects i8mm
cortex-a710  Snapdragon 8 Gen 1/2          selects i8mm
max          QEMU max (all features)       selects sve

Look at Graviton3 and Cobalt 100. Identical i8mm and sve feature bits. The only difference is the SVE vector: 256-bit versus 128-bit. On a 128-bit implementation, SVE SDOT consumes exactly the same 16 bytes per instruction as NEON SDOT while paying for predicate setup — so it's worse, and packed SMMLA is the right choice. The feature bit tells you nothing; the vector length decides. Selecting on HWCAP2_SVE2 alone, which plenty of code does, gets this wrong.

That prediction then held on real hardware. CI's native-Arm runner is a Neoverse N2 (MIDR 0x410fd490) reporting SVE VL = 16 bytes (128-bit), and it selected i8mm — exactly what the emulated matrix said it should.

And it's worse on phones than on servers: Qualcomm's Oryon cores (Snapdragon 8 Elite, X Elite) dropped SVE entirely. LLVM's oryon-1 target lists FeatureDotProd and FeatureMatMulInt8 and conspicuously no FeatureSVE. An inference path gated on SVE2 falls off a cliff on Qualcomm's current flagship.

The mistake that shaped the training data

tools/gen_data.py synthesises the entire corpus — no download, no Hugging Face, no pretrained checkpoint.

The first version emitted single sentences. The model trained on it scored 0.947 span-F1 on held-out data and then fell over on a realistic email: it labelled the subject line as PERSON, and so did it label the word "Thanks". The architecture was fine. The data was the problem — every capitalised token the model had ever seen at the start of a line was a name.

So the corpus now generates whole documents: emails with subject lines, greetings and sign-offs; chat threads; form dumps; server logs. It deliberately supplies capitalised non-entities as hard negatives — weekdays, months, "Thanks", "Subject", "Approved", version strings, ticket and order references. Roughly one document in ten contains no PII at all, so the model is rewarded for staying quiet.

For a redactor, "capitalised means PERSON" is the single most damaging error available: it destroys ordinary prose while giving a false sense of safety.

Model quality

Trained for 4 epochs on 90,000 synthetic documents, evaluated on 6,000 held-out ones. tools/validate.py then runs the shipped C binary over 150 held-out documents and scores it against the same gold labels:

precision recall span-F1
PyTorch float32, raw argmax 0.904 0.967 0.934
C engine, int8 + span cleanup 0.933 0.925 0.929

117 of 150 documents produce a span set byte-identical to the float reference. int8 costs 0.005 span-F1, and the split is informative: the engine trades a little recall for three points of precision, which is the cleanup suppressing exactly the "capitalised word tagged as PERSON" errors that wreck real prose.

Challenges I ran into

I had no Arm hardware. Everything was cross-compiled and verified under qemu-aarch64, which meant being disciplined about what emulation can and cannot tell you. QEMU executes every guest instruction, so tracing and differencing gives an exact retired-instruction count — but wall-clock under emulation is meaningless. SVE even appears to be the slowest tier under QEMU, which is translation cost, not silicon. So the repo labels every number by how it was established: executed, traced, or modelled. No blending.

SME defeated me, and I shipped that fact instead of a broken kernel. I prototyped an SME kernel far enough to execute under QEMU with the right tile geometry — which needed clang 18 plus a hand-written AAPCS64 SME ABI shim, since GCC 13 has no SME support at all. Its results were wrong, and the SMOPA operand-to-tile mapping didn't probe consistently. Without real SME silicon there's no way to tell whether I'm misreading the ACLE or the emulator differs from hardware. So it isn't shipped, and docs/OPTIMISATION.md §7 documents the attempt.

Ragged span edges. A byte tagger will label Priy of Priya, or split a card number across its spaces. Emitting Pr[PERSON]a both leaks two letters of the name and mangles the sentence. Three deterministic passes fix it — snap edges outward to word boundaries (capped, so one mislabelled byte can't drag in a whole word), merge neighbours across short non-alphabetic gaps, then trim sentence punctuation back off. Because all three operate on argmax labels only, every kernel tier still produces identical output.

What I learned

That a measurement contradicting your expectation is the most valuable thing you can get, provided you chase it instead of explaining it away. The naive SMMLA kernel losing to SDOT is the reason this project has a packing pass at all.

And that "compiled in" and "executable here" are genuinely independent properties. Almost every ecosystem bug I read while building this — the Stockfish naming trap, the llama.cpp SME2 crash — comes from collapsing those two into one.

What's next

Threading, measured properly rather than guessed — it interacts with big.LITTLE scheduling in ways that deserve their own experiment. An SME kernel, if I can get access to real SME silicon. And src/arm_features.{h,c} is ~400 lines with no dependencies and is the piece I'd most like other people to just take.

Built With

Share this project:

Updates

Submission history