EdgeTuner

Dashboard home page

Inspiration Most quantization work benchmarks on paper numbers — FLOPs, param count — and calls it a day. But a smaller model isn't automatically a faster one on ARM, the instruction sets and kernels are just too different from x86. Optuna already finds you the best training setup; nobody was doing the same thing for deployment. We wanted a system that actually measures every candidate on a real Raspberry Pi instead of guessing.

What it does You hand it a trained model, it exports to ONNX, and it searches quantization settings and runtime configs — automatically, with the accuracy floor enforced as a hard rule. Every candidate that survives gets benchmarked for real on a Raspberry Pi 5: latency, memory, size, accuracy. Instead of spitting out one "best" config, it gives you a shortlist where each one wins on something — fastest, smallest, lowest memory, most accurate — so you pick based on what you actually need.

How we built it Two stages. First we train three CIFAR-100 models (ResNet-18, MobileNetV2, a custom CNN) and export to ONNX. Then the real part: we measure how much each layer group actually costs in accuracy when quantized, price that against how much latency it costs to keep it FP32, and use that to shrink the search space before we ever touch the device. Whatever's left gets walked through one setting at a time — per-channel vs per-tensor, activation type, graph optimizations, memory arena, calibration — and every single candidate gets benchmarked for real on the Pi.

Challenges we ran into The nastiest one: our dev laptop's CPU doesn't have the instruction set ONNX Runtime wants for INT8, so it was quietly corrupting our accuracy numbers and made per-channel quantization look worse than per-tensor, which shouldn't even be possible. Took a while to realize the bug was in our test machine, not our method. We also had a dumb but costly bug where re-running the pipeline for one model would silently wipe out the saved results for the other two — hit that twice before finally catching it for good. And at one point we predicted ahead of time which layers would be most sensitive to quantization, and the actual Pi measurements just... disagreed. We left that in as a real result instead of quietly fixing the prediction to match.

Accomplishments that we're proud of Getting real, measured speedups on actual hardware — not estimates, not simulator numbers. All three models ended up multiple times faster with barely any accuracy lost, and every number on the dashboard traces back to a real device run. We're also proud we caught the saturation bug instead of shipping wrong conclusions, and that we reported the failed sensitivity prediction honestly instead of burying it.

What we learned Mostly that you can't trust anything you haven't measured on the real target — even stuff that feels obvious on paper can flip once you're on different silicon. Also that the boring infrastructure work (sealed test sets, checking measurements are stable, catching silent bugs) ended up mattering just as much as the actual search algorithm.

What's next for EdgeTuner Structured pruning is the obvious next lever — we deliberately left it out because it needs a real graph rebuild and fine-tuning, not just zeroing weights. We'd also like to try it on other datasets and other ARM boards, not just CIFAR-100 on a Pi 5, and push the search to actually reach the calibration round instead of stopping at the structural ones by default.