LiDAR 3D Detection: int8 QAT Optimization for Arm

Inspiration

A self-driving perception model does not run in the cloud. When a car is moving, there is no time to send LiDAR data to a server and wait for an answer. The model has to run on a low-power chip inside the vehicle, and that chip is almost always Arm.

I had already built a LiDAR 3D object detector from scratch, and it ran in real time on my laptop GPU but crawled to about 1 frame per second on a CPU. That gap is the whole problem with edge perception in one number. It made me want to take the model all the way to real Arm hardware and prove, with measurements, that it could be made fast enough to matter. The constraint was real from the start: I trained the whole model under a 4GB VRAM limit, so a lean, deployable design was baked in from the beginning.

What it does

It detects vehicles and pedestrians in 3D from raw LiDAR point clouds. Points come in, get turned into a bird's-eye-view image, run through a convolutional network, and come out as 3D bounding boxes placed in real-world meters.

The project then takes that detector through a full optimization pipeline and deploys it as an int8 model on an Arm CPU (Ampere Altra), with measured before-and-after numbers:

• Model size: 26.47 MB down to 6.67 MB (3.97x smaller)
• Arm latency: 2703 ms/frame down to 703 ms/frame on one core (about 3.8x faster)
• Accuracy: mAP 0.4656 to 0.4307 (92.5% retained)

How I built it

The detector is SFANetV2, a 6.9 million parameter encoder-decoder convolutional network I wrote from scratch, trained on 170 nuScenes scenes. The bird's-eye-view encoding, the training loop, the decoder, and the nuScenes-style evaluation were all built by hand, not pulled from an existing repo.

The optimization pipeline had six stages. First, lock the fp32 baseline (size, speed, mAP). Second, export to ONNX and verify it matches the original. Third, quantize to int8. Fourth, get real Arm hardware, a Google Cloud Ampere Altra instance. Fifth, benchmark fp32 against int8 on that Arm CPU with ONNX Runtime. Sixth, package it.

The interesting part was stage three, which did not go as planned, and stage four, quantization-aware training, which is what actually made it work.

Challenges I ran into

Standard post-training quantization broke the model. Converting the trained fp32 model to int8 dropped mAP from 0.4656 to about 0.30. I tried three different methods (dynamic, static per-tensor, static per-channel) and they all landed at the same place, around 0.30. That was a catastrophic drop, not the small rounding loss quantization is supposed to cause.

The fact that three methods converged on the same number was the clue. I checked that the fp32 ONNX model still scored a perfect 0.4656, which proved the loss came purely from quantization and not from a broken pipeline. I ran a decode-threshold sweep from 0.03 to 0.15, in case the model was just producing lower confidence scores that were getting filtered out. The mAP stayed flat at 0.30 across every threshold, which proved the detections were genuinely gone.

The conclusion was that this is a hard floor for post-training quantization on this kind of architecture. Anchor-free detectors with precise coordinate-regression heads depend on numerical precision that int8 rounding destroys, and no post-training setting recovers it. I needed a different approach.

Accomplishments that I'm proud of

I broke the quantization floor with quantization-aware training. Instead of quantizing a finished model, QAT inserts fake int8 rounding into the model and continues training, so the model learns weights that survive the rounding. It recovered mAP from the 0.30 floor back up to 0.4307, which is 92.5% of the original accuracy, at int8.

I also deployed it on real Arm silicon and measured a genuine 3.8x speedup of int8 over fp32, on top of the 3.97x size reduction. Every number in this project comes from the actual model, measured, not estimated.

Just as important, I ruled out several further optimizations with evidence instead of guessing. Structured pruning was killed by an L1-norm analysis that showed zero prunable channels (the model is dense, consistent with it being data-limited rather than over-parameterized). Multi-core and parallel execution were measured and found to be slower than single-core for this small model. The XNNPACK backend was checked and ruled out on cost. Knowing when to stop is part of the work.

What I learned

Two lessons stand out. First, when three quantization methods fail identically, the problem is not the method, it is the architecture, and the fix is a different class of technique entirely. Diagnosing that by isolating variables, rather than trying more settings, is what got me unstuck.

Second, and this one surprised me: when I extended QAT past its sweet spot (from 9 epochs to 20), the training loss kept falling but the actual mAP collapsed from 0.43 to 0.29. The loss and the real detection metric had diverged. If I had trusted the loss curve and shipped the final epoch, I would have shipped a much worse model that looked like it was still improving. Choosing the best checkpoint by measured mAP, not by loss or by taking the last epoch, was essential.

What's next

Scaling to the full nuScenes dataset (700 scenes) should push the base accuracy higher, since the model is data-limited. Beyond that, a lighter backbone using depthwise-separable convolutions, or running at a lower input resolution, could push the speed further, though both would mean retraining from scratch. And the model is ready to benchmark on dedicated edge hardware like a Jetson or an in-vehicle SoC, where the multi-core and accelerator support would take it from the current single-core numbers toward real-time.

Built With

• ampere-altra
• arm
• computer-vision
• google-cloud
• lidar
• nuscenes
• onnx
• onnxruntime
• python
• pytorch
• qat
• quantization