ARM YOLOv8 Edge Optimization

Performance comparison between baseline FP32 and optimized INT8
Baseline YOLOv8 FP32 benchmark on ARM
Optimized YOLOv8 Static INT8 4-thread benchmark

Inspiration

Our main goal was to see how far a small ARM-based edge computer could be pushed for real-time object detection. When we first ran the standard YOLOv8n model on the Raspberry Pi, the performance was only around 3.69 FPS. This made real-time detection difficult.

Instead of changing the hardware, we decided to optimize the software pipeline and investigate whether quantization, runtime configuration, and input-size reduction could provide a meaningful improvement.

What We Built

We built a benchmarking pipeline to compare the original YOLOv8n model with an optimized INT8 version on an ARM-based Raspberry Pi.

The baseline model was executed in FP32. We then created a static INT8 ONNX model and used ONNX Runtime with a four-thread CPU configuration. The optimized pipeline uses a 512×512 input size.

Both versions were tested using the same 1,729-frame video so that the performance comparison would be consistent.

How It Works

The workflow starts by running the original YOLOv8n model and recording its performance on the Raspberry Pi.

We then:

converted the model to ONNX format,
applied static INT8 quantization,
configured ONNX Runtime for CPU inference,
used four intra-operation threads,
reduced the input resolution from 640×640 to 512×512,
and measured the optimized model using the same video.

During benchmarking, we recorded FPS, inference latency, end-to-end latency, model size, RAM usage, CPU utilization, and temperature.

Results

The optimization produced a noticeable improvement in CPU inference performance.

Metric	Original FP32	Optimized INT8
FPS	3.69	16.64
Inference latency	257.20 ms	55.32 ms
End-to-end latency	260.69 ms	59.75 ms
Model size	6.25 MB	3.41 MB
Input size	640×640	512×512

The optimized configuration increased the measured frame rate from 3.69 FPS to 16.64 FPS, which is approximately a 4.5× improvement.

Inference latency also dropped from 257.20 ms to 55.32 ms, while the model size decreased from 6.25 MB to 3.41 MB.

Challenges

The main challenge was getting useful inference speed from the Raspberry Pi without using a GPU. Different optimization choices affect both speed and resource usage, so we had to test the configuration on the actual ARM hardware.

Another challenge was keeping the comparison fair. We used the same video workload and collected measurements from both the baseline and optimized configurations.

What We Learned

This project helped us understand that improving edge AI performance is not only about selecting a smaller model. Runtime settings, numerical precision, input resolution, and CPU utilization can all have a significant effect on the final performance.

We also learned the importance of measuring an optimization on the target hardware instead of assuming that a theoretical improvement will translate directly into real-world speed.

Impact

The result shows that a relatively lightweight ARM device can achieve a much higher YOLOv8 inference rate through software-level optimization.

This approach can be useful for edge applications where running computer vision locally is preferred because of hardware, latency, or resource constraints.