VSCODE DEVELOPING ENVIRONMENT
FILES
SOURCE FILE
OPTIMIZATION AND MODEL
DASHBOARD
PDF(or)HTML REPORT GENERATION
OPTIMIZATED REPORT
GITHUB LAUNCH FEATURE
GITHUB PUSH COMPLETED

Inspiration

Due to the appearance of new AI models, there is constantly a demand for efficient deployment of models to ARM based edge devices. In current situation, optimization of such deployment is manually done and includes numerous tests of quantization, CPU threads number, run-time settings and graph optimizations. One question motivated us in development: what if it will be possible for AI to optimize itself, learn from the results and give automatic recommendations on the best configuration? ARM Developer AutoPilot became the answer to that question and was created in order to automate, measure and optimize the process for developers.

What it does

ARM Developer AutoPilot is a platform, which is automated, benchmark driven, and designed to optimize the process of deployment of AI models on ARM. AutoPilot analyzes your AI project, detects AI models in it, generates several candidates for AI optimization, benchmarks those candidates, ranks configurations and gives recommendations on which configuration will suit you best. The system checks different configurations and optimizations such as quantization, CPU threads tuning, graph optimization and run-time configurations. Instead of making assumptions about best configuration, the system calculates such important metrics as latency, memory consumption and model size and gives recommendations on optimization of AI project based on this information. At the end, AutoPilot generates reports in HTML and PDF formats and is able to deploy optimized project with documentation on GitHub. Workflow of ARM Developer AutoPilot is Analyze -> Optimize -> Benchmark -> Rank -> Report -> Deploy.

How we built it

ARM Developer AutoPilot is an optimized pipeline with ONNX and ONNX Runtime utilized for optimization and benchmarking and Streamlit utilized as dashboard for interaction with developers. The system consists of the following elements: project analysis, AI model detection, generation of optimization candidates, benchmarking, ranking of candidates and recommendation of the best candidate. The engine of ARM Developer AutoPilot does not use one hard-coded strategy for optimization but rather generates several combinations of quantization, CPU threads, graph optimization and run-time settings and benchmarks their performance. All of these results are shown to the developer in the dashboard, reported and deployed to GitHub.

Problems we encountered

One of the main issues that we had to deal with was an awareness of the fact that optimization for AI is not a “one-solution-for-all” type of job, meaning that adding CPU threads wouldn’t always result in low latency due to some scheduling and synchronization overhead. Similarly, while using INT8 quantization can lead to the reduction in model size and memory consumption, it doesn’t guarantee faster performance. That’s why we have decided to create AutoPilot based on the benchmarking process, instead of making any assumptions. Another issue we faced was the integration of all aspects of the project into GitHub.

Accomplishments that we're proud of

We are proud of the fact that ARM Developer AutoPilot made the process of optimization automation of the search and selection of the most optimal configuration among others. The system is able to generate multiple configurations, compare them with the real workload and choose the most optimal configuration by benchmarking. We managed to unify FP32/INT8 optimization, CPU thread tuning, graph optimization, benchmarking, candidate ranking, interactive visualization, reporting, and GitHub deployment into the single solution. Moreover, the system doesn't declare any optimization as being better than another, but provides the reasoning of choice with benchmark results. We think that this feature makes our solution practical and valuable for developers who need to have reliable optimizations and don't spend hours on checking configurations manually.

What we learned

First of all, we have learned that hardware optimization has to be empirical. It means that there can't be one universal configuration which will be always the best for any particular workload. Optimization includes trade-offs between latency, memory usage, model size, and other resources consumption. Secondly, we have understood that benchmarking is not only the final step of the optimization pipeline but it may be used in the middle of it for decision-making. Finally, developer experience is as important as technical capabilities. A powerful engine for optimization becomes even more valuable when its results are clear, understandable, and easily deployable.

What's next for ARM DEVELOPER AUTOPILOT

Our next step is to enhance the hardware awareness of the ARM Developer AutoPilot and make it smarter. We are going to implement automatic detection of the target platform and adaptation of the optimization search process to the platform's characteristics. We would also like to go further from the static candidates search to the intelligent optimization approaches that will learn from previous benchmark results and reduce the number of experiments. Next versions of ARM Developer AutoPilot will support more ARM edge devices and AI models' formats, implement accuracy-aware quantization to measure performance gain, and integration into CI/CD pipelines to automate the process. Our vision of the future of ARM Developer AutoPilot is the full optimization layer where developers could upload their AI models and then the system could understand the workload, explore the optimization space, benchmark the possibilities, explain the trade-offs, and recommend the way to ARM deployment.