optimus-cloud-ai

Arm Cloud AI inference API with int8 optimization, batching, and benchmark proof for CPU-only cloud servers.

Comment

Inspiration

Inspiration

Cloud AI is often associated with GPU-heavy deployments, but many useful AI features can run efficiently on CPU-only cloud infrastructure. Optimus-cloud-ai was inspired by the Arm Create: AI Optimization Challenge goal of showing how AI inference can be optimized for Arm-based cloud servers such as AWS Graviton, Microsoft Cobalt, GCP Axion, Ampere, or Arm64 on-prem servers.

What it does

Optimus-cloud-ai is a Track 2 Cloud AI project that provides a CPU-only inference API for batched operational requests. Each request is processed by an inference endpoint and returns a route, priority, and short summary.

The project compares a baseline fp32 inference path with an optimized int8 quantized path. It reports model size, latency, throughput, accuracy, CPU architecture, Python version, and NumPy version. The benchmark output is saved as a JSON artifact that can be rerun on Arm cloud infrastructure.

How we built it

We built Optimus-cloud-ai as a standalone Python project using NumPy and the Python standard library HTTP server. The model uses deterministic hashed text features and a compact route classifier trained with ridge regression.

The optimized inference path quantizes fp32 model weights into int8, quantizes request vectors, performs integer dot-product inference, and dequantizes only the final route scores. We also added a CLI, HTTP API endpoint, batch benchmark runner, environment reporter, and unit tests.

Challenges we ran into

The main challenge was designing a Cloud AI workload that is simple enough to run anywhere but still realistic enough to represent API-scale inference. We wanted the project to show cloud behavior such as batching, endpoint-style inference, throughput measurement, and CPU-only deployment.

Another challenge was preserving prediction quality after quantization. The int8 path needed to reduce model size while still matching the fp32 baseline on the sample evaluation set.

Accomplishments that we're proud of

We are proud that Optimus-cloud-ai creates a clear Cloud AI optimization story: API inference, batching, fp32 baseline, int8 optimization, benchmark output, and reproducible validation.

The current benchmark shows a 73.71% model size reduction while maintaining accuracy parity between fp32 and int8 inference. We are also proud that the project is open source, standalone, and easy to run on local machines or Arm cloud instances.

What we learned

We learned that Cloud AI optimization is not only about using larger models. It is also about making inference efficient, repeatable, and scalable on cost-effective CPU infrastructure.

We also learned that int8 quantization is a practical starting point for Arm cloud AI because it reduces memory use and can improve deployment efficiency for CPU-only inference endpoints.

What's next for optimus-cloud-ai

Next, we want to run the benchmark on real Arm cloud instances such as AWS Graviton, Microsoft Cobalt, GCP Axion, or Ampere servers and attach Arm-specific results to the submission.

Future versions could add ONNX Runtime, llama.cpp or vLLM-based inference, load testing, autoscaling examples, a small frontend, and deployment templates for Kubernetes or serverless Arm cloud environments.

Built With

  • ai
  • api
  • arm64
  • aws-graviton
  • azure-cobalt
  • benchmarking
  • cloud-ai
  • cpu-inference
  • gcp-axion
  • http
  • int8
  • machine-learning
  • model-optimization
  • numpy
  • python
  • quantization
Share this project:

Updates

Submission history