ArmForge AI Studio — The VS Code for Arm AI Optimization

Inspiration

Edge and Cloud AI workloads on Arm architecture often face memory bandwidth bottlenecks and CPU cache line eviction. Existing tools focus on single-model inference rather than an end-to-end developer optimization workflow. We set out to build "The VS Code for Arm AI Optimization" — an internal-grade platform that diagnoses model layers, auto-generates C++ Arm NEON SIMD micro-kernels, profiles execution timelines, and exports official Arm Performix reports.

What it does

ArmForge AI Studio is a multi-modal AI optimization platform and developer IDE for Arm64 architectures:

• AI Optimization Advisor: Graph scanner that analyzes model layers (embed_tokens, q_proj, k_proj, layernorm, gate_proj, lm_head), forecasting 1.9x speedups and -52.7% memory reduction.
• Auto SIMD Kernel Generator: Synthesizes and JIT-compiles C++ Arm NEON micro-kernels with vdotq_s32 vector dot-product acceleration for MatMul, GEMM, Conv2D, LayerNorm, and Softmax.
• Replay Timeline Profiler: Chrome DevTools-style microsecond latency profiler (0.73ms MatMul ➔ 3.67ms KV Cache ➔ 5.14ms SwiGLU ➔ 6.37ms Softmax).
• Vision Canvas Demo: Real-time YOLOv8 object detection processing live canvas frames at 23.8 FPS with Arm SIMD acceleration vs 11.2 FPS baseline.
• One-Click Exporters: Generates official arm_performix_report.json and standalone C++ header library armforge.h.

How we built it

• Core Engine: Python 3.9 + C++ micro-kernels compiled via clang/gcc targeting Arm NEON, Dot-Product (SDOT), and SME intrinsics.
• Backend Server: FastAPI + Uvicorn + WebSockets for real-time telemetry streaming.
• Frontend Studio UI: Glassmorphic dark-mode web application (HTML5, CSS3 variables, Chart.js, Lucide Icons).

Challenges we ran into

• Aligning grouped INT4 quantization nibbles and scales to 64-byte Arm CPU L1 cache line boundaries without runtime copy overhead.
• JIT-compiling C++ SIMD extensions dynamically across macOS Apple Silicon (arm64) and Linux aarch64 environments.

Accomplishments that we're proud of

• Achieving 1.97x throughput speedup (29.72 tok/s vs 15.06 tok/s) and 85.9% RAM footprint reduction on Arm64 hardware.
• Creating a zero-dependency standalone header armforge.h for instant drop-in deployment in C++, Rust, Android, and iOS projects.

What's next for ArmForge AI Studio

• Expanding SVE2 / SME2 matrix multiplication kernels for scalable Arm Neoverse V-series cloud instances.
• Adding direct execution provider hooks for ONNX Runtime and ExecuTorch.

Built With

• arm-performix
• arm64
• c++
• chartjs
• css3
• fastapi
• html5
• int4
• int8
• javascript
• jit-compiler
• kleidiai
• low-latency
• neon
• onnx
• python
• pytorch
• quantization
• simd
• uvicorn
• yolo