CortexLoop

Inspiration

Arm asked for Physical AI, not another chatbot screenshot. Warehouse robots live or die on control-rate latency, model size, and kernels that actually run on the CPU in the machine. CortexLoop is that loop: a simulated RGB camera, an INT8 CNN on Arm CPU kernels, a planner that turns class labels into pick / haul / dock / avoid, and differential-drive control.

What it does

1. Sense — on-robot camera (simulated 96×64 RGB) → bilinear resize → INT8 CNN.
2. Plan — A* + mission policy (pick, haul, dock, avoid hazards).
3. Act — differential-drive with energy and collision accounting.
4. Prove it — live HUD at http://127.0.0.1:8765 where judges toggle scalar → NEON DotProd → I8MM → KleidiAI and watch microseconds change.

Native library libcortexloop ships the CNN and kernels. Trained artifact models/warehouse_int8.bin is 1.7 KB.

How we built it

• native/src/scalar.c compiled with -fno-vectorize so the baseline is honest.
• native/src/gemm.c — hand-written vdotq_s32 (FEAT_DotProd / SDOT) and vmmlaq_s32 (FEAT_I8MM / SMMLA).
• native/src/kleidiai_backend.cpp — optional Arm KleidiAI INT4 packed GEMM.
• Runtime detection of NEON, DotProd, I8MM, SME / SME2.
• Python simulator, trainer, and dashboard (python -m cortexloop demo).

Measured results (Apple Silicon Arm64)

Same math, four backends. GEMM 128×128×128 INT8.

INT8 warehouse CNN: 1.7 KB, 8/8 on held-out class close-ups. KleidiAI is slower at this tiny GEMM because LHS quant+pack runs every call; the headline for judges is 33× SDOT in native/src/gemm.c.

Challenges we ran into

The first trainer was essentially random (~12%). We rewrote conv backprop so INT8 inference in C matches training. KleidiAI via CMake on an external drive hit AppleDouble ._* files; the Make path and a /tmp CMake tree fixed it. An honest scalar baseline required -fno-vectorize — otherwise clang silently vectorizes and the “speedup” is fake.

Accomplishments that we're proud of

• A closed Physical AI loop, not a kernel microbench in isolation.
• 33× INT8 GEMM vs a non-vectorized scalar baseline, plus 3.4× end-to-end perception.
• Optional KleidiAI INT4 path linked from source.
• One-command train / bench / demo. MIT license. No cloud account required.

What we learned

At warehouse-CNN sizes, DotProd SDOT beats I8MM and KleidiAI packing overhead. Feature detection matters: the same binary picks SDOT, SMMLA, or scalar. Physical AI judging is about the loop and the numbers.

What's next

SME2 outer products on the GEMM, a real CSI camera on Arm Linux, and caching KleidiAI LHS packs so INT4 wins at small M.

Setup (Arm64 macOS or Linux)

Needs clang, make or cmake, Python 3.10+, numpy.

make -j                 # NEON / I8MM (no network)
# or: cmake -S . -B build -DCORTEXLOOP_WITH_KLEIDIAI=ON && cmake --build build -j
python -m cortexloop train    # optional: regenerate weights
python -m cortexloop bench
python -m cortexloop demo     # http://127.0.0.1:8765

Public repo (MIT): github.com/moscraciunxxx/cortexloop

Track: Physical AI.

Built With

• arm
• c
• c++
• cmake
• html
• kleidiai
• makefile
• neon
• numpy
• python
• shell