NeonRecall

Up to 10.2x faster search-only semantic retrieval on native Arm64, with INT8 vectors at about 25% of FP32 payload and reproducible NEON proof.

Comment

Project overview

Track 2: Cloud AI

NeonRecall accelerates the CPU search stage that follows embedding inference in a cloud semantic-retrieval service. On native Arm64, its purpose-built INT8 index delivered up to 10.214x higher single-threaded search throughput while using 25.1%-25.3% of the FP32 encoded-vector payload in the accepted three-run evidence set.

The project is deliberately narrow and auditable. It does not claim to accelerate model inference, embedding generation, index construction, or quantization. It compares only FP32 and INT8 retrieval arithmetic over identical, already-produced embeddings.

What it does

NeonRecall provides:

  • separate FP32 and symmetric per-vector INT8 indexes;
  • one shared deterministic top-k implementation for a fair comparison;
  • scalar, AArch64 NEON widening, and AArch64 dot-product kernels;
  • runtime CPU-capability dispatch;
  • deterministic benchmark inputs with locked fingerprints;
  • a real-model quality path using embeddings from a revision-pinned qint8 MiniLM ONNX encoder;
  • raw ranked document IDs with independent qrels-based recall@10 recomputation; and
  • a three-repetition native Arm workflow that re-executes uploaded binaries and derives ELF and disassembly proof from those exact binaries.

The final output is reusable Apache-2.0 source code, tests, benchmark contracts, machine-readable results, and a native-Arm evidence pipeline—not just a benchmark screenshot.

Judge it in 60 seconds

On native Arm64 Linux:

git clone https://github.com/LubuSeb/neonrecall-arm.git
cd neonrecall-arm
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON
cmake --build build --parallel
ctest --test-dir build --output-on-failure
python3 -B -m unittest discover -s scripts/tests -v
./build/neonrecall selftest --json
./build/neonrecall search-demo

Then inspect the accepted aggregate at https://github.com/LubuSeb/neonrecall-arm/blob/main/evidence/results/summary.json and the five passing jobs plus downloadable raw artifacts at https://github.com/LubuSeb/neonrecall-arm/actions/runs/29539633599.

The local search-demo is a deterministic smoke test. The full accepted route—including the pinned model, real embeddings, three benchmark repetitions, raw retrieval results, environment capture, and instruction proof—is documented step by step in the repository README.

Reproducible Arm evidence

The submitted chart and figures use one canonical evidence set: native-Arm run 29539633599 for source commit 904c8bd7120ef419a68019ed5ab92f43bf49d55c.

A later evidence-media commit, 8f9a79f95869fda2897424463de759b001150d5a, independently passed the same validation gates in run 29548062166. Its fresh samples are intentionally kept separate rather than mixed into the canonical aggregate below.

Canonical results:

  • Kernel: aarch64-neon-dotprod
  • 384-dimensional workload median throughput speedup: 10.214x
  • 768-dimensional workload median throughput speedup: 9.354x
  • 1536-dimensional workload median throughput speedup: 7.517x
  • INT8/FP32 encoded-vector payload ratio: 25.1%-25.3% across all runs and workloads
  • FP32 recall@10 across three runs: 1.0, 1.0, 1.0
  • INT8 recall@10 across three runs: 1.0, 1.0, 1.0

The speed, latency, and payload numbers come from the exact deterministic synthetic-vector workloads above. Separately, the quality sanity check compares FP32 retrieval arithmetic with INT8 retrieval arithmetic over embeddings from the same pinned qint8 MiniLM ONNX encoder. Both paths achieved macro recall@10 of 1.0 in all three runs on the included 18-document, six-query demo, with identical top-ten lists. This is not a full-FP32-model versus INT8-model comparison and does not claim general retrieval quality.

The public evidence is GitHub Actions run 29539633599 on commit 904c8bd7120ef419a68019ed5ab92f43bf49d55c. The three workload speedup samples were 9.349x/10.465x/10.214x, 9.311x/9.921x/9.354x, and 4.988x/9.243x/7.517x, respectively. The aggregate uses the median of each three-run set and requires every individual run to be directionally faster. Each variant collected 256, 128, and 64 timed search samples per run for the 384d, 768d, and 1536d workloads, respectively.

Why Arm and Cloud AI

NeonRecall targets the CPU retrieval layer of a cloud AI stack. The evidence workflow executes a pinned MiniLM ONNX encoder on native Arm to produce embeddings; the measured optimization then accelerates search over those embeddings. This boundary makes the result useful for RAG, recommendations, and vector-search services without pretending the retrieval benchmark is a model-inference benchmark.

The optimized path is native AArch64 code and selects an Arm NEON implementation from runtime capability detection. The public aggregate runs on a native ubuntu-24.04-arm runner, re-executes every uploaded binary, verifies ELF64 little-endian AArch64 identity, regenerates disassembly, and requires the corresponding sdot or widening-multiply instruction inside the selected kernel.

The workloads model single-threaded semantic search over precomputed embedding indexes, a common building block for retrieval-augmented generation, recommendations, and vector-search services on Arm cloud instances. The output is reusable source code, tests, benchmark contracts, and a reproducible evidence pipeline rather than a screenshot-only optimization claim.

Built with

  • C++20
  • CMake
  • Arm AArch64 NEON intrinsics
  • GitHub Actions native Arm64 runners
  • Python 3.12
  • ONNX Runtime
  • Hugging Face MiniLM model assets pinned to one revision
  • Apache License 2.0

Why it should win

NeonRecall combines a substantial measured Arm64 gain with a reusable way to trust that gain. The baseline and optimized paths share ranking logic and inputs; quality is recomputed from raw outputs; every individual run must improve throughput and p95 latency; and native instruction proof comes from the executable the aggregate runner actually re-executes.

For developers, it is a compact reference for per-vector quantization, runtime Arm dispatch, deterministic retrieval testing, and evidence-driven optimization. For judges, the claim can be traced from the headline result to machine-readable samples, source commit, CPU identity, binary hash, ELF architecture, and selected instructions.

That combination—fast native code, a one-minute evaluation path, reusable artifacts, and an evidence contract designed to reject misleading results—is what makes NeonRecall stand out.

Project provenance

NeonRecall was created from scratch during the June 10-August 14, 2026 submission period. OpenAI Codex assisted with implementation, adversarial review, validation, and submission preparation. No pre-existing project code was incorporated; third-party model and runtime assets are identified, licensed, revision-pinned, and digest-locked in the public repository.

Built With

  • aarch64
  • arm64
  • c++
  • cloud-ai
  • cmake
  • github-actions
  • int8
  • neon
  • onnx-runtime
  • python
  • semantic-search
  • vector-search
Share this project:

Updates

Submission history