Polygraph

Fail-closed proof and rollback gates for Arm64 cloud inference performance claims.

Comment

Project Overview

Track: Cloud AI. Polygraph is a fail-closed verification and deployment gate for Arm64 cloud inference. Instead of trusting a startup banner or timing result, it checks three separate layers: whether accelerated kernels exist in the server binary, what the runtime says it selected, and whether those kernels actually executed under a non-halting debugger breakpoint. It then measures throughput, time to first token, end-to-end latency, memory, and recovery separately before a candidate is kept or rolled back.

The Arm CPU is not "lying." The potentially misleading signal comes from the software build, feature probe, startup report, dispatcher, or benchmark interpretation. L1/L2/L3 proves execution provenance for the measured workload; it does not by itself prove speed, optimality, application correctness, or production readiness.

The headline baseline was a llama.cpp build on a DGX Spark that completed successfully and printed KLEIDIAI = 1, yet contained 0 usable kai_run_matmul entry points. The technical change was an explicit Arm feature-target build plus a reusable L1/L2/L3 verification tool. The measured impact on the tested 7B model was 48.64 to 222.14 tok/s prefill, a 4.57x comparison, and 11.17 to 18.45 tok/s decode, a 1.65x comparison. The 0.5B decode result was 0.99x, so we report it as no measurable effect rather than generalizing the 7B result. This is a broken-versus-corrected source-build comparison on one Arm CPU configuration. Polygraph did not invent a new matmul kernel, and stock llama.cpp releases are not claimed to contain this defect.

We also built an experimental optimization candidate: patch 0002 changes only the default generation thread count to KleidiAI's runtime-detected SME2 cap while preserving the batch/prefill default. An isolated Apple M4 Max run was favorable, but the stronger test was the distinct-binary Arm64 llama-server campaign. Its median candidate/baseline throughput was 0.9330x and its verdict was FAIL / ROLLBACK_TO_BASELINE. We therefore do not present the patch as a promoted optimization.

The repository is public and Apache-2.0 licensed. Its measurement artifacts, raw request rows, telemetry, workflow receipts, validation receipts, negative results, and claim checker are all committed for judges to inspect.

Challenge-period confirmation: all work submitted here was created or meaningfully updated during the challenge period. The public commit history, timestamped workflow receipts, and immutable evidence release provide the provenance record.

The final evidence package also adds a same-artifact Arm64 temporal-control campaign: 79,684 measured requests, 0 measured failures, 36,000 aggregate measured seconds across two shards, and a 9,000-second longest continuous segment. Both shards passed the synthetic gate and rollback decision. Because the baseline and candidate artifacts are identical, this is evidence about the campaign machinery and longer aggregate exposure — not an optimization or uplift claim.

Functionality / Output

The primary interface is:

tools/polygraph list
tools/polygraph explain TARGET
tools/polygraph check TARGET
tools/polygraph check --binary PATH --symbols REGEX --run "COMMAND"

Polygraph returns human-readable and JSON output plus contractual exit codes: 0 for a match, 1 for a measured mismatch, and 2 for undetermined. The two-minute make demo compiles two tiny programs that print the same using fast path: yes banner. L3 then proves that the liar never calls the fast function and the honest build does. This portable fixture demonstrates the verifier and exit-code contract; it is not the Arm benchmark.

The current Arm64 evidence is GitHub Actions run 31294460364. It adds a production-shaped CPU-only llama-server campaign (-ngl 0): mixed traffic, concurrency 1/2/4 capacity points, a 10-minute concurrency-4 soak, request-level JSONL, one-second telemetry, and two controlled restarts. It completed 1,543 measured requests with 0 failures; all 10 configured checks passed. Its exact claim boundary is still:

actualProductionTraffic:false
productionReady:false
candidateOnly:true

The complementary RTX PRO 6000 run 31289517517 retained 7,573 measured requests with 0 failures on a larger model, but it is explicitly armContestEvidence:false and is included only as a non-Arm systems control.

The supplementary Arm64 run 31312723300 used the exact same server/model artifacts for baseline and temporal-candidate labels. It retained 79,684 measured requests with 0 measured failures. Its 36,000 seconds are 10 aggregate measured hours across four independent segments on two shards; the longest continuous segment is 9,000 seconds (2.5 hours). Both shards returned PASS / KEEP_CANDIDATE, with productionReady:false and deploymentAuthorized:false.

Setup Instructions

Fastest path, with no Arm hardware and no model download:

git clone https://github.com/tomyimkc/polygraph.git
cd polygraph
make demo
python3 tools/check_claims.py

To reproduce the contest's free Arm64 pipeline, fork the public repo and run verify-free-arm64.yml from GitHub Actions. The workflow uses GitHub's hosted ubuntu-24.04-arm runner, pins the llama.cpp revision and model hashes, runs the three-model dispatch/throughput matrix, and on manual dispatch runs the longer server-readiness job.

The separate verify-production-arm64.yml manual workflow reproduces the same-artifact smoke/long campaign and is restricted to the fork's main branch. The exact long dispatch fields and receipt-verification commands are in docs/CONTEST-EVIDENCE-MAP.md.

Exact local commands, checksum checks, workflow commands, and artifact inspection commands are in docs/CONTEST-EVIDENCE-MAP.md.

Why it should win

Technological Implementation — 40 points: Polygraph does not stop at a benchmark. It combines static symbols, runtime selection, and real execution counts; works across lldb and gdb; has a fail-closed CLI and CI contract; validates its own debugger probe; exposes an MCP interface; and pins every numeric prose claim to evidence. For Cloud AI it connects that provenance to CPU-only llama-server concurrency, latency, memory, restart, and rollback checks.

"WOW" factor — 25 points: a successful build said KLEIDIAI = 1 while shipping 0 usable matmul entry points. Correcting that tested source build produced a 4.57x measured 7B prefill comparison. This is explicitly not a new kernel or universal Arm speedup. In a second finding, the banner, selection log, and symbol count were identical while L3 changed from 0 to 7,968 actual kernel calls.

Potential Impact — 20 points: the tool is generic to any binary and symbol regex, the repo is public under Apache-2.0, two upstream reports cover the project's original findings, and an independently reported mechanism was reproduced with 15 debugger-backed runs. The impact statement stays honest: the zero-kernel defect does not affect stock releases. The target users are source builders, framework/release engineers, CI owners, and cloud inference operators who need to stop a silent fallback or invalid performance assumption before fleet rollout.

User Experience / Developer Experience — 15 points: one clone and make demo proves both the positive and negative detector paths in about two minutes. JSON output, presets, ad-hoc mode, explicit exit codes, graceful degradation, and a free hosted Arm64 workflow make the method usable without specialized hardware.

Most importantly, Polygraph applies its standard to itself. We published a retraction, kept a slower patch visible, recorded a no-effect result, and refused to turn a synthetic readiness PASS or a non-Arm control into a production or Arm claim.

Judge evidence index

  1. Start here — interactive demo page: https://tomyimkc-polygraph-arm-demo.static.hf.space/.
  2. Website media captures: media/website-capture-20260813/ and its receipt/checksum manifest.
  3. Reproducible static-page source bundle: space/ (the exact website-only bundle used by the public HF Space, with no terminal video asset).
  4. Claim-by-claim evidence: docs/CONTEST-EVIDENCE-MAP.md.
  5. Scope and objections: docs/JUDGE-FAQ.md.
  6. Run the product: make demo, then inspect tools/polygraph and docs/QUICKSTART.md.
  7. Headline build finding: results/server/spark-provenance.txt and results/scale/scale-experiment.json.
  8. Experimental candidate and rollback: patches/0002-kleidiai-sme-aware-thread-default.patch, results/AUTODEFAULTS.md, and the Arm64 differential receipt.json.
  9. L1/L2 agree but L3 fails: results/upstream/FINDING-4-CUDA-HOST-BUFFER.md and its 15-run JSON.
  10. Arm64 run 31294460364: results/production-readiness/arm64-31294460364/summary.json, validation-receipt.json, and workflow-receipt.json.
  11. Long same-artifact Arm64 campaign 31312723300: results/production-readiness/arm64-campaign-31312308726-31312723300/long-validation-receipt.json, aggregate/shard receipts, and package-sha256sums.txt.
  12. Immutable evidence release: arm-create-evidence-31312723300, preserving original GitHub artifact ZIPs, build provenance, campaign source, external validations, and historical video provenance.
  13. Non-Arm control 31289517517: results/production-readiness/pro6000-31289517517/aggregate.json and README.md.
  14. Negative results and corrections: results/REMEASURE-2026-08-04-QUIET.md, patches/README.md, results/GENERALIZATION.md.
  15. Claim integrity: docs/CLAIMS.md, tools/check_claims.py, and .github/workflows/claims.yml.
  16. Final submission check: docs/CONTEST-SUBMISSION-CHECKLIST.md.

Built With

  • arm64
  • bash
  • c
  • c++
  • cmake
  • dotprod
  • gdb
  • github-actions
  • i8mm
  • jsonl
  • kleidiai
  • llama.cpp
  • lldb
  • mcp
  • neon
  • python
  • sme2
  • sve2
  • vllm
Share this project:

Updates