ModelOpsPilot Arm

Evidence-driven local AI routing that uses a 1.5B fast path and a measured 7B quality fallback on Arm64.

Comment

Inspiration

On-device AI teams often treat model choice as a one-time decision: deploy the largest model that fits, then accept its memory and latency cost. That wastes scarce device resources on bounded tasks, while a blanket switch to a smaller model can silently weaken privacy and safety decisions.

ModelOpsPilot Arm turns model selection into a measured runtime policy: use the smaller model when evidence says it is safe, and preserve a higher-quality fallback when correctness dominates.

Track and challenge-period work

Track: Mobile AI. The project targets on-device AI constraints on an Arm-powered laptop: memory footprint, local inference latency, privacy, and developer workflow. During the 2026 challenge period I significantly extended the original policy router with the Arm64/Ollama benchmark harness, measured resident-memory budget, dual-model profiles, versioned raw evidence, responsive dashboard, and automated dashboard checks.

What it does

ModelOpsPilot evaluates every candidate against six gates:

  1. runtime health,
  2. context capacity,
  3. task-tier quality floor,
  4. P95 latency budget,
  5. request-cost budget, and
  6. resident-memory budget.

Only eligible models are ranked. Standard bounded work can take the 1.5B fast path; high-risk and safety-critical work escalates to the 7B profile. Each decision returns the selected model, fallback, estimated latency/cost/memory, and a rejection trace for every excluded model. Prompt bodies stay outside the control-plane audit data.

Arm64 optimization and measured output

I benchmarked two Q4_K_M Qwen2.5 models locally with Ollama on an Apple M5 Pro Arm64 laptop. Both models ran the same five deterministic tasks at temperature zero, with one cold start and two measured repetitions.

Metric 7B baseline 1.5B optimized Improvement
Model file 4,466 MB 940 MB 78.9% smaller
Resident memory 4,521 MB 1,109 MB 75.5% lower
Warm P95 latency 286 ms 159 ms 44.5% lower
Median throughput 106.5 tok/s 303.3 tok/s 184.7% higher
Exact-match quality 100% 80% 20-point loss

The quality loss is intentionally visible: the 1.5B model missed the privacy-policy task. ModelOpsPilot converts that result into a routing constraint instead of presenting the smaller model as universally better. The measured profiles make 1.5B eligible for standard work while the 7B model remains the required path for high and safety-critical tiers.

These are local single-host measurements. I do not claim energy savings or production-workload performance that the benchmark did not measure.

How I built it

The core is a typed Python policy engine with FastAPI and Pydantic. A pure-standard-library Ollama benchmark client:

  • unloads each model before the cold-start measurement,
  • warms the model once,
  • captures Ollama's native load and evaluation timing counters,
  • reads live model residency from /api/ps,
  • stores every raw response and exact-match decision, and
  • emits ready-to-use ModelOpsPilot profiles.

The evidence dashboard loads the committed benchmark JSON directly and renders the footprint, latency, throughput, and quality tradeoff with Chart.js. The same data is available as raw JSON for independent review.

I reused established open-source components instead of rebuilding runtimes: Ollama for local inference, Qwen2.5 under Apache-2.0, Chart.js for evidence visualization, and architecture patterns from TensorZero and OpenTelemetry. The repository documents the reuse boundary and reviewed upstream revisions.

Reproduce it on Arm

ollama pull qwen2.5:7b-instruct
ollama pull qwen2.5:1.5b-instruct-q4_K_M

uv venv --python 3.12 .venv
uv sync --frozen --extra dev

.venv/bin/modelopspilot-arm-benchmark \
  --repeats 2 \
  --output artifacts/arm64-benchmark.json \
  --output docs/arm64-benchmark.json

python3 -m http.server 8000 --directory docs

The repository also includes a one-command static dashboard smoke test for desktop and mobile viewports.

Challenges

The hardest part was avoiding a misleading optimization story. A smaller model was dramatically faster, but its privacy answer failed. Instead of changing the task until the result looked better, I kept the failed samples in the evidence artifact and designed the router around the measured boundary.

Cold-start numbers were another challenge: a first model download can dominate load time. The harness therefore requires models to be pulled first, explicitly unloads them, separates cold and warm measurements, and stores the benchmark method beside the results.

What I learned

Arm optimization is not only about making one model faster. A reusable policy can allocate Arm memory and latency according to the task's actual quality requirement. Transparent escalation creates a better developer primitive than either “always large” or “always small.”

Validation

  • 23 automated tests pass
  • 93.5% source coverage
  • 1,200-case deterministic routing regression
  • desktop and mobile evidence-dashboard smoke tests
  • raw benchmark responses, timing counters, pass/fail results, host architecture, and claim boundary committed to the repository

Why it should win

ModelOpsPilot Arm combines measurable device-level gains with a responsible quality boundary and a reusable developer workflow. Judges can inspect the exact responses, rerun the benchmark on another Arm64 system, consume the generated profiles, and see why every routing decision was made.

Live evidence: https://yangyangnovelist-hub.github.io/modelopspilot-arm/

Source: https://github.com/yangyangnovelist-hub/modelopspilot-arm

Built With

Share this project:

Updates