EdgeRoute Arm

Track

Mobile AI — fully on-device inference on an Arm-powered Apple Silicon laptop. EdgeRoute requires no cloud classifier, private prompt upload, or external API.

Inspiration

Agentic software increasingly decides whether to browse, send, upload, buy, or delete. Sending every prompt to a cloud safety classifier creates cost and latency, and it can expose the exact text most likely to contain credentials, identity records, or destructive instructions. EdgeRoute moves that first decision onto the user's Arm device.

What it does

EdgeRoute classifies a request into five execution lanes: local, read-only, confirm, sensitive, or destructive. A host agent can use the result to run a local task, allow a network read, request confirmation at the moment of an external write, protect sensitive data, or block a destructive action pending target and backup checks.

The included web demo calls the real ONNX model on localhost. No prompt is sent to a hosted model or third-party API.

How it was built

The project uses a deterministic local dataset generator and a stable BLAKE2b unigram/bigram hashing vectorizer. A PyTorch MLP maps 2,048 input features to a 128-unit hidden layer and five route scores. The training pipeline exports an FP32 ONNX baseline, then applies ONNX Runtime dynamic INT8 weight quantization.

Both artifacts are evaluated through the same ONNX Runtime CPU provider on an Apple Silicon arm64 Mac. The repository includes training, export, quality evaluation, fair warm-up benchmarks, unit tests, a CLI, and a responsive local web interface.

Measured optimization on Arm64

• Model file: 1,052,284 → 266,035 bytes (74.72% smaller)
• Batch-128 throughput: 335,407 → 446,317 requests/s (33.1% higher)
• Batch-128 median: 0.3740 → 0.2742 ms (26.7% lower)
• Held-out template split: 100% for FP32 and INT8
• Prediction agreement after quantization: 100%
• Separate hand-authored phrasing challenge: 20/20

There is an honest tradeoff: single-request median increased from 0.0124 ms to 0.0171 ms. EdgeRoute's INT8 build is therefore most compelling where footprint, privacy, and batched routing matter. All raw JSON measurements are committed.

Challenges

The first model looked perfect on a random template split but scored only 35% on separately written phrases. Rather than submit that misleading result, I expanded semantic coverage, retrained, and kept the independent challenge set visible in the report. I also corrected the benchmark methodology after the first FP32 load included one-time runtime startup overhead.

What I learned

Optimization is a system decision, not a single percentage. INT8 delivered a large footprint win and stronger batch throughput, but it did not improve every latency regime. Reproducibility and transparent limitations made the result more useful than a selective benchmark would have.

What's next

The next step is an audited multilingual corpus, calibrated confidence thresholds, and deterministic policy rules around the classifier. The ONNX artifact can then be embedded in desktop and mobile agent runtimes without requiring a separate safety service.

Why it should win

EdgeRoute turns a high-stakes agent-safety decision into a practical Edge AI workload: a reusable, open-source routing layer that keeps sensitive prompts on the user's Arm device. It pairs a 74.72% footprint reduction with 33.1% higher batch throughput, preserves 100% FP32/INT8 prediction agreement, and publishes both the raw measurements and the single-request latency regression. The result is not only a smaller model; it is a transparent pattern other developers can embed before AI agents browse, send, upload, buy, or delete.

Run and validate on Arm64

On an Arm64 macOS or Linux environment:

git clone https://github.com/frank70321-hub/edgeroute-arm.git
cd edgeroute-arm
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python train_export.py
python benchmark.py
python -m unittest discover -s tests -v
python route.py "send an email to the client about the invoice"
python app.py

Open http://127.0.0.1:8765 to use the local web demo. The training data is generated locally with fixed seeds; no API key, paid service, private dataset, or cloud classifier is required.

Built With

• apple-silicon
• arm64
• css
• html
• javascript
• numpy
• onnx
• onnx-runtime
• python
• pytorch
• scikit-learn