Arm Adaptive Guardrail Middleware

An SLA-aware middleware that dynamically picks the strongest LLM guardrail tier that fits a real-time latency budget turning security strictness into a tunable and measured tradeoff on Arm.

Comment

Inspiration

Almost every LLM guardrail runs at one fixed strictness level, no matter how much time budget is actually available for a response. That felt like a missed optimization opportunity; security shouldn't be all-or-nothing, it should be a resource you can trade off against speed, just like any other constraint in systems design. I wanted to build something that treats guardrail strictness itself as the thing being optimized, rather than optimizing the LLM around a fixed guardrail.

What it does

This project is an SLA-aware adaptive security middleware for LLM applications. Given a latency_budget_seconds, it dynamically picks the strongest defense tier that still fits within that budget:

  • No Defense - baseline, no protection.
  • Output Filter - regex-based redaction of known sensitive patterns.
  • LLM Judge Only - a second LLM call reviews the draft reply before it's shown.
  • Full Judge - combined output filtering + LLM judge review.

Under a tight budget, it falls back to the fastest tier. Under a looser budget, it upgrades to the strongest available tier. All decisions are based on real calibrated latencies measured on the actual hardware, not estimated guesses.

It runs entirely locally on an Arm64 (AWS Graviton) instance using a quantized open-weight model (Qwen2.5-0.5B) via Ollama - no external API calls, no rate limits, no cost.

How I built it

  1. Set up an AWS EC2 Graviton (Arm64/aarch64) instance and confirmed the architecture with uname -m.
  2. Installed Ollama and ran a small quantized LLM locally, after discovering that even a 1B-parameter model was too large for the instance's ~1GB RAM (it caused severe swap thrashing) - I switched to a 0.5B model that fit comfortably.
  3. Ported four existing guardrail configurations (no-defense, output-filter, judge-only, combined judge) into standalone callable tiers.
  4. Built an adaptive controller that measures real per-tier latency and picks the strongest tier that fits a given time budget.
  5. Ran a real attack suite (prompt injection, data leakage attempts) against all four tiers and the adaptive controller, measuring actual leak rates.
  6. Generated a Pareto tradeoff chart (leak-rate vs. latency) to visualize the security/speed tradeoff.
  7. Benchmarked a more aggressively quantized model variant (q2_K) against the standard quantization, measuring a real ~34% tokens/sec improvement and ~15% smaller model size - a concrete Arm-specific optimization result.

Challenges I ran into

  • RAM constraints: the EC2 instance only has ~1GB RAM. A 1B-parameter model caused the OS to swap to disk, making inference painfully slow (30+ seconds per call). Switching to a smaller 0.5B model and adding swap space fixed this.
  • Disk space: ran out of space mid-dependency-install (torch and its dependencies are large); resized the EBS volume from 8GB to 20GB live without restarting the instance.
  • Latency variance: real hardware latency varied wildly between runs (one tier ranged from 1.6s to 28.7s across 5 runs) - likely due to model/cache warm-up effects. Documented this honestly rather than hiding it.
  • Judge non-determinism: the same "Full Judge" tier leaked 0/7 attacks standalone but 2/7 when called via the adaptive controller on identical attacks - attributed to genuine non-determinism in the small local judge model's verdicts, not a bug in the logic.

What I learned

That optimization on constrained edge/cloud-Arm hardware is as much about working within real memory and compute limits as it is about algorithm design - and that being transparent about variance and limitations (rather than cherry-picking clean numbers) makes for a more credible, useful result.

What's next

  • Multi-sample judge voting to reduce the non-determinism found in the Full Judge tier.
  • Testing the full 12-attack suite (currently limited to a 6-attack representative subset due to hardware speed constraints.
  • Real hardware power telemetry instead of the current estimated energy metric.
  • Exploring Arm KleidiAI/SVE2-accelerated inference for the guardrail models specifically.

Built With

Share this project:

Updates

Submission history