Edge Deploy Agent

Inspiration

Most AI demos run on a big cloud GPU. A lot of real products run on small devices instead, like a Raspberry Pi. On those devices, "make the model faster" is a slow job. You try a setting, wait, measure, and try again, and one wrong choice can ruin accuracy or crash the board.

I wanted to see if an AI agent could do that job safely, on real hardware, with a budget it has to respect.

What it does

Edge Deploy Agent tunes a speech model (wav2vec2-base-960h) for my Raspberry Pi 4.

I give it a budget: p95 latency, RAM, and word error rate (WER). NVIDIA Nemotron plans one experiment at a time: which model variant, how many threads, which optimization level. The Pi runs each benchmark. The agent reads the result and decides what to try next, until it finds the best model that fits the budget or stops and says why.

Results on the Pi, same settings:

In the live demo run the agent chose the same winner again (1962 ms, 388 MB).

How I built it

• Agent: a LangGraph state machine: research, propose, guard, human approval, execute, evaluate, report. Nemotron 3 Super plans, served through Nebius Token Factory. Tavily searches the docs while it plans.
• Safety: the planner is never trusted. A deterministic guard with an allowlist decides what can run. A regex filter screens documents. NeMo Guardrails input and output rails are judged by Nemotron 3 Nano. The agent also can't finish before it has measured every model variant that passes the WER gate.
• Pi worker: the Pi pulls jobs over HTTPS and runs only allowlisted commands as an unprivileged user. It cools down before each benchmark and records temperature and throttling, so a hot Pi can't give false numbers.
• Infrastructure: Terraform builds a local Kubernetes cluster (kind) with Triton Inference Server, my API and MLflow.
• Tracking: MLflow records every trial as a nested run. OpenTelemetry traces each agent step and each LLM call.

What I tested

I wrote 60 prompt-injection attacks that try to hide commands in documentation the agent reads. With only prompt hardening, 2 of 60 fooled the planner. With the guards on, none reached the Pi. The guards had a cost: the NeMo input rail wrongly blocked 1 of 30 harmless documents, and the output rail blocked one real proposal in my live demo. I report both in the README.

Challenges

• Static INT8 broke accuracy. My first static quantization gave 100% WER, then 27%. Skipping the matrix multiplies that multiply two activations together brought it to 12%. That was still worse than dynamic INT8, which the agent correctly chose.
• A hot Pi lies. The board throttled at 82 C and gave noisy numbers. I added a cooldown before every benchmark and recorded throttling, so each run is marked valid or not.
• The agent quit too early. A smaller planner model tried to finish after one step without measuring anything. I added a rule that refuses a finish until every passing variant has been measured, and switched to Nemotron 3 Super.
• Plumbing. Port conflicts on my Mac, MLflow running out of memory in the cluster, and API overload errors during the guardrails test. I fixed these with retries and fewer workers.

What I learned

• A planner model is useful, but the safety has to live in plain code that doesn't depend on the model.
• On real hardware, measurement quality matters as much as the optimization itself.
• Honest results are more useful than impressive ones. The best model was the simple one.

What's next

I wrote a Terraform plan for Nebius AI Cloud, but I have not applied it. Next I want to run the same agent on more devices and models.

Built with

Nemotron 3 Super and Nano, Nebius Token Factory, NeMo Guardrails, LangGraph, Tavily, ONNX Runtime, Triton Inference Server, Terraform, Kubernetes (kind), Docker, MLflow, OpenTelemetry, Prometheus, Python, Raspberry Pi 4

Built With

• docker
• fastapi
• github-actions
• kind
• kubernetes
• langgraph
• llm-agents
• mlflow
• nebius-token-factory
• nemotron
• nvidia-nemo-guardrails
• onnx-runtime
• openai-api
• opentelemetry
• prometheus
• python
• quantization
• raspberry-pi
• tavily
• terraform
• triton-inference-server
• wav2vec2