AI Workbench

An agent on Nemotron for the engineer on call: it reads a failed Terraform or Ansible log, checks the host without changing it, and fixes it only after you approve.

Comment

About the project

Inspiration

At 2 a.m. a Terraform apply fails with connection refused. The fix is usually in a runbook someone wrote months ago, but the engineer on call still has to find it, check the host and type the fix by hand. Chat assistants guess without seeing the host; autonomous agents see it, but nobody wants one running docker volume rm unasked. I wanted the middle ground for small platform teams and homelab operators without an SRE rota: an agent that does the legwork and leaves the decision to a person.

What it does

Paste a failed log, or pick a sample, and press Run triage. The steps stream into the page:

  1. Retrieve. The error lines are matched against 13 runbooks, 12 of them written from real fixes.
  2. Diagnose. Nemotron runs checks such as docker ps or php -v. The server allows only commands that read, checks every part of a chained command and refuses secret reads.
  3. Propose. Any change goes through propose_fix with the commands, the reason and the risk, and the run stops at an approval card. Fixes that delete data are flagged DESTRUCTIVE.
  4. Approve or reject. Approve and the fix runs; reject with a note and the model tries again.
  5. Verify and report. The agent checks the host again and writes the root cause, what was done and what to do next.

The newest sample follows the news. On 21 October 2026 PHP fixed two critical remote code execution flaws (PHP2Shell, CVE-2026-109618 and CVE-2026-109619). In the sample, a PHP version gate blocks a production deploy. With runbooks, the agent ran 4 checks and found in 7 turns and 22 seconds why the server was never patched: PHP came from the ondrej/php PPA, which unattended upgrades didn't cover. It held the PHP restart for approval. With Use runbooks off, it ran 2 checks, upgraded a single package and stopped at "running 8.3.30", guessing instead of checking.

Runbooks go stale. When one cites a CVE, the server sends the first fix back and asks the agent to check the live advisory with Tavily on official security sources, and the approval card lists only sources the search returned. The report then says "Runbook check: confirmed" or "may be out of date", with the link. In the PHP2Shell run Tavily returned the CISA, Ubuntu and Red Hat advisories and the agent cited CISA; without a runbook nothing triggers the check, and an open search returned install guides.

Commands run against a simulated host that tracks state, so a fix really changes what the next check sees. A Token usage page shows calls, tokens and time on Token Factory as they happen.

How I built it

  • Model: NVIDIA Nemotron 3 Super (nvidia/nemotron-3-super-120b-a12b) on Nebius Token Factory, through its API compatible with OpenAI's, using native tools / tool_calls. In early single prompt tests Super answered as well as Ultra at about a quarter of the cost. A Python and FastAPI loop caps each run at 12 turns and 6 checks, keeping 3 to verify the fix.
  • Retrieval: runbooks embedded with Qwen/Qwen3-Embedding-8B, also on Token Factory; an eval finds the right runbook for 15 of 15 test logs.
  • Host: a simulated host per sample that maps commands to output and tracks state. A real executor (SSH or Ansible) can replace it behind the same run / apply interface.
  • Hosting: one Docker container on a Nebius Serverless AI endpoint (cpu-d3, 2 vCPU, 8 GB, uk-south1), with secrets in Nebius SecretStash. It runs as one replica today, and the same image can run on Kubernetes, for example Nebius Managed Kubernetes.
  • Shipping: GitHub Actions runs 136 tests, a gitleaks secret scan, an image smoke test and Trivy on every push, and only images that pass reach the demo. The public demo has a shared password, a daily cap, a 20 KB log limit and Super as the only model.

Challenges I ran into

  • Keeping a model to checks that only read. Nemotron adds its own flags (docker ps --format ...), so exact allowlists failed; the guard now matches patterns, and each bypass a review found is now a test.
  • Budgeting the loop. The model spent every check before the fix and had none left to verify it, so 3 are now reserved.
  • Runbooks that retrieve. Prose symptoms matched badly, so runbooks now quote the literal error strings, and the eval guards that.

Accomplishments I'm proud of

  • The approval gate is enforced on the server, not by the prompt: the model can't change the host until a person presses Approve.
  • A complete product, not a notebook: streaming UI, approval card, report, usage page and a live demo shipped through CI.

What I learned

  • Native tool calls on Nemotron Super were reliable enough to build the agent on directly.
  • A guardrail list is not a security boundary. A real executor should also run as an unprivileged user that can only read, and the README says so. In my tests, adding runbooks also changed the answer more than moving from Super to Ultra did.

What's next

  • A real executor over SSH or Ansible, as an unprivileged user.
  • Scaling out on Kubernetes by moving paused runs, run history and the daily caps (now in process memory and a local file) into Postgres, next to the pgvector runbooks, so replicas share them.
  • Updating runbooks from stale advisory checks and approved fixes, and masking personal data in logs before they reach the model.

Built With

  • docker
  • fastapi
  • github-actions
  • nebius-ai-cloud
  • nebius-serverless
  • nebius-token-factory
  • nemotron
  • nvidia
  • pgvector
  • postgresql
  • python
  • qwen3-embedding
  • tavily
  • trivy
Share this project:

Updates

Submission history