Groundkrew

AI DevOps agents on NVIDIA Nemotron that map your cloud, deploy and self-fix builds in a sandbox, and trace outages to their real cause. Nothing changes without your approval.

Comment

Inspiration

Small engineering teams run production on AWS, Vercel and GitHub with nobody whose job is DevOps. When something breaks at 2 a.m., the error rarely points at the cause: checkout says "database unreachable", but the real problem is a secret someone changed in a different system an hour earlier. Connecting those dots means digging through logs, deploys and cloud settings by hand. We wanted the DevOps engineer these teams can't hire: one that understands the whole system, does the digging, and never changes anything without asking.

What it does

Groundcrew is an AI DevOps team made of NVIDIA Nemotron agents, running on Nebius Token Factory.

  • Discovery maps your system. Connect AWS (a read-only role by default), GitHub, Vercel, Kubernetes, or any server through a small signed agent. Groundcrew draws one map: which app runs where, which database it reads, which secret holds its password, what deploys what. Every link shows its evidence (the config file or workflow that proves it), and risks like a port open to the internet are flagged.
  • The Deployer ships your apps. Pick a repository and a target. Nemotron Nano reads the code (framework, port, health check). Nemotron Super writes the Dockerfile, and Groundcrew builds and smoke-tests it in an isolated sandbox. When a build fails, Super reads the error and fixes the Dockerfile until it passes. Only then do you see the plan, listing every resource it will create, and nothing touches your cloud until you approve it.
  • Medic fixes incidents. Groundcrew watches logs and health on its own. When an error spike starts, it opens an incident, lines up logs, deploys and cloud changes on one timeline, and finds the cause across systems. Every claim links to the log lines that prove it. It remembers past incidents and reuses a diagnosis when the new one matches. It proposes fixes in order, each waiting for a person's approval, and closes the incident only after rollout, health and error rates are back to normal. When the fix belongs in code, it opens a pull request; it never merges.
  • It works from your editor too. Coding assistants can ask Groundcrew for a change over MCP; the change still waits in Groundcrew for a person's approval, and the audit log records which assistant asked.

In the demo, Groundcrew detected a broken database secret in 1 minute 48 seconds, traced two failing services back to it, and fixed it with three approved steps. The whole investigation cost about one cent in model calls.

How we built it

  • Models: NVIDIA Nemotron 3 on Nebius Token Factory. Every call goes through one client using Token Factory's OpenAI-compatible API, with retries, fallback and JSON-schema output. A router picks the model size per step: Nano reads repositories, summarizes the system, classifies signals and checks recovery; Super writes and fixes Dockerfiles, plans deploys and runs Medic's tool steps; Ultra handles root causes and fix proposals. Each route also sets Nemotron's reasoning mode (off, low effort or on), so short steps don't pay for long thinking.
  • Escalation. When a smaller model fails a step, the step moves up one size with full reasoning. In the demo, a build Super couldn't fix passed on Ultra.
  • Cost you can see. Every model call is stored with its model, tokens and dollar cost, next to what the same tokens would cost on Ultra alone. Daily spend caps stop runaway loops. A deploy that fixed its own build cost $0.0075.
  • Incident memory. Incidents are embedded with Qwen3-Embedding-8B, also on Token Factory, so Medic can recognize a repeat and rank evidence by similarity.
  • Web research. For a new incident, Medic searches the web with Tavily and uses the results as background only, never as evidence.
  • Safety by design. Read-only by default; writes and emergency actions need an approval bound to the exact plan; secret values are never read; build sandboxes are blocked from cloud credentials.
  • Stack. Python with FastAPI, SQLAlchemy and PostgreSQL with pgvector for the API and workers; Next.js, React and Tailwind for the web app; a Go agent for servers, Kubernetes and ECS; AWS (ECS, RDS, Secrets Manager, CloudWatch, CloudFormation), Vercel and GitHub App integrations; an MCP server for coding assistants.

Challenges we ran into

  • Long context isn't the same as usable context. Nemotron 3 Nano accepted a 65,000-token prompt but started answering wrongly around 20,000 tokens. We gave every model a measured "usable context", and bigger prompts go to a larger model.
  • Strict schemas can force confident wrong answers. With a strict enum, every size picked a wrong language for a Rust repository rather than "none of these". Every enum now includes other, and values are normalized before validation.
  • Cheaper reasoning isn't always cheaper. Low-effort reasoning saved tokens on Nano and Super but not reliably on Ultra, so we set reasoning per route from measurements.
  • Trust. An agent that touches production must be checkable. Evidence links on every claim, approvals bound to the exact plan and a full audit log took as much work as the agents themselves.

We wrote all of this up, with measurements, in FEEDBACK.md in the repository.

Accomplishments that we're proud of

  • A real outage traced across two systems, with evidence for every claim, in under two minutes and for about a cent.
  • Builds that repair themselves in a sandbox before anything reaches the cloud.
  • Model routing that sends most of the work to the small, cheap models and uses Ultra only where it matters.
  • Nothing changes without a person, including requests that come from AI coding assistants.

What we learned

Picking the right model size per step matters more than picking the biggest model. Nano and Super handle most of the work quickly and cheaply, and Ultra earns its cost on the hard reasoning. We also learned that for an AI that touches production, being able to show its proof is as important as being right.

What's next for Groundcrew

  • Run builds on Nebius Serverless Jobs instead of a sandbox on our own host.
  • Use Nemotron 3.5 Lightning for fast tool steps once it has public pricing, and an NVIDIA embedding and rerank model when Token Factory offers one.
  • More targets (Azure is in preview), and pull requests that fix the cause in code for more kinds of incidents.

Built With

Share this project:

Updates

Submission history