Heatsink

Inspiration

NMC² rents out GPUs. When a customer's GPU runs out of memory or overheats at 3am, someone gets woken up to read dashboards, dig through logs and find the right runbook, all half-asleep. AI agents could do most of that work. But no ops team will let an agent touch production unless they can see what it does, limit what it can touch, and keep a human on every risky action. So we didn't build one agent. We built the platform NMC²'s own teams would use to build agents safely.

What it does

Heatsink lets an NMC² engineer build, test and run AI agents for their own operations, with the safety rails built in.

• Build in plain English. Describe the job ("watch the GPUs and tell me when something's wrong") and Heatsink drafts the agent: its instructions, the tools it needs and nothing more. You see the full spec before anything runs.
• Crash-test before deploying. Each draft runs in a throwaway sandbox and faces 14 attacks (prompt injection, fake urgency, instructions hidden in data) and gets a score. An agent that can do something sensitive can't deploy without passing, unless an approver overrides it, and that's logged.
• Deploy on Kubernetes. Every agent gets its own pods, keys, service account and network rules. Delete it and everything it owned goes with it.
• Least privilege, enforced in code. Agents only reach the tool functions they were given. Sensitive actions, like stopping a customer's job or moving money, pause until a human approves that exact call. Agents can message only the teammates they're allowed to, at both the app and the network layer.
• A real incident, worked by a team of agents. On a real NVIDIA T4, the platform reads the GPU every 10 seconds. When a job crashes or the GPU passes 75°C:
  • the triage agent wakes up, reads the readings and logs, and finds the right runbook section;
  • it hands its diagnosis to a fixer agent, which writes a code change so the problem doesn't happen again;
  • it proposes the immediate fix, which waits for a human;
  • the platform phones the on-call engineer and explains what happened.

You open the console, read the incident, and approve with one click.

• Nothing is a black box. Every request has a trace showing each model call and tool call. An append-only audit log records every deploy, alert, approval and phone call. Roles (viewer, builder, approver, admin) control who can do what, and every agent has a daily token budget.
• Not just GPUs. The same platform builds any agent: a research assistant with live web search, a shopping helper with real prices, or a Capital One banking assistant whose every transfer waits for approval.

How we built it

• Kubernetes operator (Python, kopf), with Agent and Tool custom resources. It turns each Agent into a Deployment, Service, ServiceAccount, Secret and NetworkPolicy, and gives each team namespace a ResourceQuota.
• One agent runtime (FastAPI) runs every agent. Policy lives in code, not in the prompt: allowlists, read/write/sensitive classes, and signed one-time approval grants tied to the exact arguments.
• Tools are MCP servers:
  • GPU fleet: live readings and jobs on a Brev T4 over an SSH tunnel.
  • Runbooks: a searchable Markdown knowledge base.
  • SerpApi: web, shopping and local search.
  • Capital One Nessie: sandbox banking.
• Platform API (FastAPI + Postgres): drafting, crash tests, approvals, traces, the audit log, alerting and paging.
• Voice: Cartesia places the 3am phone call. A Pipecat voice buddy in the console answers questions about the fleet and the latest incident.
• Console (React + Vite): a live map of agents and the tools they can use, plus pages for incidents, runbooks, approvals and traces. It uses a thermal-camera design where orange means a human has to decide.
• Models go through OpenRouter (Claude Haiku for the agents).

Challenges we ran into

• Linking an alert to its fix safely. Our first version matched alerts to approvals by time window. A review caught that an unrelated approval could land on the wrong incident. Now each link is bound to the exact approval id, namespace and agent.
• Agents getting killed for being busy. Stress-testing with many agents on one laptop, Kubernetes kept restarting healthy agents because their health check took longer than one second to answer. We added a startup probe and gave the checks realistic timeouts.
• Making a phone call sound human. The voice read "job-train-b8d3d7" out loud and said "g-poo" for GPU. We rewrite job names, units and acronyms into speakable words before anything reaches the voice.
• Long agent handoffs got cut off. The fixer's prevention plan was truncated in traces until we stopped capping teammate replies.

Accomplishments that we're proud of

• A real GPU incident, from crash to phone call to one-click fix, with no one typing to an agent.
• Safety that doesn't depend on the model behaving. A tricked agent can ask for anything, but only allowed calls run, and risky ones wait for a person.
• Delete an agent and every Kubernetes object it owned disappears. Re-add it and its teammates can reach it again within seconds.
• Our bank agent scores 100 on the crash test. A deliberately naive version on a cheaper model scores 46–52, and the gate blocks it.

What we learned

• Kubernetes is a great fit for agents: namespaces, network policies, quotas and owner references give you isolation and cleanup for free.
• Approvals have to bind to the exact action, not just the agent, or "approve" means less than you think.
• Observability (traces, audit, activity between agents) is what makes people willing to trust an agent at all.

What's next for Heatsink

• Connect the second GPU node (an NVIDIA L4) and test agents across a real multi-node fleet.
• Replace polling with Kubernetes watches and Postgres LISTEN for instant updates.
• Approve from Slack or the phone with a verified identity (today the call only explains).
• Let the fixer open a real pull request with its prevention change.
• Run Heatsink on NMC²'s own clusters, with their SSO and their runbooks.

Built With

• kubernetes
• python