Anvil Agents — Distributed Codex Workloads on Kubernetes

Orchestrate Codex and multi-harness agent workloads on Kubernetes with durable state, reusable policy, and verifiable execution.

Comment

Inspiration

Developer agents do far more than model inference: they clone repositories, compile large codebases, run integration suites, scan dependencies, build images, query indexes, and wait on external systems. Those workloads can overwhelm a laptop and are difficult to schedule, observe, reproduce, and stop safely.

We wanted agent execution to look like real infrastructure: explicit intent, bounded resources, durable evidence, and a clean security boundary.

What it does

Anvil Agents is an open-source Kubernetes operator that turns each independent agent task into an append-only AgentRun and exactly one isolated Kubernetes Job.

Operators compose reusable run profiles, harness profiles, skill sets, tool sets, credentials, resource limits, node placement, concurrency controls, and PVC-backed state. The same control plane can run Codex, OpenCode, Hermes Agent, OpenClaw, Grok Build, Pi, or a custom container behind a common payload and structured-status contract.

This is horizontal fleet execution, not an implicit agent mesh: Kubernetes places independent runs across available machines, while Git, storage, databases, queues, or service APIs provide deliberate cross-run coordination.

A separately deployed, optional OIDC API exposes authorized run summaries and bounded live logs over Server-Sent Events. It has its own ServiceAccount and read-only RBAC, cannot read Secrets, cannot mutate AgentRuns, and does not let clients select arbitrary Pods, Jobs, or containers.

How we built it

  1. A user, schedule, CLI, or approved event creates a new AgentRun.
  2. The controller resolves namespace-local run, harness, skill, tool, Secret, and volume references.
  3. It materializes an immutable payload and records composition and content digests.
  4. It creates one Job with the chosen runner image, workload identity, resources, storage, and placement.
  5. The runner prepares source and tools, executes the selected harness, and emits structured status.
  6. The controller projects Job, Pod, result, and optional PostgreSQL archive evidence back onto the AgentRun.

The implementation is Go with Kubernetes controller-runtime and eleven v1alpha1 CRDs. It ships an OCI Helm chart, controller plus six built-in runner images, a Kubernetes-native CLI, PostgreSQL archive modes, Kind end-to-end tests, and collector-neutral labels/logs for observability pipelines.

What changed during OpenAI Build Week

The AgentRun foundation began inside Anvil Primaris before the event; we are deliberately not presenting all of it as new work.

During Build Week, we extracted that foundation into a standalone Apache-2.0 project, removed Anvil Primaris API dependencies, added deterministic harness/skill/tool composition and execution evidence, introduced the provider-neutral read-only OIDC/SSE API, expanded runner and archive support, added a standalone operator CLI and provider-neutral adverse signals, hardened the production cutover, and built digest-locked release and credential-free Kind validation paths.

The primary build thread ran in Codex with recorded model identifier gpt-5.6-sol. Codex accelerated code archaeology, extraction, controller and API implementation, CRD generation, tests, security hardening, documentation, release automation, Kind validation, and failure diagnosis. The human entrant retained authority over product scope, append-only semantics, security/RBAC, compatibility, deployment, and release decisions.

That collaboration statement does not claim that an authenticated production AgentRun completed under GPT-5.6; that separate runtime proof was not completed.

Challenges

  • Preserving the existing API identity and PVC owner-reference safety while making the runtime truly standalone.
  • Preventing crashes or ambiguous Job creation from causing duplicate expensive or mutating work.
  • Making skills and tools composable without silently transferring Secret, ServiceAccount, image, or storage authority.
  • Supporting multiple harnesses while preserving one deterministic payload and status contract.
  • Streaming useful logs without turning observability into a mutation or credential boundary.
  • Keeping the controller, six runner images, chart defaults, source revision, and release evidence aligned.
  • Clearly separating the pre-event foundation from the meaningful Build Week work.

Accomplishments we are proud of

  • Published immutable release v0.1.2 with a seven-image linux/amd64 digest lock.
  • Established an independent public repository with no Go imports from Anvil Primaris.
  • Added deterministic composition, immutable payload evidence, and fail-closed launch receipts.
  • Added a separately deployed, deny-by-default, read-only OIDC/SSE API.
  • Added first-class OpenCode, reusable external-tool composition, a standalone CLI, and three PostgreSQL archive deployment modes.
  • Passed repository verification, source-build Kind lifecycle tests, and release publication checks.
  • Built a free public Kind path that proves installation, composition, isolated Jobs, structured status, append-only behavior, and PVC persistence without credentials. For precision: that public script installs and validates the published v0.1.1 runtime, not v0.1.2, and makes no model API call.

What we learned

Durable agent infrastructure needs execution receipts, not just prompts and logs. A retry should be a new immutable object. Reusable instructions and tools must stay separate from runtime identity and credential authority. Kubernetes is a strong horizontal execution plane for independent agent tasks, but one run is still one Pod. Observability deserves its own deny-by-default process and RBAC boundary. And a free, credentialless proof path makes an AI developer tool much easier to evaluate.

What's next

We want earlier immutable acceptance for queued work, generalized pinned workspace and context sources, extensible harness registration, explicit storage reclaim policy, stronger quota and scheduling diagnostics, SBOM/signing/provenance, certified multi-architecture releases, workflow fan-out with approvals, and explicit cross-cluster dispatch.

Try it

Built With

Share this project:

Updates