ArcadeOps — The Release Gate for AI Agents

Inspiration

The most dangerous status in agent infrastructure is “completed.”

An AI agent can finish running without achieving the user’s goal. It may use the wrong model, call the wrong tool, rely on stale configuration, or validate a deployment that is no longer current — while still producing a convincing answer and a green status.

Traditional CI/CD systems verify deterministic software. Production AI agents introduce probabilistic reasoning, tool use, model fallback, budgets, mutable policies, and distributed workers.

We created ArcadeOps to answer a more important question:

Is this AI result genuinely ready to ship — and can the team prove it?

Our Build Week idea was to use ArcadeOps as the release gate for its own production deployment. This created a memorable but demanding test: the platform had to evaluate its own runtime evidence honestly, including when that evidence was insufficient.

What it does

ArcadeOps is a mission control and release gate for production AI agents.

It separates three truths that most agent platforms combine:

Execution truth: did the process finish?
Goal truth: did the agent achieve the requested outcome?
Release truth: does the evidence authorize the result to ship?

A run can therefore be technically completed while its goal fails or its release remains blocked.

For each mission, ArcadeOps records and exposes:

requested and recorded runtime models;
model calls, tokens, latency, and estimated cost;
tool calls and their results;
deterministic validation checks;
goal and release status;
code-revision provenance;
budget and asynchronous-input provenance;
a frozen replay contract and its hash;
run lineage;
a hash-linked evidence certificate.

The Build Week Release Gate uses GPT-5.6 and calls monitor_check against the production worker. It verifies HTTP success, worker presence, stale workers, and revision alignment. Missing, malformed, stale, or contradictory evidence fails closed.

Mission Control turns this technical depth into a guided judge experience: verdict first, then execution/goal/release truth, model and tool evidence, replay information, certificate, live health, and clearly labelled historical runs.

How we built it

The execution path is fully asynchronous:

ChatGPT → OAuth/MCP → ArcadeOps Control Plane → BullMQ/Redis → OVH worker → OpenAI Responses API → GPT-5.6 → monitor_check → deterministic validation → release decision

The control plane runs on Next.js and Vercel. Agent work is processed separately through BullMQ, with Redis over TLS. Prisma and PostgreSQL persist organizations, agents, tasks, runs, model calls, tool steps, budgets, replay contracts, and evidence.

GPT-5.6 provides bounded reasoning and tool calling. ArcadeOps then independently evaluates the resulting evidence. The model participates in the mission, but it cannot simply declare its own work valid.

We also persist the recorded runtime model rather than trusting only the requested model. Model fallback provenance is exposed so another model cannot masquerade as GPT-5.6.

ArcadeOps existed before the event as a broader agent-operations platform. The principal Build Week additions were:

execution / goal / release truth separation;
deterministic release validators;
frozen replay contracts and drift detection;
recorded runtime model and fallback provenance;
hash-linked evidence certificates;
the self-release Agent Release Gate;
the focused English Mission Control journey;
adversarial tests and evidence-driven documentation.

Codex accelerated the work by auditing the baseline, isolating semantic failure modes, implementing test-first changes, generating adversarial cases, hardening the asynchronous MCP path, diagnosing provenance loss, building the judge experience, and reviewing every public claim against its evidence.

Human decisions remained explicit: scope, architecture, merge approval, deployment, account access, and submission stayed under owner control.

Challenges we ran into

The hardest challenge was not getting an agent to return SAFE. It was making SAFE trustworthy.

A diagnostic run completed, returned the correct output, achieved its goal, and passed deterministic checks. However, its certificate was refused because a scoring hook had replaced the run metadata and removed its code and replay provenance.

We did not rewrite or backfill that run.

We preserved it as a non-certified diagnostic, fixed the destructive metadata write with an organization-scoped metadata merge, and added regression tests covering code provenance, budget information, asynchronous input, replay contracts, and certificate issuance.

We also faced several distributed-systems challenges:

keeping Vercel and the worker on the same revision;
preventing stale release contracts from approving a newer deployment;
preserving an agent’s configured GPT-5.6 model through MCP execution;
separating historical evidence from current production health;
presenting complex provenance without overwhelming the reviewer.

These failures became useful product demonstrations: ArcadeOps blocked releases precisely when the evidence was incomplete or contradictory.

Accomplishments that we're proud of

We are proud that ArcadeOps produces evidence instead of merely displaying confidence.

A verified Build Week run:

recorded three gpt-5.6-sol model calls;
invoked monitor_check exactly once;
received HTTP 200;
passed its applicable deterministic checks;
achieved goal status pass;
achieved release status approved;
returned the exact output SAFE;
stayed below its $0.20 execution cap.

A separate historical run returned BLOCKED when its release contract referenced the wrong production revision. This demonstrated fail-closed behavior with real deployment drift.

We are equally proud of the product design. Mission Control condenses a distributed agent trace into a review path that can be understood in under two minutes without hiding the underlying evidence.

The strongest accomplishment is the system’s honesty: successful, blocked, historical, non-certified, and live states remain visibly distinct.

What we learned

We learned that trustworthy agent infrastructure requires evidence to be a first-class product object.

Three lessons shaped ArcadeOps:

Execution is not success. A process ending does not prove that the user’s objective was achieved.
Success is not authorization. Even a correct output should not ship without adequate release evidence.
Evidence must be run-scoped. A model call, tool result, contract, or certificate from another run or revision cannot prove the current result.

We also learned that model provenance matters. The requested model is an intention; the recorded runtime model is evidence.

Finally, we learned that a well-designed failure can be more convincing than a polished success. A trustworthy platform should clearly explain why it blocked a release.