Proving Ground

The stress test that tells you if your AI agent is actually production-ready. It runs your agent through adversarial and tool-failure scenarios and scores its reliability.

Comment

Inspiration

I come from tech, but with no AI background. My work has always been about keeping systems reliable and planning for the moment things break. Like a lot of people right now, I've watched the AI era arrive with an overwhelming amount of information about agents, and seen just how easy it's suddenly become to spin one up and build a business around it. That accessibility is exciting. But it left me with a question I couldn't shake: how do you actually know if it works and what happens when it fails?

Everyone was racing to build agents. Nobody seemed to answer the quieter questions underneath the hype. Is this agent reliable, or just impressive in a demo? Would it hold up with real users? Do I even need an agent here at all?

The uncomfortable truth behind those questions is a real number: about 88% of AI agents fail once they hit production. They pass every demo and look flawless on the happy path, then the first time a tool errors or a user sends something adversarial, they quietly make something up and report success, and nobody catches it until a real customer does.

So instead of guessing, I built the thing that would answer it for me, and for anyone else entering this space wondering the same thing: a proving ground an agent has to survive before you ship it.

What it does

Proving Ground stress-tests an AI agent and returns a reliability scorecard with a clear verdict: Demo-grade, Pilot-grade, or Production-grade.

You give it a target agent either by pasting the agent's own instructions and mock tools, or by pointing it at a live HTTP endpoint (a deployed agent's API). It then runs the agent through a standardized battery of six scenarios:

  • Happy path — baseline competence
  • Ambiguous input — does it clarify, or guess wrong?
  • Adversarial injection — does it resist prompt-override attacks?
  • Tool failure — when a tool errors, does it recover honestly or fake success?
  • Multi-step — does it stay coherent across dependent steps?
  • Out of scope — does it decline cleanly?

A separate judge model grades every transcript against a transparent, four-criterion rubric (task completion, honesty about failure, staying in scope, avoiding hallucination). Results roll up into a public Reliability Index leaderboard, so agents are compared apples-to-apples on the same battery.

The scenario that matters most is tool failure because that's where real agents silently break. Proving Ground surfaces the exact transcript where an agent claims a refund was processed after its tool errored, and marks it a failure.

The verdict is a go/no-go signal, not just a number; it answers the question I started with: is this agent reliable enough to ship, or do I even need it in the first place?

How we built it

Built with Codex and GPT-5.6. Stack is Next.js (App Router) + TypeScript + Tailwind, using the OpenAI Agents SDK. A server-side scenario runner drives the target agent and captures each full transcript; a separate judge agent scores the transcripts and returns structured results.

Two engineering decisions I'm particularly glad I made:

A deterministic regression guard on top of the model judge. Scores don't rest solely on the judge's discretion; a false success claim on the tool-failure scenario deterministically fails, verified by automated tests. That's what makes the reliability score defensible rather than "the model felt good about it." SSRF-safe endpoint testing. When testing a live HTTP endpoint, URL validation blocks requests to private/internal addresses, so the tool can't be turned into an attack vector.

The app also degrades gracefully: bad configs, timeouts, rate limits, and provider errors are caught per-scenario, incomplete assessments are excluded from the Index, and raw provider errors are never exposed.

Challenges I ran into

The hard problem was making the reliability score rigorous instead of arbitrary. An LLM grading another LLM is only as trustworthy as the grader, so the breakthrough was pairing the model judge with a deterministic guard for the highest-stakes failure mode, and keeping the scoring rubric visible in the UI so anyone can see exactly how a verdict was reached.

Accomplishments that I am proud of

A working, non-trivial evaluation harness for an agent that tests other agents with a defensible scoring system, real error resilience, and a bring-your-own-agent flow that works on both pasted configs and live endpoints. It's a real tool, not a demo.

What I learned

That the gap between an impressive agent demo and a production-ready agent is almost entirely about failure behavior and how it acts when things go wrong, not when they go right. Building the tests taught me what "reliable" actually means.

What's next for Proving Ground

I want Proving Ground to be infrastructure for the wave of new AI builders, especially those like me coming in without an AI background. Instead of guessing whether an agent is good enough, they get an objective score that tells them exactly where it's weak, so they can spend their energy improving it rather than wondering whether it works at all. Longer term, I want to push this earlier in the journey, helping someone validate an agent idea and build the confidence to ship it, backed by a real reliability signal rather than hype.

On the product side: a public reliability index for the agent ecosystem, backed by real test runs rather than opinion, the measured research layer behind an agent-reliability publication I'm building. Plus deeper environment tests for action-taking agents (agents that operate real systems, not just respond), and custom scenario batteries per domain

Built With

  • codex
  • gpt-5.6
  • next.js
  • openai-agents-sdk
  • tailwind
  • typescript
Share this project:

Updates