ProxyBreak

ProxyBreak uses GPT‑5.6 and deterministic simulation to expose and repair gameable KPIs before they reach the real world.

Comment

Inspiration

We framed ProxyBreak around one practical question:

Can someone follow the written rules, win the score, and still lose the mission?

Organizations often pursue broad goals through narrow measurements: tickets closed, response time, units shipped, cases reviewed, or tasks completed. Those numbers may be precise, but they are not always the outcome the organization actually cares about.

Our demonstration uses a synthetic customer-support team whose mission is to resolve customer issues correctly and promptly. Its proposed KPI awards 10 points when a ticket closes within 15 active SLA minutes, deducts only 3 points if the ticket reopens within 24 hours, and pauses the SLA clock while a ticket is marked as waiting on the customer.

That creates a gap between the mission and the measurement. A strategy can follow the written policy, receive a strong score, and still leave every customer without a solution.

We did not want to ask a chatbot, “Could this KPI be gamed?” and receive a list of hypothetical risks. We wanted to make the rules executable, let GPT-5.6 search for a concrete counterexample, and require deterministic code to prove whether the strategy actually worked.

The result is best described as unit testing for incentives:

Fuzz your KPI before it gets gamed.

What it does

ProxyBreak keeps two measurements separate:

  • The proxy: how well a strategy performs under the written KPI.
  • The mission: whether the strategy produces the outcome the organization actually wants.

The product follows one complete workflow:

  1. The user states the mission and proposed KPI.
  2. GPT-5.6 compiles the natural-language KPI into a strict, bounded policy.
  3. The user reviews and approves the interpreted rules and assumptions.
  4. ProxyBreak calculates an application-authored honest baseline.
  5. GPT-5.6 generates a declarative strategy and calls the single run_strategy tool.
  6. A deterministic simulator executes every action and calculates the proxy score and mission outcome.
  7. ProxyBreak independently reruns the complete result before accepting it as evidence.
  8. A qualifying result becomes a Verified Counterexample with its strategy, run ID, evidence events, and SHA-256 trace digest.
  9. GPT-5.6 proposes a bounded policy repair tied to the verified evidence.
  10. Application code validates and applies the repair, recalculates the honest baseline under the repaired policy, and reruns the exact same counterexample strategy and seed.

In the verified customer-support demonstration:

  • The honest baseline earns 10 proxy points, resolves 12 of 12 tickets, and produces 0 reopenings.
  • The verified counterexample earns 84 proxy points, resolves 0 of 12 tickets, and produces 12 reopenings.

The score wins. The mission loses.

GPT-5.6 then proposes requiring genuine resolution before closure receives credit. ProxyBreak applies the change and reruns the unchanged witness. Under the repaired policy, that strategy falls from 84 proxy points to −36 and no longer qualifies as a counterexample.

This proves that the specific regression witness was blocked. It does not claim that the entire policy is universally safe.

The public application offers two clearly separated paths:

  • Run Live Stress Test makes real GPT-5.6 API calls.
  • Replay Verified Run deterministically replays a previously captured and fully verified GPT-5.6 workflow.

The replay is visibly labeled as a replay and is never presented as a new live model call.

How we built it

ProxyBreak is a Next.js App Router application written in strict TypeScript and deployed on Vercel.

The GPT-5.6 runtime uses the OpenAI Responses API for three meaningful operations:

  • Natural-language policy compilation through Structured Outputs.
  • Bounded strategy generation through one strict, serial run_strategy function.
  • Evidence-supported policy-repair proposals through Structured Outputs.

GPT-5.6 proposes policies, strategies, evidence selections, and repairs. It never calculates authoritative scores or certifies its own success.

Application code owns:

  • Action legality and state transitions.
  • Active SLA time, customer wall-clock time, and genuine-work timing.
  • Proxy-score and mission-outcome calculation.
  • Qualification and every numeric delta.
  • Policy, run, and event identifiers.
  • SHA-256 trace digests.
  • Complete independent replay verification.
  • Patch validation and application.
  • Same-policy repaired baseline and unchanged-witness regression.

Strategies use a bounded declarative DSL. GPT-5.6 cannot execute arbitrary JavaScript or Python, access the shell or filesystem, call the network, or modify simulator source code.

The simulator uses integer time, stable ticket ordering, deterministic reopening behavior, canonical serialization, content-derived IDs, and rolling trace digests. Every result used as evidence is regenerated from the policy, fixed scenario, strategy, and seed and then compared across the complete run-result object.

We used Codex as an implementation partner in one primary supervised thread. We made the key product decisions first: discard the initial Python command-line prototype, focus on one customer-support arena, keep proxy score separate from mission outcome, prevent the model from verifying itself, use a bounded DSL instead of arbitrary code, and require complete deterministic replay before accepting a finding.

Codex then helped us turn those decisions into working software: scaffolding the Next.js application, implementing the simulator and strict Zod schemas, integrating the Responses API, writing and debugging tests, hardening replay and operation provenance, building the judge-facing interface, preparing the Vercel deployment, and assembling the final documentation.

We reviewed the project in explicit implementation gates and rejected or corrected work that did not satisfy the product or evidence contract. Codex accelerated the implementation, but it did not autonomously choose the product, arena, trust boundary, evaluation standard, or submission scope.

Challenges we ran into

Recovering from a false start

Our first implementation was a Python command-line prototype where GPT was optional and the scenario did not correctly demonstrate the metric–mission gap. Instead of trying to polish the wrong foundation, we stopped, preserved that prototype outside the submitted product, and restarted with a stricter contract:

The model proposes. Deterministic code verifies.

That decision shaped the rest of the architecture.

Keeping GPT-5.6 creative without making it authoritative

GPT-5.6 needed enough freedom to discover an unexpected strategy, but it could not be allowed to invent actions, scores, run IDs, or evidence.

We solved this by giving it one bounded function tool. The model can propose a StrategySpec, but application code validates the strategy, executes it, calculates the result, replay-verifies it, and decides whether it qualifies.

Preserving the Responses API tool lifecycle

The strategy search had to preserve exact call_id values, return every deterministic result as the corresponding function_call_output, and continue through previous_response_id.

The search is deliberately bounded to four strategy requests and five total Responses API calls. Tool calls are serial, and every accepted strategy passes the same strict schema and simulator.

Making the simulation reproducible

Timing, reopening, delayed rewards, settlement order, event IDs, run IDs, and trace digests all had to remain stable.

We separated strategy execution from scheduled-event settlement, used integer timing, enforced stable ordering, and made complete replay comparison part of the evidence contract. A stored result is not trusted merely because its most important measurements match; every field must match its canonical rerun.

Evaluating a repair fairly

A repaired counterexample cannot be compared with an honest baseline generated under the original policy. The repair may also change the honest strategy’s score.

ProxyBreak therefore recalculates both the honest baseline and the unchanged counterexample under the same repaired policy before deciding whether the witness was blocked.

Handling a real no-finding result honestly

Live generation is nondeterministic. In one evaluation, GPT-5.6 used all four permitted strategy attempts without producing a qualifying counterexample.

We did not hide that result or silently replace it with a successful run. ProxyBreak keeps a truthful no-finding state and separately offers a verified replay captured from an actual successful GPT-5.6 workflow.

The live path and replay path are always labeled and displayed differently.

Accomplishments that we're proud of

  • Built and deployed a complete browser experience rather than a technical-only proof of concept.
  • Produced a memorable verified result: 84 proxy points with a 0% resolution rate.
  • Replayed the counterexample using canonical event evidence and a SHA-256 trace digest.
  • Applied a bounded repair and proved that the exact same strategy and seed no longer qualify.
  • Preserved distinct provenance for policy compilation, strategy search, run_strategy calls, finding synthesis, and policy repair.
  • Passed 152 unit and mocked integration tests across 20 files.
  • Passed 2 Playwright end-to-end journeys, covering the live-shaped workflow and verified-replay path.
  • Passed ESLint, strict TypeScript checking, the Next.js production build, credential scans, and a production dependency audit with 0 vulnerabilities.
  • In the designated three-trial live evaluation, 3 of 3 trials discovered qualifying verified counterexamples.
  • Across all five observed live trials, 4 succeeded and 1 truthfully returned no finding.

We are especially proud that ProxyBreak does not hide uncertainty. “No finding under this budget” remains a valid result, and blocking one verified witness is never presented as proof that the whole policy is safe.

What we learned

Structured Outputs are only the beginning of trustworthy agentic software. A response can satisfy a JSON schema and still be semantically wrong, unsupported by evidence, or inconsistent with the application state.

We learned that a clear trust boundary matters more than a persuasive model explanation. GPT-5.6 is most useful here as a creative search and interpretation layer. Deterministic code must own execution, measurements, qualification, and proof.

We also learned that repairing a policy changes the comparison itself. Both the baseline and counterexample must be regenerated under the repaired policy.

Finally, nondeterministic AI and reproducible evidence are not opposites. GPT-5.6 can search creatively, while the resulting strategy, policy, seed, event trace, and digest become a deterministic regression artifact that can be rerun without another model request.

What's next for ProxyBreak

The submitted version deliberately focuses on one carefully authored synthetic customer-support arena so that its assumptions, transitions, and evidence remain understandable.

Possible next steps include:

  • Additional curated arenas for other incentive-design problems.
  • Counterexample minimization to identify the smallest strategy that exposes a metric–mission gap.
  • Side-by-side evaluation of multiple valid policy repairs.
  • Larger measured GPT-5.6 evaluation batches.
  • Exportable regression witnesses for policy-review workflows.
  • Versioned KPI testing so teams can rerun previously discovered witnesses whenever a metric changes.
  • Integration with policy-design and operational-review processes without exposing real employee or customer data.

The goal is not to predict how real people will behave. It is to help teams ask a better question before deploying a metric:

Can someone follow the written rules, win the score, and still lose the mission?

Built With

  • eslint
  • function-calling
  • github
  • gpt-5.6
  • next.js
  • node.js
  • openai-codex
  • openai-javascript-sdk
  • openai-responses-api
  • playwright
  • react
  • structured-outputs
  • tailwind-css
  • typescript
  • vercel
  • vitest
  • zod
Share this project:

Updates