• 

  Which AI works for this job? Compare a single LLM, retrieval RAG, and a read-only tool agent under the same rules.

• 

  One live synthetic ticket, three real OpenAI-backed runs, with hard gates, cost, latency, and observed calls kept separate.

• 

  Inspect the actual reply, action, policy citation, hard-gate result, cost, and latency behind each comparison cell.

• 

  GPT-5.6 reviews blinded X/Y/Z evidence for advisory risk signals; deterministic gates remain authoritative.

• 

  A human selects the simplest sufficient candidate, then GPT-5.6 writes a Decision Memo from the active evidence.

• 

  A defective change triggers a new hard-gate failure and BLOCK without deploying or rolling back any external system.

AI Challenge Room

Inspiration

AI Challenge Room began with a simple question:

What if a company could turn a real business problem into a private challenge and compare very different AI approaches under the same rules?

Public model rankings cannot answer which configuration will work best for a company's own documents, policies, tools, failure conditions, and budget. A single language model may be sufficient. Another task may require retrieval or a tool-using agent. Today, teams often discover the difference through separate proofs of concept that were not designed to be compared.

We initially focused on finding more possible solutions. We soon learned that discovery was only part of the problem. A company also needs a trustworthy way to define success, reject critical failures, compare quality against cost and complexity, and keep the final decision human-owned.

That insight became AI Challenge Room: a decision workspace for testing different AI configurations on the same business task and selecting the simplest option that is sufficient.

What it does

For OpenAI Build Week, we implemented one bounded, end-to-end private challenge around a synthetic customer-support task.

The browser sends the same locked ticket to three candidates:

• Candidate A: a single LLM with a basic prompt
• Candidate B: an LLM with policy-retrieval RAG
• Candidate C: a bounded, read-only agent that can search policy and look up the synthetic order

AI Challenge Room then guides the user through one evidence-backed decision:

1. Run all three candidates under the same task boundary.
2. Apply deterministic policy hard gates before comparing averages.
3. Inspect candidate outputs, citations, tool evidence, cost, and latency.
4. Ask GPT-5.6 for advisory risk signals using blinded X/Y/Z evidence.
5. Complete a blind human review.
6. Let a human select an eligible candidate and record the rationale.
7. Generate a GPT-5.6 Decision Memo from the active evaluation evidence.
8. Replay a representative defective change and show a deterministic BLOCK.

The most complex candidate does not automatically win. The objective is to find the simplest candidate that is sufficient for the locked task. If every candidate fails, NO_APPROVED_CANDIDATE is a valid result.

How we built it

The product is a TypeScript and React decision workspace backed by a hosted Cloudflare Worker API.

We created one shared execution contract and three candidate adapters so that the single-LLM, RAG, and read-only tool-agent configurations receive the same locked synthetic case. OpenAI Responses API calls power the candidate runs, the blinded auxiliary Judge, and the Decision Memo.

The hosted path stores session and workflow state in D1 and content-addressed evaluation artifacts in R2. A server-verified access gate, bounded run counts, duplicate-execution protection, retry accounting, and cleanup receipts protect the live judging path.

The evaluation architecture deliberately separates three kinds of authority:

• Deterministic hard gates enforce explicit policy failures.
• GPT-5.6 auxiliary review identifies risks but cannot clear a failed gate or select a candidate.
• Human review and selection remain the final decision authority.

That separation is reflected in the interface: fatal failures appear before quality trade-offs, candidate identities are blinded during review, and the Decision Memo explains a human decision instead of making one.

Challenges we ran into

Preserving evidence provenance

The application supports both live execution and a recorded synthetic fallback. The difficult part was not displaying both; it was making silent substitution impossible. We added explicit source labels, locked each demo session to one evidence source, prevented live and recorded metrics from being mixed, and required the user to choose the fallback manually.

Keeping evaluation authority clear

An LLM can produce a convincing qualitative assessment that conflicts with an explicit policy rule. We had to make the hierarchy unambiguous in both code and UI: deterministic failures remain authoritative, GPT-5.6 signals are advisory, and the human owns the final selection. A NO RISK Judge result is therefore never presented as a hard-gate pass.

Making long-running external work understandable

Candidate runs, retrieval, tool calls, auxiliary review, and Memo generation can take time. Early versions did not always make that waiting state clear. We added persistent progress stages, elapsed time, candidate completion counts, retry and cleanup states, disabled duplicate actions, and authoritative state restoration after a reload.

Moving from a local runner to a hosted demo

The first runnable path depended on a local Node server, process memory, and POSIX files. The judging experience needed a browser-accessible URL without exposing secrets or private artifacts. We separated persistence behind interfaces, moved hosted state to D1 and artifacts to R2, kept OpenAI credentials server-side, and restricted every state-changing or billable API to an authenticated judge session.

Learning from real provider failures

Live runs exposed issues that fixtures alone had not revealed: evaluator edge cases, persistence and hydration defects, provider limits, cleanup behavior, authentication differences, and inaccessible UI states. We classified each failure as a platform defect, candidate failure, or external operating issue, then used Codex to turn reproducible defects into fake-provider and workflow tests instead of modifying candidate results to force a successful demo.

Accomplishments that we're proud of

• A working browser flow from a locked task through live A/B/C execution, evidence inspection, blinded review, human selection, Decision Memo, and defective-change BLOCK
• Actual cost, latency, retry, retrieval, and tool-call evidence owned by the runner rather than self-reported by a candidate
• A design that treats critical failure and NO_APPROVED_CANDIDATE as honest outcomes instead of forcing a winner
• A live/recorded boundary that never disguises fallback evidence as a live run
• A complete enterprise-style decision experience rather than a model-score leaderboard

What we learned

More complex AI is not automatically better. A tool agent can provide useful information access, but it also adds calls, latency, cost, and more failure surfaces. Complexity should be justified by task evidence.

Fair comparison starts with the task contract. Adding more candidates does not create a fair evaluation unless the input, constraints, fatal failures, and evidence boundary are fixed first.

Rules, models, and people have different jobs. Deterministic checks are best for explicit policy constraints. GPT-5.6 is useful for finding qualitative risks that deserve attention. Human judgment is still necessary for open-ended quality and the final business trade-off.

Failure is useful evidence. A failed candidate, a provider error, or a no-approved-candidate result should be classified and preserved, not hidden.

Operational evidence is part of the product. Progress, provenance, cost, retries, cleanup acknowledgements, and state restoration are not backend details when a team must defend an AI adoption decision.

Codex is especially effective when paired with tests. Converting defects from real API runs into repeatable tests let us improve a complex workflow without losing the boundaries that make its evidence credible.

What is live and what is recorded

The evidence boundary is explicit.

During a LIVE comparison:

• the browser starts one OpenAI-backed run for each candidate;
• Candidate B performs policy retrieval;
• Candidate C performs policy retrieval and read-only order lookup;
• cost, latency, retries, search, and tool evidence come from that execution;
• the GPT-5.6 auxiliary Judge is a live call; and
• the GPT-5.6 Decision Memo is a live call.

Because each candidate runs once in the hackathon walkthrough, repeat stability is labeled Single run · not measured.

The existing synthetic recorded run remains available only through the user-selected RECORDED FALLBACK. It is never selected automatically and is never mixed with live candidate evidence. Judge and Memo call states identify whether their external calls are running, complete, or failed.

This is a one-ticket synthetic demonstration. It is not presented as a hidden 72-run Benchmark, production approval, automatic purchase, or real enterprise deployment.

How we used GPT-5.6

GPT-5.6 powers the candidate responses, helps structure evaluation evidence, produces blinded auxiliary risk signals, classifies failure types, and writes the final Decision Memo.

It is deliberately not the sole evaluator. Deterministic code checks explicit policy rules, and a human makes the final decision. GPT-5.6 cannot erase a failed rule or select a winner by itself.

How Codex accelerated the project

Codex helped turn a detailed product requirement into a working TypeScript application. It accelerated:

• the shared candidate execution contract and A/B/C adapters;
• Responses API, retrieval, and read-only tool integration;
• usage, cost, latency, retry, and cleanup evidence;
• deterministic hard gates and blinded review boundaries;
• the Worker, D1, and R2 hosted architecture;
• the server-verified judge access gate;
• accessible loading, evidence, review, and decision states; and
• the automated test suite.

Codex also helped us investigate failures across the complete workflow, distinguish code defects from candidate behavior and provider limits, and preserve those findings as repeatable tests.

What's next

The hackathon demo proves one private, end-to-end AI selection workflow.

The next step is to support more task-specific private challenges and reuse each accepted evaluation as an ongoing check when models, prompts, company knowledge, or workflows change.

Data and safety boundaries

• All tickets, policies, and order records are synthetic.
• OpenAI requests use store: false; this is not a Zero Data Retention claim.
• Temporary Vector Store and uploaded-file cleanup responses are recorded as API acknowledgements, not proof of physical erasure.
• Live usage is bounded by server-side session, concurrency, run-count, retry, and cost limits.
• The demo does not purchase, contract, deploy, roll back, or modify an external system.

Try the demo

Live demo: https://ai-challenge-room.aside-hazle.chatgpt.site

The judge access code and testing steps are provided only in Devpost's private testing field.

Built With

• cloudflare-workers
• codex
• d1
• gpt-5.6
• openai
• openai-responses-api
• openai-retrieval-api
• openai-sites
• r2
• react
• typescript
• vitest