TestPilot

Agent-native QA mission control that turns release risk into traceable evidence.

Comment

Inspiration

A green test dashboard does not always mean a release is safe. Important requirements can remain untested, execution evidence can be scattered across different tools, and defects can lose their connection to the tests and requirements that exposed them. QA leaders and release managers often spend more time assembling this context than making the release decision itself.

I built TestPilot to demonstrate a better model: one shared release workspace where a human can review the evidence visually while an AI agent can discover and invoke the same QA capabilities as structured WebMCP actions.

What it does

TestPilot is an agent-native QA Mission Control application for requirements, test cases, executions, defects, traceability, and release readiness.

The demonstration starts with fictional e-commerce release 2.4. All existing tests pass and there are no critical defects, but the release is still AT RISK because REQ-003 - the critical requirement preventing concurrent customers from purchasing the final inventory item - has no test coverage.

The user asks:

“Check whether version 2.4 is safe to release.”

The agent then:

  1. Inspects the release and its requirements.
  2. Detects the missing REQ-003 coverage.
  3. Creates TC-004 for two simultaneous purchases of the final item.
  4. Runs the new test.
  5. Observes both purchases succeeding and inventory reaching -1.
  6. Diagnoses a non-atomic concurrent inventory update.
  7. Creates a critical defect with structured evidence.
  8. Links the defect to TC-004 and REQ-003.
  9. Reassesses release 2.4 as NOT READY.

Every action immediately appears in the human-facing dashboard, producing a complete evidence chain from requirement to test, execution, defect, and release decision.

How I built it

TestPilot uses React, Next.js, strict TypeScript, and Vinext for the application interface and server endpoints. It is hosted publicly on OpenAI Sites, with Cloudflare Workers and Cloudflare D1 providing persistent, isolated demo state. Drizzle ORM manages the data model, Zod validates inputs, and Vitest verifies the important release logic.

The page registers 14 semantic tools through document.modelContext.registerTool(). These include get_coverage_gaps, create_test_case, run_test, inspect_failure, create_defect, link_defect_to_test, and get_release_readiness.

These are business-level QA actions, not low-level instructions such as clicking buttons. The UI and WebMCP tools use the same domain services, APIs, and persistent state, so there is no separate agent-only version of the data.

Each browser receives an isolated anonymous workspace. This allows judges to run and reset the deterministic demonstration without creating an account or interfering with another evaluation.

Challenges I ran into

The main challenge was keeping the human interface and agent workflow synchronized around one authoritative state. An agent-created test or defect needed to appear in the dashboard without duplicating business logic or maintaining a second data model.

Another challenge was making the demonstration deterministic. A real concurrency failure can be timing-dependent, which is unsuitable for a short judging session. I designed TC-004 to reproduce the negative-inventory result reliably while keeping the evidence realistic and structured.

I also needed to expose meaningful semantic tools with clear descriptions, typed inputs, validation, predictable JSON responses, and useful errors. Finally, the public application had to support anonymous access while keeping every visitor’s mutable demonstration data isolated.

Accomplishments that I'm proud of

I am proud that TestPilot demonstrates an end-to-end agent workflow rather than adding a chatbot beside an existing dashboard.

The agent can discover missing coverage, create and execute a test, diagnose a failure, create a critical defect, link the evidence, and produce a release recommendation using structured WebMCP actions. At the same time, the human sees every state transition and can inspect or reset the workflow.

The final result is a clear trace:

REQ-003 → TC-004 → failed execution → DEF-001 → NOT READY

The entire TestPilot project - including its concept, interface, application code, WebMCP integration, automated tests, documentation, and deployment - was created during the OpenAI WebMCP Challenge hackathon period.

What I learned

I learned that agent-native applications should expose intent rather than presentation details. A tool such as create_defect remains meaningful even if the website’s layout changes, while DOM-based automation can break when a button label, loading state, or responsive layout changes.

I also learned that trustworthy agent workflows require shared state, input validation, deterministic evidence, explicit error responses, and visible human oversight. A release verdict is much more valuable when the user can trace exactly how the agent reached it.

What's next for TestPilot

The current e-commerce system and test runner are intentionally simulated to provide a reliable competition demonstration.

A production version could connect the same domain and WebMCP boundaries to real CI/CD pipelines, automated test runners, requirements platforms, defect trackers, source-control checks, and deployment approval systems. Future versions could also add organization authentication, role-based access, configurable release policies, audit retention, notifications, historical risk trends, and human approval gates.

Built With

  • cloudflare-d1
  • cloudflare-workers
  • drizzle-orm
  • next.js
  • openai-sites
  • react
  • typescript
  • vinext
  • vite
  • vitest
  • webmcp
  • zod
Share this project:

Updates

Submission history