AutoPlay QA

Evidence workspace: a saved Unity floor-support session with recorded controls and a cited Nemotron Ultra review

Inspiration

A useful game tester needs to do more than produce an interesting playthrough. Developers need to know what happened, which controls caused it, and whether they can reproduce it in their own runtime.

AutoPlay QA brings those needs into the design of an AI testing tool. Our vision is a testing partner that observes a game, asks useful questions, investigates unexpected behavior, and leaves an evidence trail that another person can inspect. We are building the open source foundation for that workflow, so the community can improve the agents and evaluate them against the same recorded facts.

What it does

AutoPlay QA connects a game runtime to a bounded AI exploration loop. NVIDIA Nemotron chooses controls from the game's exposed interface and receives the resulting public state and feedback. The host records accepted actions and their outcomes, then makes that evidence available for replay, reporting, and review.

A separate Nemotron Ultra review step reads candidate evidence and produces a structured assessment: a tentative verdict, severity, player impact, citations, and limitations. Developers can follow those citations back to the actual recorded transitions and inspect the reproduction evidence alongside the assessment.

The public dashboard brings saved sessions, playback controls, candidate reports, and downloadable evidence into one place. Its owner controls connect to a local runner for bounded tests while the runner is online. Saved evidence remains available for public inspection independently of a live session.

The project includes a terminal Grid test bed, a real Unity physics integration, and adapters for the Outpost: Mars interpreter and browser editor. These exercise different kinds of state and controls through a shared testing core. The public showcase currently presents curated Grid and Unity sessions; the additional adapters and research protocols live in the repository.

What makes it different

AutoPlay QA makes the testing process inspectable from model decision to game result.

A concrete execution contract. The model chooses from exposed controls; the host validates and executes them, records the acknowledgment, and preserves the resulting feedback.
Separate claims and observations. A model's suspicion, a software signal, a repeated behavior, and a developer's judgment have distinct roles in the evidence.
Replay in the original runtime. Saved controls can be checked against actual game behavior, with the scope and result of that check recorded.
A place for conventional automation. Scripted coverage sweeps and model exploration can contribute to the same evidence pipeline, with their sources identified.
Explicit resource limits. Action, request, time, and estimated cost limits make individual sessions bounded and auditable.
An extensible research surface. The repository includes protocols for bounded context, observed experience, predictions, investigations, and controlled comparisons, so new agent designs can be evaluated using concrete outcomes.

These are the capabilities we add around a model when asking it to test a game. They give developers a practical way to inspect, reproduce, and improve AI testing behavior.

How we built it

The Python host connects to adapters through an observation/action boundary. A versioned local protocol carries that boundary into other runtimes, including a Unity package. The browser adapter exposes ordinary visible controls such as click, input, select, and wait. Session recordings retain the action and feedback sequence; offline tools handle replay, report generation, reproduction checks, and evaluation.

Nebius Token Factory serves two NVIDIA Nemotron roles. nvidia/Nemotron-3_5-Lightning is the default exploration model. nvidia/Nemotron-3-Ultra-550b-a55b reviews recorded candidate evidence and is the designated actor in the experimental semantic testing protocol. The application supplies public observations and controls, while developer rules and private evaluation labels remain outside model inputs.

The dashboard uses React and Vinext with a Cloudflare Worker and D1 task service. The local runner executes game sessions and publishes sanitized progress and evidence. The provider key stays with the runner.

Engineering decisions we are proud of

We kept the game interface small while preserving detailed execution evidence. Existing recordings and protocols remain replayable as the research modules evolve. The testing workflow has both scripted and model-driven paths, and the same reporting tools can inspect their results.

We also built the research tools around questions a developer can evaluate: Did an action execute? Did the observed result recur? Was a prediction written before the experiment? Does an experience record belong to this game build? Those distinctions make the project useful as a shared foundation for future work.

What is next

We want the open source community to help develop more capable experiment selection, more useful experience retrieval, clearer developer review, and additional game adapters. The research modules already provide a place to work on hypothesis-driven testing and executable regression export. Their effectiveness remains a question for controlled evaluation.

Longer term, our vision is an agent that carries justified knowledge across builds and games, asks targeted questions, and produces regression cases from independently confirmed findings. Cross-game transfer is a future research goal. The shared evidence format and runtime adapters are the foundation we are contributing today.