QA Rehearsal Theater

Evidence-backed Android QA where AI personas explore real apps and turn UI states, recordings, and logs into reproducible bugs.

Comment

QA Rehearsal Theater

Inspiration

Scripted mobile tests are good at repeating known checks. They are weaker at finding the odd sequence a real user might take.

QA Rehearsal Theater treats exploratory QA like a rehearsal: give different user types a mission, let them use a real Android app, and keep the record of what happened. A useful finding is not “the screen looked wrong.” It is a claim tied to an action, a timestamp, and evidence someone else can inspect.

What it does

QA Rehearsal Theater runs exploratory QA against real Android APKs on local emulators.

Teams can:

  • Upload an APK and define a QA mission.
  • Cast personas with different goals, habits, and risk sensitivities.
  • Run several persona sessions across local Android emulators.
  • Record screenshots, video, UIAutomator trees, foreground state, actions, and logcat.
  • Detect target-app crashes independently of the language model.
  • Track the requested provider, reported model, latency, and fallback path for each decision.
  • Generate findings with reproduction steps and links to the supporting evidence.

For Build Week, we added an evidence-oriented decision contract for GPT‑5.6. A decision may include:

  • a failure hypothesis;
  • the signal that would confirm it;
  • a bounded confirmation action;
  • a confirmed, rejected, or inconclusive verdict; and
  • the timeline steps supporting that verdict.

The runtime rejects invalid node IDs, unsafe values, invented evidence references, arbitrary shell access, and raw ADB commands before they become actions or findings.

How we built it

The product is a local-first execution runtime with a desktop dashboard:

  • FastAPI manages APKs, missions, personas, runs, routing, and evidence.
  • ADB and UIAutomator operate APKs on Android emulators.
  • React, Vite, and TanStack Router power the dashboard.
  • SQLite is the canonical local store.
  • Tauri packages the dashboard and backend for macOS.
  • GPT‑5.6 structured outputs produce bounded QA decisions.

We used GPT‑5.6 through Codex CLI in warm app-server sessions. Sol, Terra, and Luna received the same 24-screen Android golden set and the same structured JSON action contract. We compared schema validity, safety, type match, exact match, and p50/p90 latency.

Codex was used as an engineering partner, not just a code generator. It helped us inspect the existing runtime, define the Build Week delta, implement the evidence contract, add tests, build provider routing and the benchmark scoreboard, create cold-start demo data, and review failure states.

The model sees a bounded observation: mission, persona, current UI state, recent actions, and stable IDs for visible elements. It can choose only validated actions such as tap, type, swipe, back, or wait. It does not receive unrestricted device, shell, or filesystem access.

Challenges we ran into

The difficult part was turning a plausible explanation into a defensible finding.

For every report, we had to answer:

  • Did the problem actually occur?
  • Which action triggered it?
  • Was the target app still in the foreground?
  • Does the video timestamp match the recorded step?
  • Which UI dump, screenshot, or log supports the claim?
  • Did the requested model make the decision, or did routing fall back?

That led to two layers: a model that explores and forms hypotheses, and a deterministic runtime that validates actions, detects crashes, preserves evidence, and rejects unsupported conclusions.

We also had to handle emulator scheduling, UI synchronization, stale nodes, external login flows, and provider failures without hiding what went wrong.

Accomplishments that we’re proud of

The demo runs against a real Android application, not a mocked interface.

  • Crash detection and evidence capture continue independently of the provider.
  • Persona and run snapshots remain immutable after execution.
  • Multiple emulator workers keep isolated evidence directories.
  • Each action records provider, model, latency, and fallback provenance.
  • Findings link video, screenshots, UI dumps, logs, and reproduction steps.
  • The decision contract cannot cite a timeline step that does not exist.
  • Local QA and historical evidence remain available when remote services are down.
  • The cold-start demo seeds Sample Pay, personas, scenarios, and bilingual collections.

In the demo benchmark, the Luna warm route completed all 24 screens with 100% schema validity, 100% safety, and no execution errors.

The point is simple: the system does not stop at “AI found a bug.” It records what happened, what caused it, and how to reproduce it.

What we learned

Trust came from constraints, not from longer model explanations.

The model is useful for exploring unfamiliar paths and forming hypotheses. The runtime is responsible for deciding what actions are allowed, what evidence exists, and what the report can claim.

Model provenance also belongs in the product. When users can see the requested model, actual model, latency, and fallback path, they can judge a result instead of treating it as an opaque AI answer.

Most importantly, a bug report is valuable only when another person can reproduce it.

What’s next

We plan to extend the current evidence loop with:

  • richer hypothesis confirmation across multiple actions;
  • clearer handling for findings that need human confirmation;
  • compact proof cards for GitHub and issue trackers;
  • regression comparison across APK versions;
  • a larger fixed Demo QA Pack; and
  • human review workflows for promoting verified findings into reusable QA knowledge.

The long-term goal is a reviewable rehearsal of real user behavior, not another report full of unverified AI prose.

Built With

Share this project:

Updates