Triage Zero

A regression test fails. QA Sentinel reads the error, logs and screenshot, decides if it's a product bug, test bug, flaky or environment, and files the defect itself or stops when it isn't sure.

Comment

Inspiration

A failed end-to-end test rarely explains what actually went wrong. Engineers must manually inspect screenshots, traces, stack traces, console errors and failed network requests before deciding whether the problem came from the application, frontend, test automation, infrastructure or an external dependency.

We created TriageZero to automate that investigation. Our goal was to build an autonomous system that turns fragmented Playwright failure evidence into a structured, auditable and actionable diagnosis without waiting for a person to gather and interpret every artifact.

What it does

TriageZero is an autonomous AI investigator for failed browser tests.

Every 30 minutes, Google Cloud Scheduler launches a Cloud Run job that runs a real Playwright test suite against NovaCart, our deployed demonstration storefront. When a test fails, the reporter automatically captures and sanitizes the available evidence, including:

The reporter automatically captures:

  • Expected and actual results
  • Failure message and stack trace
  • Browser console errors
  • Failed network requests
  • Screenshot and Playwright trace references
  • Browser, environment and test metadata

The evidence is submitted automatically to TriageZero as a strictly validated failure package. A Google ADK agent then uses Gemini through Vertex AI to classify the failure, identify its likely root cause, estimate confidence and recommend the next action.

Release risk is calculated separately using deterministic policy. The completed investigation is stored in Cloud SQL and displayed in a Firebase-authenticated dashboard where a reviewer can inspect the evidence, approve or reject the recommendation, or record a corrected outcome.

Human-reviewed outcomes can become labeled precedents for future investigations, creating a feedback loop without exposing private expected answers to the AI.

How we built it

We built TriageZero as two connected systems: NovaCart, a deployed application-under-test with Playwright automation, and TriageZero, an AI investigation platform that receives and analyzes test failures.

Google Cloud Scheduler launches a dedicated Cloud Run job every 30 minutes. This job runs a real Chromium browser through Playwright against the deployed NovaCart storefront. When a test fails, the reporter captures the expected and actual results, failure message, stack trace, console errors, failed network requests and artifact references. It sanitizes this evidence and submits it to the TriageZero API using a dedicated machine identity and the closed failure-package v1.0 contract.

The FastAPI backend authenticates the request, validates the package, prevents duplicate investigations and searches for relevant human-reviewed precedents. A Google ADK agent then uses Gemini through Vertex AI to inspect the sanitized evidence with five bounded, read-only tools. The AI response is validated against a strict schema, while release risk is calculated separately using deterministic application policy.

Completed investigations are stored in Cloud SQL for PostgreSQL 16 and displayed in a React and TypeScript dashboard protected by Firebase Authentication. Cloud Run hosts the NovaCart frontend and API, the TriageZero frontend and API, and the scheduled Playwright workload. Secret Manager and Google Cloud service accounts protect production credentials.

The primary production analyzer is gemini_adk, implemented with a real ADK Agent, Runner and session service. If that path cannot complete, TriageZero can gracefully fall back to direct Gemini and then to a deterministic evidence-based analyzer. Every investigation records its provider, model, token usage and fallback status so the result remains auditable.

Challenges we ran into

One of our biggest challenges was combining ADK tool use with strict structured output. When tools and an output schema were bound simultaneously, the model could continue producing function calls without returning a final verdict. We solved this by limiting the number of agent events, allowing each read-only tool to run at most once, requiring a final JSON response and validating that response inside the application before storing it.

We also encountered an asynchronous session lifecycle issue. Reusing an ADK session across closed event loops could leave investigations stuck in the analyzing state. Rebuilding the ADK Runner and session service for each investigation removed the deadlock and made repeated cloud executions reliable.

Another challenge was ensuring that controlled defect labels and expected answers never reached Gemini. We created a strict separation between submitted evidence and private evaluation metadata, added forbidden-field detection and rejected any package that attempted to include oracle information.

We also had to handle duplicate and concurrent uploads safely. Evidence fingerprints, idempotency keys and database uniqueness constraints ensure that retries do not create duplicate investigations, while conflicting evidence receives a clear error.

Finally, we needed to balance autonomy with safety. The evidence collection, AI analysis and recommendation pipeline runs unattended, but release risk is calculated through deterministic policy and consequential actions still require human approval.

Accomplishments that we're proud of

We are proud that TriageZero is a complete working production pipeline rather than a static prototype. A scheduled Cloud Run job launches a real browser, tests a deployed storefront, detects failures, captures evidence, submits it automatically and creates an investigation without requiring a person to start the process.

We successfully integrated Gemini through Vertex AI with a real Google ADK Agent and Runner. Each completed investigation records the provider, model, token usage, confidence and fallback status, making the AI result transparent and auditable.

We built strong safety boundaries around the agent. It receives only sanitized evidence, uses five bounded read-only tools and cannot access the source repository, filesystem, shell or arbitrary external networks. Private evaluation answers remain separate from the evidence supplied to Gemini.

We also implemented durable Cloud SQL storage, Firebase-authenticated reviewer access, idempotent ingestion, deterministic release-risk policy, graceful fallback behavior and a human feedback loop that can turn reviewed outcomes into future precedents.

Finally, we created six controlled defect scenarios covering different failure categories, along with automated frontend, backend, safety and evaluation tests. This gave us a repeatable way to demonstrate the system and verify that new changes do not silently break the investigation pipeline.

What we learned

We learned that autonomy does not require removing people from every decision. Evidence collection, validation, investigation and recommendation can run unattended, while high-impact remediation can remain protected by human approval.

We also learned that trustworthy AI evaluation requires a strict separation between the evidence shown to the model and the private answer used to evaluate it. If an expected classification or controlled-defect label reaches the model, the resulting accuracy measurement is no longer meaningful. TriageZero therefore stores private evaluation metadata separately and rejects any failure package containing oracle fields.

Another important lesson was that generative reasoning and deterministic policy work better together. Gemini is effective at interpreting fragmented evidence and explaining a likely cause, while deterministic application rules provide a stable boundary for release-risk decisions.

Building with Google ADK also taught us to place clear limits around agent behavior. Bounded tools, event limits, strict result validation and explicit fallback paths made the agent more reliable and auditable than an unrestricted prompt-and-response workflow.

Finally, we learned that controlled evaluation results are useful for regression testing but are not the same as universal real-world accuracy. Meaningful long-term accuracy must be measured against human-reviewed outcomes collected across different applications, teams and failure categories.

What's next for Triage Zero

Our next priority is to expand TriageZero beyond the controlled NovaCart demonstration by testing it against additional applications, repositories and failure categories. This will help us build a larger human-reviewed historical corpus and measure classification accuracy, confidence calibration and release-risk quality across real engineering environments.

We plan to add richer artifact storage for screenshots, videos and Playwright traces, along with optional integrations for GitHub issues, pull requests and enterprise observability platforms. These integrations will remain approval-gated so TriageZero can recommend actions without making unsafe changes automatically.

We also want to introduce organization workspaces, role-based access control, investigation assignment and team-level accuracy reporting. Over time, reviewed resolutions could help TriageZero retrieve increasingly relevant precedents for each application while keeping one customer's evidence isolated from another.

Our long-term goal is for TriageZero to become an autonomous reliability layer between continuous testing and engineering teams: detecting failures, gathering evidence, identifying likely causes and presenting the safest next action before an engineer begins manual investigation.

Built With

Share this project:

Updates

Submission history