Watchman — Guardian of FIFA 2026

Powered by Google Cloud Agent Builder, GiLAB and Arize, Watchman is an autonomous SRE agent that safeguards the FIFA 2026 ticketing infrastructure using strict authority zones.

Comment

Imagine the final match of the FIFA 2026 World Cup. Millions of fans worldwide are simultaneously trying to purchase tickets. In this high-stakes environment, infrastructure failure is not an option; a few minutes of downtime would cost millions in revenue and create global chaos.

To meet this challenge, we needed to design a system capable of maintaining a strict Service Level Agreement target (SLA≥99.99%). This means our allowed downtime, calculated by Error Budget=1−Availability, is nearly zero. Traditional static alerts and manual SRE (Site Reliability Engineering) interventions are simply too slow. We were inspired to build Watchman: an autonomous AI agent that doesn't just answer questions, but actively monitors, diagnoses, and safely heals production infrastructure in real-time. ⚙️ What it does

Watchman is an autonomous SRE agent acting as the Guardian of the FIFA 2026 Ticketing platform. Instead of giving an AI direct, dangerous access to edit production databases or Kubernetes clusters, Watchman operates on a strict "3 Authority Zones" model: 

🟢 GREEN Zone (Auto-Remediation): For clear, low-impact resource issues (e.g., Pod RAM > 95%). Watchman autonomously triggers a GitLab CI/CD pipeline to scale the infrastructure.  

🟠 ORANGE Zone (Action + Alert): For API degradation or minor connection limits. Watchman attempts a targeted fix and mandates engineering supervision.  

🔴 RED Zone (Strict Block): For critical incidents (e.g., Cloud SQL database failure). Watchman explicitly blocks any automated action, generates a critical incident report, and demands human validation. This ensures the AI never accidentally wipes production data.

🏗️ How we built it

We architected Watchman using a "Zero-Direct-Touch K8s" philosophy. The AI acts as the "Brain," while a secure CI/CD pipeline acts as the "Muscle". 

The Brain (Google Cloud Agent Builder): We built the core agent using Google Cloud Agent Builder, instructed with a strict system prompt to enforce the 3 Authority Zones.  

The Tools (FastMCP on Cloud Run): We developed a Python MCP (Model Context Protocol) server hosted on Google Cloud Run. It exposes specific tools to the agent: read_cluster_state, check_db_health, get_pod_metrics, and trigger_gitlab_repair.  

The Muscle (GitLab CI/CD): When Watchman decides to act, it triggers a GitLab pipeline via API. A private GCE Debian 12 runner safely applies the idempotent Terraform and Kubectl changes.  

The Infrastructure (GCP): The ticketing application (FastAPI + React) runs on a GKE cluster (watchman-gke), connected to a highly available Cloud SQL MySQL 8.0 database (fifa_tickets).  

The Partners (Observability):

    Dynatrace: We deployed the Dynatrace Operator (OneAgent) on our GKE cluster to monitor CPU, RAM, and Pod health, providing the raw telemetry our agent relies on.  

    Arize Phoenix: To ensure the AI's decisions are transparent, we integrated OpenTelemetry. Every time Watchman calls an MCP tool or classifies a zone, a trace is pushed to the Arize Phoenix collector, enabling a full forensic LLMOps audit.

🚧 Challenges we faced

AI Safety in Production: The biggest challenge was preventing the LLM from hallucinating destructive commands (like terraform destroy). We solved this by removing Kubernetes write-access from the agent entirely. It can only trigger pre-approved GitLab pipelines.  

Complex Tool Orchestration: Chaining multiple MCP tools so the agent reliably checks cluster state before taking action required extensive prompt engineering and strict routing logic.  

Cloud Costs (FinOps): Leaving a GKE cluster, Cloud SQL, and VMs running 24/7 is expensive. We wrote aggressive Bash scripts (stop_watchman.sh and start_watchman.sh) to resize node pools to 0 and halt databases during idle periods, bringing passive costs down by 97% (from ~$246/mo to ~$7/mo).

🏆 Accomplishments that we're proud of

Successfully building a production-ready GitOps workflow where an AI safely collaborates with a CI/CD pipeline.  

Integrating Arize Phoenix to create an immutable audit log of the AI's internal reasoning and actions.  

Designing a fully automated deployment (Zero-Manual-Touch) where one Terraform command spins up the VPC, GKE, Cloud SQL, and GitLab runners seamlessly.

📚 What we learned

We learned the immense power of the Model Context Protocol (MCP). Standardizing how LLMs interact with external systems allowed us to easily plug our agent into GKE, Cloud SQL, and GitLab.

We deepened our understanding of LLMOps. Treating an agent like a traditional software component—requiring tracing, error handling, and performance monitoring via tools like Arize—is mandatory for enterprise adoption.

🚀 What's next for Watchman

In the future, we plan to expand Watchman's capabilities by integrating it directly into Slack or Microsoft Teams. This will allow the SRE team to receive the agent's RED Zone alerts on their phones and grant human validation with a single click, instantly unpausing the CI/CD remediation pipeline.

Built With

Share this project:

Updates