Aegis — Autonomous Resilience Game-Day Agent
Elevator Pitch
A Gemini-powered SRE agent on Google Cloud that safely runs resilience game days, auto-aborts on SLO burn, generates hardening follow-ups, and verifies fixes with Dynatrace MCP.
Inspiration
I built Aegis for the Google Cloud Rapid Agent Hackathon to solve a real SRE problem: teams can detect incidents, but they often struggle to safely prove resilience risk and verify whether a fix actually worked.
In many organizations, game-day and chaos workflows are fragmented. Observability lives in one tool, deployments live somewhere else, code changes happen in GitHub, and the final decision often depends on human memory, screenshots, or chat messages.
I wanted to build an agent that connects those steps into one supervised, auditable workflow.
That became Aegis: an autonomous resilience game-day agent for SRE and DevOps teams, built on Google Cloud with Gemini and verified through Dynatrace MCP.
The core idea is simple:
Gemini plans. Dynatrace verifies. Python enforces safety.
What it does
Aegis runs a complete closed-loop resilience game day for a payment service dependency.
The workflow is:
Rank → Approve → Inject → Auto-abort → Harden → Verify → Report
Aegis first ranks risky service dependencies using live signals such as bad ratio, p95 latency, and hardening status. In the demo, it identifies the payment → store dependency as vulnerable to latency amplification.
Before making any mutation, Aegis asks for human approval.
After approval, it injects a controlled dependency failure by adding latency to the selected dependency. The burn-rate signal starts to rise in real time.
The most important part is the safety brake: Aegis does not ask the LLM whether the system is safe. A deterministic Python loop watches the numeric burn score and automatically aborts when the threshold reaches 10.
After the experiment, Aegis generates a hardening follow-up for the payment dependency client. The proposed fix adds timeout and retry behavior to reduce the impact of dependency latency.
Aegis then verifies the result with Dynatrace MCP output and produces a notebook-style report with evidence from the experiment.
The demo also includes an honest fail scenario. If the failure mode is not solved by the hardening strategy, Aegis reports that instead of pretending the fix worked.
How I built it
I built Aegis using:
- Gemini via Google ADK and Vertex AI for workflow reasoning and tool selection
- Google Cloud Run for the hosted dashboard and demo services
- Dynatrace MCP for live observability checks and verification
- OpenTelemetry for service telemetry
- FastAPI for the dashboard and backend APIs
- GitHub integration for hardening follow-ups
- Google Secret Manager for secure runtime configuration
- Python for the deterministic inject → poll → abort safety loop
A key design choice was separating agent intelligence from safety enforcement.
Gemini can reason about the workflow, select tools, generate explanations, and write the report. But the safety-critical abort logic is deterministic Python code with a numeric threshold. The LLM never decides whether the experiment is safe enough to continue.
That boundary makes Aegis useful without making it unsafe.
Challenges I ran into
The hardest challenge was making Aegis genuinely agentic without giving the LLM unsafe control over operational decisions.
I wanted the system to do more than explain what an SRE should do. I wanted it to perform a real game-day workflow: choose a dependency, ask for approval, inject a fault, watch burn rate, abort when the threshold is crossed, generate a hardening artifact, and verify the result.
At the same time, I did not want the LLM to be responsible for the safety brake.
That forced a clear architectural boundary:
Gemini plans the workflow. Python enforces the safety threshold. Dynatrace verifies the evidence.
Another challenge was making the demo honest. In one GitHub hardening flow, Aegis correctly surfaced a GitHub permission issue: the fine-grained token needed Contents: write and Pull requests: write permission. Instead of hiding that failure or pretending the PR succeeded, the system reported the error clearly.
That behavior is important for operational agents. In real SRE workflows, a trustworthy agent should expose partial success, missing permissions, and failed actions instead of hallucinating success.
Accomplishments that I’m proud of
I am proud that Aegis demonstrates a full closed-loop SRE workflow instead of only showing a chatbot.
It connects agent reasoning, human approval, controlled failure injection, deterministic safety enforcement, Dynatrace MCP verification, GitHub hardening follow-up, and notebook-style reporting into one workflow.
I am also proud of the safety model. Aegis shows how AI agents can participate in operational workflows without giving the LLM unchecked authority over production-style decisions.
The agent is useful because Gemini can reason, coordinate tools, and explain outcomes. The system is safe because the critical abort decision is deterministic, auditable, and threshold-based.
What I learned
I learned that the most useful operational agents are not fully autonomous. They are supervised systems with clear control boundaries, deterministic safeguards, real tools, and auditable behavior.
I also learned that observability becomes much more powerful when it becomes part of a closed loop:
detect risk → approve → test safely → auto-abort → harden → verify → report
That pattern can make resilience engineering more repeatable, measurable, and trustworthy for real SRE and DevOps teams.
What’s next
Next, I would expand Aegis with broader dependency analysis, richer remediation strategies, stronger grounding evaluations, more Dynatrace notebook evidence, and deeper production controls.
I would also extend the game-day patterns to more failure modes beyond dependency latency, including dependency errors, partial outages, saturation, and regional degradation.
The long-term goal is to make resilience engineering safer, faster, and evidence-driven for real teams.
SignalPath — Agentic Observability Onboarding for Dynatrace on Google Cloud
Elevator Pitch
SignalPath is a Gemini-powered observability onboarding agent that helps SRE and DevOps teams configure, validate, and troubleshoot Dynatrace onboarding on Google Cloud through a guided, self-verifying workflow.
Inspiration
I built SignalPath to solve one of the most painful problems in observability adoption: onboarding.
In real SRE and DevOps teams, getting an application fully onboarded into monitoring can take days or even weeks. Engineers often need to configure cloud services, enable APIs, manage secrets, wire telemetry, validate permissions, check whether data is flowing, troubleshoot partial failures, and repeat the process across different environments.
The problem is not only technical setup. The problem is trust.
Operators need to know:
Am I targeting the right cloud project? Are the required APIs enabled? Is the telemetry token stored securely? Is Cloud Run configured correctly? Is Dynatrace receiving data? Did onboarding succeed, or did it only partially succeed? What should I verify next?
I wanted to build an agent that could guide this workflow, perform deterministic setup steps, validate the result with Dynatrace MCP, and explain the onboarding state clearly.
That became SignalPath: an agent-assisted observability onboarding workflow for Dynatrace on Google Cloud.
What it does
SignalPath turns Dynatrace onboarding into a guided operational workflow instead of a manual checklist.
The agent helps onboard a Google Cloud Run service into Dynatrace by walking through the setup and validation flow.
It can:
- identify the target Google Cloud project and region
- enable required Google Cloud APIs
- store the Dynatrace OTLP token securely in Google Secret Manager
- configure Cloud Run telemetry from the stored secret
- wire OTLP telemetry with delta metrics
- bridge Cloud Run platform metrics into Dynatrace without requiring Helm or GKE
- verify readiness using Dynatrace MCP
- detect setup issues such as disabled APIs
- auto-remediate supported setup failures and retry
- provide post-onboarding validation checks for services, logs, GCP signals, GKE signals, and Davis problems
The goal is not to replace SREs or DevOps engineers. The goal is to remove repetitive onboarding friction so teams can reach observability value faster and with fewer setup mistakes.
How I built it
I built SignalPath using Gemini through Google ADK and Vertex AI, Google Cloud Run, Dynatrace MCP, FastAPI, Google Secret Manager, Google Cloud project-aware logic, and curated Dynatrace onboarding knowledge.
Gemini drives the workflow reasoning and helps guide the operator through the onboarding process.
Google Cloud provides the runtime and infrastructure integration layer.
Google Secret Manager stores sensitive Dynatrace tokens securely instead of placing secrets in source code.
Cloud Run hosts the service and receives the onboarding configuration.
Dynatrace MCP provides the validation layer so the agent can check observability readiness using real Dynatrace context instead of generic instructions.
A key design choice was to separate reasoning from infrastructure execution.
Gemini helps decide what step should happen next and explains the result, but deterministic Google Cloud tooling handles the actual setup work such as API enablement, secret storage, Cloud Run configuration, and validation steps.
Challenges I ran into
The biggest challenge was the real complexity of observability onboarding.
A full enterprise onboarding platform would need to support Google Cloud, AWS, Azure, on-prem servers, Kubernetes, Compute Engine, services, logs, metrics, traces, secrets, permissions, network paths, and platform-specific validation.
For this hackathon, I focused on building a strong Google Cloud and Dynatrace foundation rather than overclaiming a finished multi-cloud platform.
Another challenge was making the workflow feel trustworthy. Onboarding is risky because a setup can appear successful even when telemetry is missing, APIs are disabled, permissions are incomplete, or validation is only partially complete.
That is why SignalPath treats onboarding as a self-verifying workflow. The agent does not only configure the environment. It also checks readiness, reports partial failures, and explains what needs to be verified next.
Accomplishments that I’m proud of
I am proud that SignalPath reframes observability onboarding as an agentic workflow rather than a static setup form.
The project shows how Gemini, Google Cloud, and Dynatrace MCP can work together to guide setup, perform secure configuration, validate readiness, and reduce manual troubleshooting.
I am also proud of the practical design. The workflow uses Secret Manager for secure token handling, Cloud Run for the hosted runtime, deterministic tooling for setup actions, and Dynatrace MCP for validation.
This makes the project useful for real SRE and DevOps teams because onboarding is one of the highest-friction parts of the monitoring lifecycle.
What I learned
I learned that observability onboarding is not a one-time configuration task. It is an operational workflow that needs environment awareness, secure secret handling, validation, retries, and clear reporting.
I also learned that agentic systems are most useful when they combine reasoning with grounded tools.
Gemini is valuable for orchestration and explanation, but the onboarding process becomes trustworthy only when the setup actions are deterministic and the validation comes from real observability signals.
What’s next
Next, I would expand SignalPath beyond the current Google Cloud-focused prototype.
The future direction is to support AWS, Azure, on-prem servers, Compute Engine, GKE, logs, services, infrastructure metrics, traces, and richer Dynatrace Davis problem verification.
The long-term goal is to reduce observability onboarding from days or weeks of manual troubleshooting into a guided, self-verifying workflow that SRE and DevOps teams can trust.
Log in or sign up for Devpost to join the conversation.