ZeroTouch SRE

An autonomous SRE copilot that analyzes Dynatrace telemetry, investigates incidents, proposes safe mitigations, and generates operational documentation on Google Cloud.

Comment 1

Inspiration

Production incidents are messy. An alert fires, people start checking dashboards, logs, chat messages, and runbooks, and everyone is trying to figure out what happened. For small teams, that process can take a lot of time and attention.

I wanted to build something that could handle the first stage of incident response automatically. The idea was simple: when an alert arrives, the system should gather evidence, suggest a likely cause, prepare a response plan, and document everything. At the same time, I didn't want an agent making uncontrolled changes to production systems, so safety became a core part of the design.

What it does

ZeroTouch SRE receives a production alert through an API endpoint and starts an investigation workflow.

The system collects telemetry from Dynatrace when available. If live telemetry is not accessible, it switches to deterministic telemetry so the workflow can still run consistently.

Using the available evidence, the service generates a likely root cause, prepares a mitigation plan, and runs approved actions through a simulation layer. It also creates supporting artifacts such as post-mortems, runbooks, and execution traces.

The result is a structured incident report that gives engineers a clear starting point during an outage.

How I built it

I built the project using FastAPI and deployed it on Google Cloud Run.

The backend consists of several components:

  • An API layer that receives alerts and health-check requests.
  • A telemetry layer that connects to Dynatrace and handles fallback data.
  • A reasoning engine that processes incident information and generates recommendations.
  • A safety layer that validates and simulates approved actions.
  • A billing guard that tracks estimated model usage against configurable limits.

I used Google Cloud services for deployment and secret management. The application can run with live telemetry or in a deterministic local mode for testing and demonstrations.

Challenges I ran into

The biggest challenge was deciding how much autonomy the system should have.

Generating recommendations is straightforward. Allowing an automated system to interact with production environments is a different problem. I spent a lot of time thinking about what actions should be permitted and how to keep the workflow useful without introducing unnecessary risk.

Another challenge was reliability. External services are not always available during development or demos, so I added deterministic telemetry fallbacks to keep the investigation pipeline working even when live data could not be retrieved.

Deploying the application and managing cloud configuration securely also required careful attention.

Accomplishments that I'm proud of

  • Built a complete alert-to-investigation workflow as a solo project.
  • Integrated Dynatrace telemetry into the incident analysis process.
  • Deployed the application on Google Cloud Run.
  • Added safety controls through allowlisted actions and audit logging.
  • Automated the creation of post-mortems, runbooks, and execution traces.
  • Added budget tracking for estimated model usage.

What I learned

This project reinforced how important context is during incident response. Alerts alone are rarely enough. Useful recommendations depend on telemetry, operational history, and clear reasoning.

I also learned that safety mechanisms are just as important as the reasoning system itself. An operations agent needs boundaries, visibility, and traceability.

Building the entire system end-to-end gave me hands-on experience with observability tooling, cloud deployment, secret management, and production-inspired workflows.

What's next for ZeroTouch SRE

There are several areas I would like to explore next:

  • Support additional observability platforms alongside Dynatrace.
  • Add integrations with Slack and Microsoft Teams.
  • Introduce approval workflows before production actions are executed.
  • Store and reuse knowledge from previous incidents.
  • Expand remediation playbooks for common operational issues.
  • Build a dashboard for reviewing incidents, actions, and generated artifacts.

The current version focuses on investigation and planning. My next goal is to make the platform more useful in real operational environments while preserving the same safety-first principles.

Built With

Share this project:

Updates