Inspiration
Every software engineer and DevOps professional knows the dread of a 3:00 AM PagerDuty alert. More often than not, the initial response involves executing predictable, manual playbooks: parsing messy, unstructured log traces, querying metric thresholds, identifying the culprit service, and executing standard remediation steps—such as rolling back a bad deployment, flushing deadlocked database connections, or clearing congested task queues.
While monitoring platforms alert us when systems break, existing tools still rely on human engineers to bridge the gap between detection and execution. I was inspired to build AutoOps 3.5 to transform incident response from a manual, reactive grind into a fully autonomous, self-healing continuous action engine. I wanted to build an agent that doesn't just chat about a problem, but intercepts alerts asynchronously, performs diagnostic heavy-lifting, and executes verified system remediations in real time without waiting for human prompts.
What it does
AutoOps 3.5 operates as an event-driven background incident mitigation engine built on Gemini 3.5 Pro and deployed across Google Cloud Infrastructure.
- Asynchronous Interception: When an infrastructure anomaly occurs, telemetry systems emit alert payloads directly to a Google Cloud Pub/Sub topic.
- Context-Aware Diagnostics: The event triggers a Cloud Run container where AutoOps 3.5 ingests unstructured stack traces and metrics. Using Gemini 3.5 Pro, the agent reasons over the log context, executes diagnostic tools, and computes anomaly confidence scores.
- Guardrailed Execution & State Mutation: Before applying fixes, the agent passes proposed tool payloads through an inline Gemma 2 guardrail service hosted on Vertex AI to enforce zero-trust security policies. Upon validation, it mutates production infrastructure (e.g., triggering Cloud Run rollbacks or flushing connection pools) and persists full session states and audit traces to Firestore.
How I built it
I architected AutoOps 3.5 using a modular, decoupled approach leveraging Google Cloud services and SDKs:
┌────────────────────────────────────────────────────────┐
│ AutoOps 3.5 Engine │
│ │
[Alert Source] ──>│ [Pub/Sub] ──> [Cloud Run Service] │
│ │ │
│ ▼ │
│ [Gemini 3.5 Pro Agent] │
│ │ │
│ ┌───────────┴───────────┐ │
│ ▼ ▼ │
│ [Gemma 2 Guardrail] [Firestore Memory] │
│ │ │ │
│ ▼ ▼ │
│ [Gemma Model] [Audit & State Logs] │
└────────────────────────────────────────────────────────┘
│
▼
[Infrastructure Mutation]
- Core Intelligence: Gemini 3.5 Pro via the Google GenAI SDK, chosen for its extreme context window processing and dynamic tool-calling accuracy.
- Asynchronous Triggering & Runtime: Cloud Pub/Sub handles ingestion, pushing payloads directly to an autoscaling Cloud Run microservice.
- State & Memory Management: Firestore serves as the persistent memory bank, storing long-term context, operational timelines, and tool execution state across async runs.
- Secondary Model Integration: I deployed Gemma 2 on Vertex AI as an inline, deterministic payload validator to catch hallucinated arguments or dangerous infrastructure actions before execution.
To prevent false positives and runaway remediation loops, AutoOps 3.5 calculates a composite Remediation Confidence Score (\(C_{\text{rem}}\)) before executing any tool payload:
$$C_{\text{rem}} = w_1 S_{\text{log}} + w_2 (1 - \Delta M) + w_3 P_{\text{hist}}$$
Where:
- \(S_{\text{log}} \in [0, 1]\) represents the semantic error similarity derived from Gemini 3.5's log analysis.
- \(\Delta M\) is the normalized metric deviation (e.g., memory leak rate or HTTP 5xx error surge) from baseline over time period \(\Delta t\):
$$\Delta M = \frac{M_{\text{current}} - M_{\text{baseline}}}{M_{\text{baseline}}}$$
- \(P_{\text{hist}} \in [0, 1]\) is the historical success rate of the candidate playbook stored in Firestore.
- \(w_1, w_2, w_3\) are weighting factors where \(\sum_{i=1}^3 w_i = 1.0\).
An infrastructure mutation tool is only executed if \(C_{\text{rem}} \ge \Theta_{\text{threshold}}\) (set by policy, e.g., \(\Theta = 0.85\)). Otherwise, the agent safely escalates the issue to human engineers with a summarized diagnostic report.
Challenges I ran into
- State Persistence Across Asynchronous Invocations: Because Pub/Sub triggers Cloud Run statelessly, maintaining agent working memory across multi-step diagnostic loops was challenging. I solved this by implementing a transactional state machine in Firestore using the Google GenAI SDK, allowing the agent to hydrate its memory context instantly upon each push notification.
- Deterministic Guardrails on Agent Actions: Autonomous agents with infrastructure access carry operational risk. Ensuring Gemini 3.5 didn't attempt unverified tool execution required tight JSON schema constraints and a secondary validation pass through my Vertex AI-hosted Gemma 2 endpoint.
- Cold-Start Latency: Balancing Cloud Run scale-to-zero capabilities with real-time incident mitigation required optimizing container spin-up times and reusing client initializations for Vertex AI connections.
Accomplishments that I'm proud of
- True Autonomous Background Execution: I successfully built a system that operates completely outside standard chat loops. It ingests an event, reasons, executes tools, and logs updates entirely in the background.
- Zero-Trust Multi-Model Guardrails: Successfully pairing Gemini 3.5 Pro with a localized Gemma 2 model on Vertex AI to validate payload safety before executing real infrastructure mutations.
- Production-Grade Cloud Integration: Deploying a fully decoupled architecture on Google Cloud using Pub/Sub, Cloud Run, and Firestore that can scale to thousands of concurrent alerts.
What I learned
- Beyond the Chat Loop: Building AutoOps 3.5 showed me the immense power of moving AI from reactive chat boxes to continuous, event-driven background processing.
- Multi-Model Synergy: Pairing a large reasoning model (Gemini 3.5 Pro) with lightweight, dedicated model endpoints (Gemma 2) creates a robust balance between cognitive decision-making and strict deterministic safety.
- Cloud-Native Agentics: Utilizing native cloud eventing (Pub/Sub) and serverless execution (Cloud Run) proved to be the most effective way to build production-ready, failure-tolerant agentic architectures.
What's next for AutoOps3.5:Autonomous Multi-Cloud Incident Mitigation Engine
- Multi-Cloud Remediations: Expanding beyond Google Cloud to natively support AWS (Lambda, ECS) and Azure (App Services, AKS) infrastructure mutations.
- Predictive Incident Prevention: Leveraging Gemini 3.5's long-context capabilities to analyze weeks of continuous telemetry data to remediate soft failure trends before an anomaly alert is fired.
- Interactive Human-in-the-Loop Slack/Teams Integration: Adding a low-friction escalation hook where AutoOps 3.5 presents one-click approval buttons in Slack for low-confidence ($C_{\text{rem}} < 0.85$) remediations.
Built With
- ai-agents
- automated-testing
- autonomous-agents
- cloud-run
- devops
- docker
- firestore
- flask
- gemini-3.5-pro
- gemma-2
- google-cloud
- google-genai-sdk
- google-pubsub
- infrastructure-as-code
- python
- site-reliability-engineering
- vertex-ai
- zero-trust
Log in or sign up for Devpost to join the conversation.