Pipeline Guardian

"Your GitLab pipeline fails. PipelineGuardian fixes it autonomously using 6 AI agents in 7 seconds".

Comment

🔥 Inspiration

Every developer has lived this interruption.

You are deep in flow. A pipeline fails. You context-switch into log archaeology — scrolling through hundreds of lines looking for the one error that matters. You find it. You fix it. Forty-five minutes gone.

And next week, someone else on the team hits the exact same npm lock file error. Nobody remembered the fix. The forty-five minutes happen again.

We are Team 3AH — Abhishek, Akshay, Archi, and Hitendra. We have all lost hours to this exact problem. CI/CD pipeline failures are the #1 developer interruption, and yet no existing tool closes the full loop:

Detect the failure → Diagnose it intelligently → Find the pattern from history → Generate the exact fix → Apply it — all without a human lifting a finger.

GitLab Duo's Root Cause Analysis reads a log and suggests a fix. But it has no memory. The same failure appears next week and it starts from scratch again.

That gap is what PipelineGuardian fills.

⚙️ What It Does

PipelineGuardian is an autonomous CI/CD repair agent that watches every GitLab pipeline failure and fixes it in ~7 seconds using a 6-agent Google ADK pipeline.

The moment a pipeline fails, PipelineGuardian automatically:

  1. 🔍 Fetches the failed job logs via GitLab MCP (get_pipeline_jobs)
  2. 🧠 Classifies the error using Gemini 2.5 Flash: syntax | dependency | test | config_env | infra_runner | flaky_test
  3. 🗂️ Searches past similar failures via semantic_code_search (GitLab MCP) + Supabase pgvector
  4. 🔧 Generates the exact .gitlab-ci.yml fix using Gemini with historical context
  5. ✅ Validates the generated YAML before applying
  6. 🚦 Routes based on confidence score:
Confidence Action Who Acts
≥ 0.85 Creates fix MR + retries pipeline 🤖 Agent acts autonomously
0.60 – 0.85 Posts fix as MR comment 👤 Human approves
< 0.60 Creates GitLab Issue + escalates 🚨 Human investigates

The memory layer means every resolved failure makes PipelineGuardian smarter. Day 1 it fixes by classification alone. Day 30 it recognises the exact pattern from history and applies the proven fix instantly.

The same npm error that cost your team 45 minutes last Tuesday? PipelineGuardian fixes it in 7 seconds today.

🏗️ How We Built It

PipelineGuardian is a 6-agent sequential pipeline built on Google ADK Python, with Gemini 2.5 Flash as the reasoning engine, GitLab MCP Server as the action layer, and Supabase pgvector as the persistent memory brain.

Agent Architecture

# Agent Role Key Tools
1 🔍 Watcher Fetches failed job traces GitLab MCP get_pipeline_jobs
2 🧠 Classifier Classifies error type + confidence score Gemini 2.5 Flash + Vertex AI Search
3 🗂️ Memory Searcher Finds similar past failures GitLab MCP semantic_code_search + Supabase pgvector
4 🔧 Fix Generator Generates exact YAML diff Gemini 2.5 Flash
5 ✅ Validator Validates YAML before applying Gemini 2.5 Flash
6 🚦 Action Agent Creates fix MR, retries pipeline, or escalates GitLab MCP create_merge_request, manage_pipeline, create_issue

ADK Implementation

The orchestration is a Google ADK SequentialAgent wrapping 6 LlmAgent instances:

from google.adk.agents import LlmAgent, SequentialAgent

pipeline = SequentialAgent(
    name="pipelineguardian_pipeline",
    sub_agents=[
        watcher_agent,        # GitLab MCP: get_pipeline_jobs
        classifier_agent,     # Gemini 2.5 Flash + Vertex AI Search grounding
        memory_agent,         # semantic_code_search + pgvector
        fix_generator_agent,  # Gemini: exact YAML diff
        validator_agent,      # Gemini: YAML validation
        action_agent,         # GitLab MCP: create_merge_request / manage_pipeline
    ]
)

All 6 agents run gemini-2.5-flash with GenerateContentConfig and SafetySetting thresholds for HARM_CATEGORY_DANGEROUS_CONTENT, harassment, hate speech, and sexually explicit content — all set to BLOCK_MEDIUM_AND_ABOVE.

Confidence Score Formula

The routing decision uses a real confidence score — not LLM guesswork:

$$\text{confidence} = P(\text{classification}) \times \left(1 + \alpha \cdot \text{sim}_{\text{top}}\right)$$

Where:

  • \( P(\text{classification}) \) = Gemini's softmax probability for the top error category
  • \( \text{sim}_{\text{top}} \) = cosine similarity score from pgvector match \((0.0 – 1.0)\)
  • \( \alpha = 0.2 \) = similarity bonus weight

Infrastructure

Layer Technology Role
Agent Framework Google ADK Python SequentialAgent orchestration
LLM Gemini 2.5 Flash Classification, fix generation, validation
Grounding Vertex AI Search Data Store CI/CD failure pattern knowledge base
Cloud Hosting Google Cloud Run Webhook server + ADK pipeline
Secrets Google Cloud Secret Manager All API keys — zero hardcoding
Vector Memory Supabase PostgreSQL + pgvector Failure embeddings + similarity search
Realtime Supabase Realtime Live agent trace streaming in dashboard

🧗 Challenges We Ran Into

🔌 GitLab MCP SSE Transport in Production

The GitLab MCP Server uses SSE (Server-Sent Events) transport. Connecting ADK's McpToolset via SseConnectionParams inside a Cloud Run container required careful handling of the async connection lifecycle and graceful shutdown via exit_stack.aclose(). A dropped SSE stream mid-pipeline would silently stall the agent — we added a 60-second timeout and reconnect logic to handle this.

🔀 Confidence-Based Routing in a SequentialAgent

ADK's SequentialAgent passes context between agents but does not natively support branching. We implemented confidence-based routing inside the action_agent instruction — the agent reads the classifier's confidence from session context and selects the appropriate MCP tool call. Getting this to work reliably required careful prompt engineering and strict output structure validation between agents.

📐 pgvector Similarity Threshold Tuning

The failure memory is only useful if matches are accurate. Too low a cosine similarity threshold gives false positives (a React syntax error matching a Python dependency failure). Too high and nothing matches. We tuned the threshold to 0.72 through iteration on 30 seeded historical failures, balancing precision against recall.

🥶 Webhook Cold Start on Cloud Run

Cloud Run scales to zero between pipeline events. The ADK pipeline + MCP connection startup on cold start took 4–8 seconds, pushing response time above our "instant" target. We solved this with --min-instances 1 on the Cloud Run service and pre-warming the MCP connection at startup rather than per-request.

🎥 Real vs Simulated MCP Calls in the Demo

We wanted judges to see live MCP calls, not mocked data. This required a real GitLab Ultimate Trial, a real test repository with a deliberately failing pipeline, and careful credential management. The Trace Viewer now shows a 🟢 LIVE badge vs ⚫ SIMULATED badge on every MCP call panel — judges can instantly verify which calls are real.

🏆 Accomplishments That We're Proud Of

🤝 Human-in-the-loop that actually makes sense

Most agents either act blindly or always ask permission. PipelineGuardian acts when confident, asks when uncertain, and escalates when it doesn't know. That three-tier routing is the behaviour of a trustworthy colleague, not a chatbot.

🧠 Memory that compounds

Every failure PipelineGuardian resolves becomes a vector-embedded memory. After seeding 30 historical failures, the memory searcher returned accurate matches with 0.85+ cosine similarity for 26 of them. That is not a demo trick — that is the system actually working.

🔗 8 GitLab MCP tools — all live, all verified

Every tool fires in the live demo. The Trace Viewer shows the raw JSON-RPC request and response for each call so judges can verify it is real:
get_pipeline_jobs · get_merge_request_diffs · get_merge_request_pipelines semantic_code_search · search · manage_pipeline · create_merge_request · create_issue

🟢 A pipeline goes from red to green without a human touching anything

That moment — watching a failing pipeline turn green after PipelineGuardian creates the fix MR and triggers a retry — is the clearest possible proof. It is not a slide. It is not a diagram. It is a live GitLab pipeline turning green in under 7 seconds.

🚀 Built and deployed by four engineers in 6 days

Every agent is real. Every MCP call is live. Every Cloud Run deployment is functional. We are proud of the discipline it took to scope correctly and ship something that actually works — not a prototype that only runs on localhost.

📚 What We Learned

Google ADK changes how you think about agents

The shift from "write code that calls an API" to "write an instruction that tells an agent what to do" is genuinely profound. Writing "use semantic_code_search to find similar past failures" and watching ADK autonomously construct the MCP call, handle the response, and pass context forward — that is a different category of software development.

MCP is the missing abstraction layer

Before MCP, connecting an agent to GitLab meant writing and maintaining custom API wrappers. With the GitLab MCP Server, create_merge_request is just a tool the agent can call. The protocol handles authentication, serialisation, and error handling. That simplicity is what makes agents actually practical to build.

Memory is what separates a useful agent from a toy

A single-shot agent that classifies a failure and suggests a fix is interesting. An agent that recognises "we have seen this exact failure 6 times, here is the fix that worked every time" is genuinely useful. The pgvector memory layer is what makes PipelineGuardian valuable beyond the first use.

Confidence scoring beats binary decisions

The three-tier routing (act / suggest / escalate) is more useful than act-or-do-nothing. Developers trust the agent more when it knows its limits. A fix suggestion they can review beats a blind auto-apply, and both beat a GitLab issue with no context.

🚀 What's Next for PipelineGuardian

  • 🐙 GitHub Actions support — the same 6-agent architecture applied to GitHub CI/CD failures, using GitHub MCP alongside GitLab MCP
  • 🧪 Flaky test quarantine — automatically detect tests that fail >30% of runs and add retry: annotations until the underlying issue is resolved
  • 🔮 MR pre-flight prediction — before a pipeline even runs, analyse the MR diff and predict failure probability using the trained failure memory
  • 📊 Team knowledge reports — weekly aggregation showing which failure categories your team keeps repeating, with trend tracking over time
  • 💬 Slack integration — surface fix suggestions directly in the team's incident channel without requiring anyone to open GitLab

Built With

Share this project:

Updates