Pipeline Sentry

What is the inspiration behind your project? Data engineers and DevOps teams are constantly plagued by alert fatigue, and nothing is worse than getting a critical 3 AM engineering page just to realize a pipeline failed due to something trivial—like an expired API key or a misplaced token. I wanted to build something that acts as an autonomous first responder. Instead of waking up a human to do a predictable, repetitive fix, why not let an intelligent agent handle the triage and self-healing on its own? What does your project do? Pipeline Sentry is an autonomous, self-healing AI first responder for data pipelines. It continuously monitors Fivetran data pipelines for sync failures. The moment a failure occurs, it intercepts the error, uses Gemini to instantly triage dense logs into plain-English alerts for Slack or Google Chat, and executes self-healing protocols. If it's a credential issue, it fetches the updated secrets from GCP Secret Manager, updates the Fivetran connector configuration, and triggers a re-sync—completely eliminating the need for human intervention for routine pipeline breaks. How did you build it? (Which technologies did you use?) I built the entire backend orchestration using Python. For the agentic intelligence, I leveraged Google Cloud Agent Builder paired with Vertex AI and Gemini. The real magic happens through the Model Context Protocol (MCP), which allows the Gemini agent to securely communicate with external environments—specifically through custom MCP configurations for the Fivetran API and GCP Secret Manager. The notifications are tied out using webhook connectors for real-time team alerts. What challenges did you face while building your project? The trickiest part was definitely setting up seamless, reliable communication via the Model Context Protocol (MCP) between the LLM and the infrastructure APIs. Ensuring that the agent accurately parses unstructured, messy log data without hallucinating the root cause was another hurdle. I had to fine-tune the systemic prompts inside Vertex AI to guarantee that the agent only executes self-healing protocols when it is 100% certain about the error type, avoiding unintended configuration overwrites. What accomplishments are you proud of? I'm incredibly proud of achieving true closed-loop automation. It’s one thing to have an AI look at a log and say "Hey, your key is broken," but it's an entirely different level to have that agent securely jump into GCP Secret Manager, rotate the token, fix the configuration in Fivetran, and restart the sync successfully. Making a complex agent architecture feel lightning-fast and genuinely helpful is a huge win. What did you learn while working on this project? This project was a massive deep-dive into the Model Context Protocol (MCP) and Google Cloud Agent Builder. I learned how to build highly contextual, tool-use systems rather than just standard chat applications. It also completely shifted my perspective on DevOps—showing me how LLMs can move from passive monitoring tools to active, trustworthy operational infrastructure elements. What’s next for your project? (What are your future plans?) Right now, the self-healing works incredibly well for specific credential and connection errors. Moving forward, I want to expand its diagnostics capabilities to handle upstream schema changes, data drift, and complex rate-limiting issues. I also plan on adding a web-based dashboard using React to give teams a clear observability timeline of exactly what errors the Sentry intercepted, how it diagnosed them, and the actions it took to fix them autonomously.

Built With

• context
• fivetran-api
• gcp-secret-manager
• gemini-api
• google-cloud-agent-builder
• google-cloud-run
• model
• protocol
• python
• slack-api
• vertex-ai