PipelinePulse

Know your data. Trust your decisions.

Comment

Inspiration

Honestly, this came from a real frustration. I kept seeing situations where a data pipeline would silently break and nobody would know until someone noticed a dashboard looked weird or a report had old numbers. And then when you try to figure out what happened, you're staring at Fivetran error codes that mean nothing to you unless you're an engineer. I thought - there has to be a better way. What if you could just ask what's wrong, in plain English, and actually get a useful answer back?

What it does

PipelinePulse lets you have a normal conversation about your data pipelines. You ask things like "what's broken right now?" or "what do I tell my manager?" and it actually tells you - no jargon, no dashboards, no digging through logs. It pulls live data straight from Fivetran and uses Gemini 2.5 Flash to explain what's happening and what you should do about it.

How I built it

I connected Fivetran to BigQuery on Google Cloud first, got the connectors syncing, then built a Python script that talks to the Fivetran API and feeds that data into Gemini 2.5 Flash. From there I wrapped it in a Flask API so it could be called programmatically too. On the Google Cloud side I set up an Agent Platform agent with custom instructions to act as a pipeline health advisor. Honestly a lot of it was just figuring out which APIs needed to be enabled and why things weren't talking to each other.

Challenges I ran into

The Google Cloud Agent Platform UI had changed a lot from the documentation so I spent way too long trying to find things that had been renamed or moved. Also ran into disk space issues trying to deploy to Cloud Run which I just didn't have time to fix properly.

Getting the agent to give genuinely useful responses - not just "connector X has status Y" but actually explaining what that means for a business - took a lot of prompt tweaking.

Accomplishments that I’m proud of

When it finally worked end to end - you ask it a question and it comes back with something an actual non-technical person could read and act on - that felt really good. The response quality surprised me honestly. It doesn't just list statuses, it prioritizes, explains the business impact, and even helps you write the message to your manager. That part felt genuinely useful.

What I’ve learned

Fivetran's MCP integration is seriously powerful for grounding AI agents in real data. The difference between an agent that's guessing and one that's actually looking at your live connectors is huge.

Also learned that the hard part of building tools for non-technical users isn't the tech - it's figuring out what they actually need to hear and how to say it clearly.

What's next for PipelinePulse

  • Proactive Slack alerts so you don't have to ask - you just get notified when something breaks
  • Re-trigger syncs directly from the chat
  • Cloud Run deployment so it's always running, not just on a laptop
  • Role-based responses - where an engineer and a sales manager ask the same question and get answers tailored to what they actually care about

Built With

Share this project:

Updates