Healthcare BI Steward

Governs a clinical Snowflake+Metabase stack: PHI tags, dashboard certification, and pre-change impact alerts, all proposed via DataHub -- never auto-committed. Built on AWS Bedrock AgentCore.

Comment

Inspiration

I'd been curious about agents and data governance for a while, and I was already working hands-on with Snowflake, AWS, and DataHub. The spark that tied it together was a real conversation: a healthcare customer specifically asked me how to govern their BI layer — not just the warehouse, but the dashboards built on top of it. That question is what turned a vague "let's build an agent" idea into a concrete problem to point all three tools at, and became a learning-plus-experiment project I could actually build end-to-end.

Once I started building, I realized the story isn't specific to healthcare. Any industry with a data platform, multiple BI teams building their own dashboards, and nobody watching governance across both layers has the same problem. This project became my way of exploring what an open-source approach to BI, AI, and data governance could look like — healthcare was the concrete first case, but the pattern generalizes.

What it does

An agent that governs a clinical data warehouse (Snowflake) and its BI reporting layer (Metabase) together, cataloged in DataHub. It does three things, every one of them gated behind explicit human approval:

  • Tags PHI/PII at the source — scans Snowflake tables for columns that look like patient-identifying data (name, DOB, SSN, MRN, phone, email, address, ZIP) and flags any missing a privacy tag.
  • Certifies trustworthy dashboards — scans Metabase dashboards cataloged in DataHub and flags any nobody has formally vouched for.
  • Warns before a schema change ships — traces downstream lineage from a proposed column change and flags it as high severity if it would break a certified dashboard.

Nothing is ever written to DataHub automatically. Every tag or certification is staged as a proposal that a human has to explicitly approve or reject first.

How we built it

A Strands agent running on AWS Bedrock (Claude Haiku 4.5), exposing six tool functions that wrap discovery, proposal, review, and impact-analysis logic. It talks to a self-hosted DataHub instance through the official mcp-server-datahub MCP server. Snowflake (key-pair auth against an MFA-enforced account) and Metabase are both ingested into DataHub as the shared catalog connecting the warehouse and the BI layer.

The agent is deployed for real to AWS Bedrock AgentCore Runtime — not left as a local script — reaching the local DataHub instance through an ngrok tunnel. A five-tab Streamlit app gives a visual demo surface: live chat against the deployed agent with a tool-call trace, live DataHub tag/certification state, a CloudWatch log pull, live Snowflake query history, and Metabase status.

Challenges we ran into

  • DataHub's documented native propose/approve tools turned out to be Cloud-only — this self-hosted instance only exposes direct-mutation tools. I built a custom human-gated proposal ledger instead of relying on a workflow that wasn't actually available.
  • Real MCP payload shapes for tags, search, and lineage differed from what the docs implied, and the bugs only surfaced once real tagged data existed to expose them.
  • The deployed container's own observability auto-instrumentation set an OpenTelemetry propagator that leaked into a subprocess without the matching package installed, silently crashing every DataHub call until I traced it through raw CloudWatch logs.
  • A separate, unrelated telemetry client was retrying a blocked host on every single call, adding a minute or more of latency that looked exactly like an application bug until I isolated it.

Accomplishments that we're proud of

  • A real AWS-hosted deployment, not a local-only demo — proven end to end against the actual Bedrock AgentCore Runtime.
  • A genuine human-in-the-loop rejection driven by real domain reasoning: the agent flagged a state-level address column as PHI-shaped, and I rejected it citing HIPAA Safe Harbor (state alone isn't a covered geographic identifier) — exactly the safety mechanism the propose-then-approve gate exists for.
  • Pre-change impact analysis that correctly identified a certified dashboard would break before anyone touched the underlying column.

What we learned

That this problem is bigger than healthcare. Building the governance model against one customer's real ask made it obvious the same gap — a warehouse and a BI layer that nobody watches together — shows up anywhere multiple teams build dashboards on shared data without a shared source of truth for trust.

What's next for Healthcare BI Steward

  • A durable proposal ledger (S3/DynamoDB) instead of the current container-disk-scoped one.
  • DataHub Cloud's native proposal workflow, if/when it becomes available on the tier I'm using.
  • Generalizing the pattern beyond healthcare identifiers — the same tag/certify/warn model applies to any industry with a warehouse-plus-BI setup and no shared governance layer.

Built With

  • amazon-web-services
  • anthropic-claude
  • aws-bedrock
  • aws-bedrock-agentcore
  • aws-cdk
  • aws-cloudwatch
  • aws-iam
  • boto3
  • datahub
  • docker
  • graphql
  • mcp-(model-context-protocol)
  • metabase
  • ngrok
  • node.js
  • opentelemetry
  • python
  • snowflake
  • strands-agents
  • streamlit
  • uv
Share this project:

Updates