Aegisflow

Inspiration

In the modern enterprise, data is the engine driving automation, machine learning, and critical business decisions. When an upstream schema changes or data drift occurs, the failure often slips through unnoticed until it infects downstream systems, breaks executive dashboards, and compromises production AI models. Traditional observability tools are reactive—they simply alert a human engineer after the damage has already occurred, leading to hours of costly downtime and lost trust. I was inspired to build AegisFlow to shift data engineering from constant firefighting to an autonomous, self-healing state, effectively giving data pipelines their own intelligent immune system.

What it does

• The Sentinel Agent: Operates on the front line, continuously monitoring data stream ingestion points and schema signatures to instantly flag data structural drift or corruption. • The Lineage Oracle Agent: Integrates natively with the enterprise metadata catalog to inspect live lineage maps. • The Circuit Breaker Agent: The automated executor. Armed with the lineage context, it throws targeted, surgical circuit breakers to quarantine corrupted partitions while orchestrating automated rollbacks or data healing workflows, ensuring only high-quality, verified data moves downstream.

How I built it

• Data Foundation: Deployed a production-simulated environment running PostgreSQL databases to manage operational target datasets. • Governance Hub: Stood up DataHub Core via Docker to function as our centralized telemetry clearinghouse for dataset aspects, time-series usage stats, and lineage. • Orchestration & Emitters: Developed custom Python automation scripts using the DataHub REST Emitter and Metadata Change Proposal (MCP) SDKs to dynamically stream dataset profiles, change histories, and operational queries directly into the platform backend.

Challenges I ran into

• The Interface Port Disconnect: Early emitter payloads threw continuous 404 Client Errors. I discovered my code was mistakenly targeting the client-facing frontend web UI port (9002) instead of interacting directly with DataHub's backend Generalized Metadata Service (GMS) api engine running on port 8080. • SDK Schema Evolution: I ran into a python TypeError exception when streaming schema profiles. Digging deep into the core repository classes revealed that the DataHub Python SDK properties had evolved, and the older columnProfiles parameter had been refactored into fieldProfiles within DatasetProfileClass. • The Time-Series Midnight Sync Paradox: Our operational query counts and usage trends initially displayed a blank "No Data" frame in the UI. I discovered that DataHub's Elasticsearch time-series aggregators expect strict chronological bucket boundaries. I resolved this by modifying our script to cleanly floor-align every telemetry timestamp to 00:00:00 UTC midnight, instantly unlocking our data trend line graphs.

Accomplishments that I am proud of

• Surgical Zero-Human Quarantine: Successfully simulating an upstream dataset corruption event and watching our multi-agent framework isolate the asset within milliseconds without human intervention. • Dynamic Metadata Ingestion: Constructing a complete, working Python-to-DataHub automation pipeline capable of updating complex time-series operations, query tracking metrics, and historical logs seamlessly. • Resilient Problem Solving: Overcoming local hardware resource limits and container sync lags within the cloud workspace to build a fully functional enterprise architecture demo.

What I learned

I gained extensive experience working with generalized metadata schemas, aspect-oriented architectures, and time-series streaming. I learned that precision-crafted data telemetry must conform strictly to indexer boundaries (like UTC day bucketing) to ensure that downstream user interfaces.

What's next for Aegisflow

I intend to expand AegisFlow from single-node pipeline scripts into an enterprise-wide cloud daemon framework supporting cross-platform infrastructures like Snowflake, BigQuery, and Apache Kafka.

Built With

• agent
• automation
• codespace
• datahub
• docker
• github
• metadata
• multi
• postgresql
• python