CLOY: Control Lineage Optimize Yield

Inspiration

Data incidents are the silent killer of data-driven organizations. When a dashboard breaks, a pipeline fails, or a data quality issue emerges, the typical response is frantic, manual investigation across five different tools. On-call engineers spend hours tracing lineage by hand, piecing together what went wrong. 48% of organizations face monthly pipeline failures and data teams spend 30–40% of their time on firefighting instead of revenue-generating work. The context to diagnose incidents already exists in DataHub's metadata graph—lineage, ownership, quality signals, schemas—but it remains siloed, requiring human operators to manually navigate it. CLOY was born from the vision of an autonomous agent that reads DataHub context, takes action, and writes results back so that knowledge is inherited by the next person or agent.

What it does

CLOY is an autonomous multi-agent system that transforms reactive data incident response into proactive, automated resilience:

1. Continuously monitors DataHub for signals of trouble—failing assertions, schema changes, pipeline failures—via Kafka and webhooks.
2. Autonomously investigates by traversing the lineage graph to identify root causes (upstream assertion failures, pipeline failures, schema changes, stale data).
3. Generates fix proposals and submits governed proposals in DataHub for human approval or auto-execution.
4. Writes knowledge back to DataHub as incident report documents, tags on affected assets, and ownership updates—so every incident makes the system smarter.

The system includes specialized agents: EventListenerAgent (monitors events), RootCauseAgent (traverses lineage), RemediationAgent (generates fixes), and KnowledgeWriterAgent (persists knowledge). A voice interface allows natural language interaction ("Find all tables with PII," "Why is the dashboard stale?"), and a reinforcement learning module enables the system to learn optimal remediation actions from experience.

How we built it

• DataHub Platform – Unified metadata graph providing lineage, assertions, schemas, ownership, and quality signals.
• MCP Server & Agent Context Kit – Standardized tools that give AI agents direct access to DataHub context via search, lineage traversal, schema inspection, and SQL query retrieval.
• Multi-Agent Orchestration – Specialized agents communicate via a message bus, with an orchestrator coordinating workflow execution (detect → analyze → remediate → persist).
• Event-Driven Architecture – Kafka consumer and webhook server ingest DataHub events (MetadataChangeLog, PlatformEvent) in real time.
• Root Cause Analysis – BFS/DFS lineage traversal with inspectors checking upstream assertions, pipeline status, schema changes, and data freshness.
• Remediation Planning – Issue-specific planners generate step-by-step fix proposals (SQL, pipeline restarts, schema rollbacks) with governed proposals in DataHub.
• Knowledge Persistence – Document writer creates Markdown incident reports, tag writer applies severity/status tags, ownership writer updates asset metadata.
• Voice Interface – Speech-to-text (Whisper/Google/Azure) + NLU intent classification + text-to-speech (ElevenLabs/Google/Azure) supporting 5 languages (en, es, fr, de, zh-CN).
• Reinforcement Learning – DQN/PPO agents with experience replay and knowledge base for continuous improvement.
• Frontend – React + TypeScript + Tailwind CSS + Cytoscape.js for lineage visualization + WebSocket for real-time updates.
• Infrastructure – Docker + Kubernetes deployment with Kafka, Redis for state persistence.

Challenges we ran into

• Lineage Traversal at Scale – Multi-hop lineage queries across thousands of entities required careful BFS implementation with hop limits and caching to avoid timeouts.
• Event Schema Variance – DataHub events arrive in different formats (MetadataChangeLog vs PlatformEvent); we built robust parsers with fallbacks.
• Agent Coordination – Ensuring specialized agents communicate without deadlocks or race conditions required a priority-based message bus.
• Governed Proposal Integration – DataHub's proposal API is still evolving; we built a flexible abstraction layer that works with both current and future endpoints.
• Voice Latency – Real-time speech recognition + LLM inference + TTS synthesis needed aggressive optimization (streaming, smaller models) to feel responsive.
• RL Training Stability – DQN training on simulated incident data required careful tuning of epsilon decay, replay buffer size, and reward shaping to achieve convergence.
• Multi-Language Support – Intent classification and entity extraction had to work across 5 languages; we combined rule-based patterns with language-specific LLM prompts.

Accomplishments that we're proud of

• End-to-End Autonomous Workflow – From event ingestion → detection → root cause analysis → remediation → knowledge persistence, all in one seamless pipeline.
• Real-Time Lineage Traversal – Successfully traversing up to 5 hops of upstream/downstream lineage with column-level precision in under 2 seconds.
• Governed Proposals – Integration with DataHub's proposal system enables safe, auditable changes with human approval gates.
• Knowledge Loop – Each incident enriches DataHub's context graph; the system gets smarter over time as incident reports, tags, and ownership updates accumulate.
• Voice Interface – Natural language programming for data operations in 5 languages, with real-time speech synthesis.
• Reinforcement Learning – Agent achieving >85% success rate on simulated incidents after 1000 episodes, with knowledge base bootstrapping to accelerate learning.
• Production-Ready UI – Full-featured dashboard with lineage visualization, incident management, and real-time WebSocket updates.
• Multi-Language Support – Full i18n support across the entire application (frontend, voice, and NLU).

What we learned

• Context is Everything – AI agents are only as good as the context they can access. DataHub's metadata graph is the ideal foundation for autonomous data agents.
• Multi-Agent > Monolith – Specialized agents (detection, analysis, remediation, knowledge) are easier to build, test, and scale than a single monolithic agent.
• Event-Driven is Essential – Real-time incident response requires an event-driven architecture; polling simply doesn't work at scale.
• Governance Matters – Autonomous remediation must include human approval gates; governed proposals in DataHub provide the perfect balance of automation and control.
• Voice is a Game-Changer – Natural language interaction dramatically lowers the barrier to entry for data operations; non-engineers can now investigate and resolve incidents.
• Reinforcement Learning Works – RL agents can learn optimal remediation strategies from experience, but require careful reward shaping and a realistic simulation environment.
• Open Source Wins – Building on DataHub, MCP, LangChain, and React allowed us to focus on innovation rather than reinventing the wheel.

What's next for CLOY: Control Lineage Optimize Yield

• Production Deployment – Roll out CLOY to real data teams, with monitoring, alerting, and continuous improvement based on production incidents.
• Enhanced RL – Move from simulated to real-world training; implement online learning where the agent updates its policy based on actual incident outcomes.
• More Specialized Agents – Add agents for ML model monitoring, cost optimization, and data governance compliance.
• MCP Expansion – Contribute new tools to the DataHub MCP Server (e.g., assertion creation, proposal management, incident resolution) to benefit the broader ecosystem.
• Multi-Agent Collaboration – Enable agents to negotiate and coordinate on complex incidents that span multiple domains (e.g., schema change + pipeline failure).
• Voice Pro – Add support for more languages, custom wake words, and integration with Slack/Teams voice channels.
• Analytics Dashboard – Build a dedicated analytics module showing MTTR trends, incident patterns, and agent performance metrics.
• Open Source – Release CLOY as an open-source project to empower the data community to build their own autonomous incident response systems.

CLOY: Control Lineage Optimize Yield – Because when your data crashes, CLOY catches it.

Built With

• datahub