-
-
Hexacortex spiderweb datahub hackathon medo dashboard
Inspiration
About Hexacortex
🕷️ Inspiration
Most data infrastructure breaks in silence. A pipeline fails, a dataset becomes orphaned, a governance rule is violated — and nobody notices until it costs the company money. We asked: what if an agent could sense these problems the way a spider senses vibrations in its web, and repair them autonomously?
Hexacortex was born from that question. Inspired by the biomimetic efficiency of spider webs — where a single thread's vibration tells the whole story — we designed an AI agent that doesn't just alert humans, but takes action and writes the solution back so the next agent inherits the knowledge.
🧠 What We Learned
Building Hexacortex taught us that the hardest part of AI agents isn't the intelligence — it's the context. Without a complete understanding of schemas, lineage, and governance, agents hallucinate or get stuck on tasks any data engineer could finish in minutes.
DataHub's MCP Server changed that. By giving our agent a real-time sensory layer connected to the metadata graph, we learned that:
- Agents need a nervous system, not just a brain. Our 3-2-1 architecture (3 sensory threads, 2 analyzers, 1 decisor) mirrors biological systems where perception precedes decision.
- Contributing back to the graph is what separates tools from teammates. Writing remediation tags like
Remediated_by_Hexacortexmeans the next agent — human or AI — starts from a smarter state. - Biomimicry is a design pattern, not just a metaphor. The equation of our decision function:
$$\Phi = \oint_{L} \left( \frac{\nabla \Phi \cdot \vec{B}}{\pi^9} \right) dL$$
represents how context ($\vec{B}$) flows through the lattice ($L$) to produce a decision ($\Phi$).
🛠️ How We Built It
Architecture: A biomimetic 3-2-1 topology:
- 3 Sensory Threads (Schema, Lineage, Governance) connected to DataHub via MCP Server
- 2 Nervous System Analyzers (Noise Filter + Triangulation Engine)
- 1 Spider Decisor that generates production-ready repair code
Stack:
- Backend: Python + FastAPI + async MCP client
- Frontend: React + Tailwind CSS + Lucide Icons (cyberpunk dashboard)
- Integration: DataHub MCP Server for real-time metadata queries and writebacks
- License: Apache 2.0 (fully open source)
The agent reads DataHub to understand what's connected to what, triangulates real anomalies vs. routine noise, generates repair artifacts (Airflow DAGs, Python scripts), and writes the remediation context back to the graph.
⚡ Challenges We Faced
The "Not Found" trap: Early on, our agent was detecting everything and fixing nothing. We had to build a Noise Filter that distinguishes planned schema changes from real anomalies — otherwise the spider would waste energy on false vibrations.
Writeback complexity: Writing back to the DataHub graph required understanding the exact PDL (Policy Definition Language) and entity structure. We iterated multiple times to ensure our remediation tags were machine-readable for the next agent.
Real-time vs. batch: DataHub's MCP Server is powerful but requires careful async handling. We used
asyncioto keep the sensory threads listening without blocking the decisor.Demo without Docker: For the hackathon, we couldn't run a full DataHub instance in the cloud, so we built a faithful simulation layer that preserves the architecture while making the demo accessible to judges.
🎯 What's Next
Hexacortex is designed to scale. Our roadmap includes:
- Real DataHub MCP integration in production environments
- Arm64 optimization for edge deployment (<50MB RAM footprint)
- KGHOST steganographic communication layer for low-bandwidth scenarios
- Expansion to ML lineage protection (Challenge 3)
Hexacortex: We don't just read the graph. We repair it.
Log in or sign up for Devpost to join the conversation.