-
-
System blueprint: From unstructured logs to deterministic GraphRAG
-
The Knowledge Graph in action, mapping complex infrastructure relationships.
-
Gemini 3.5 instantly transforming raw text into structured Neo4j data.
-
The Multi-Agent engine executing a Root Cause Analysis query via GraphRAG
Inspiration
In enterprise IT and Level 2 infrastructure support, teams bleed time and money during critical outages not because they lack data, but because they lack context. Incident reports, chat logs, and escalation tickets create mountains of "dark data"—unstructured text where the true root cause is buried across multiple disconnected systems.
We realized that standard chatbots and RAG (Retrieval-Augmented Generation) systems fail here. They just summarize symptoms. To find a root cause, you need to traverse the chain of events. That inspired us to build the Dark Data Alchemist: an autonomous pipeline that transforms chaotic IT logs into a mathematical, highly queryable Knowledge Graph.
What it does
The Dark Data Alchemist is an asynchronous, multi-agent GraphRAG pipeline.
Autonomous Ingestion: A Python "Watchdog" agent monitors a designated directory for incoming, unstructured incident reports.
Structuring: It passes the raw text to Google Gemini 3.5 Flash, strictly prompting it to extract entities (People, Tech, Events) and weave them into an unbroken "Causal Chain" (JSON).
Storage: The structured data is instantly injected into a Neo4j graph database, serving as the permanent corporate memory bank.
GraphRAG Retrieval: When an engineer asks a plain-English question (e.g., "What caused the AWS Database crash?"), our Chat Agent queries the graph, traversing up to 4 hops deep to find the root cause, and uses Gemini to synthesize a perfect, human-readable summary.
How we built it
We prioritized a decoupled, production-ready architecture over a brittle script. Backend telemetry and audit logging are managed via Google Cloud Firestore
Compute & Reasoning: Google GenAI SDK (Gemini 3.5 Flash)
Storage & State: Neo4j Database & Cypher Query Language
Orchestration: Python (python-dotenv for secure credential management, autonomous filesystem monitoring)
Challenges we ran into
Building an agentic pipeline exposed two massive engineering hurdles that we successfully engineered our way out of:
Disconnected Data Islands: Initially, the LLM extracted isolated relationships (e.g., Sarah -> Rebooted -> Server and AuthParser -> Leaked -> Memory), leaving the graph physically broken. We engineered a strict "Causal Chain" System Prompt that forced Gemini to mathematically link the root cause all the way to the final symptom before generating the JSON.
LLM Hallucinations in Database Queries: Asking an LLM to write raw Cypher queries on the fly resulted in hallucinated node labels and syntax errors. To make the architecture bulletproof, we implemented a Fuzzy Net Retrieval System. We strictly use Gemini for what it excels at (semantic extraction of a core noun from the user's question), and we use Python for what it excels at (injecting that noun into a hardcoded, multi-hop Cypher query). This guarantees 100% database query success.
Accomplishments that we're proud of
We are incredibly proud to have built a true Agentic Workflow, not just a basic LLM wrapper. The system operates asynchronously, manages long-term database state, handles its own data structuring, and successfully executes multi-hop root cause analysis without any human hand-holding.
What we learned
We learned the massive difference between standard Vector RAG (which finds similar text) and GraphRAG (which finds connected truth). We also learned that to build resilient AI systems, you must ruthlessly decouple the AI's reasoning capabilities from strict programmatic execution.
What's next for Dark Data Alchemist: Autonomous GraphRAG
The next step for production readiness is migrating the local file ingestion to the cloud. We plan to integrate Google Cloud Storage buckets to trigger cloud functions the second a new incident ticket is closed, autonomously updating the enterprise Knowledge Graph in real-time.
Log in or sign up for Devpost to join the conversation.