Data Engineer's Buddy

• 

  Upstream dependency of a view visualization generated by DEBy

• 

  Refresh schedule visualization generated after factoring refresh delay in upstream

• 

  Dependency visualization generated from SQL code

Inspiration

Problem Statement

Data in large organizations is spread across multiple DBs on several different DBMS
This results in a significant cost of operation from a data engineering standpoint.

Proposal

Over the course of this hackathon, I've explored how ArangoDB & NetworkX can be used to aid AI in the management of enterprise data.
A variety of tasks such as data cataloging, lineage tracking, day-to-day data operations which traditionally requires the collaboration of several teams could be done within minutes by storing the metadata in a multimodal DB such as ArangoDB. Packages & libraries such as NetworkX & CuGraph can then be leveraged by AI agents to deliver insights & perform data engineering tasks within minutes.
The multimodal offering of ArangoDB allows for metadata to be stored in such a way that a variety of data engineering tasks can easily be performed withing minutes.

What it does

Note: For this hackathon, I've focused on the data engineering tasks that can be performed by AI agents as that is universal & applicable to all enterprises. The delivery of insights is specific to the data & hence won't be applicable to everyone so I've chosen to focus on that at this stage.

DEBy - Data Engineer's Buddy is an intelligent agentic app that assists with Data Operations in Enterprises
At this stage, DEBy is able to:

1. Identify upstreams & downstreams
   
2. Assist with ETL Refresh scheduling
   
3. Intelligently calculate refresh delays
   
4. Perform impact analysis resulting from changes in a data model
   

How we built it

Data - I've stored the relationships between data structures in the form of Arango Graph & metadata such as refresh details in the form of Arango Collections
App - Created using Python, LangChain, LangGraph & OpenAI API

Challenges we ran into

1. Identifying the best way to leverage multimodal features of ArangoDB to capture the structure of enterprise data
2. Agent had to learn the relationships between enterprise data structures & the nature of dependencies
3. Teaching the agent data operation tasks

Accomplishments that we're proud of

1. Using ArangoDB to capture the relationships between SQL objects & storing metadata critical for Data Operations
2. Agent can identify upstream & downstream dependencies & intuitively understand the implications of dependencies with minimal prompting.
3. Agent can perform easily perform tedious Data Operation tasks such as refresh scheduling & lineage tracking
4. The agent can create visualizations for all of the above features

What we learned

1. ArangoDB's multimodal features can be easily used to capture information about enterprise data structures.
2. Using NetworkX & CuGraph, the data can be parsed in realtime by AI agents to extract insights & automate complex Data Operations tasks.

What's next for DEBy

1. Create an enterprise ready solution leveraging ArangoDB to simplify management of enterprise data
2. Additional features for DEBy that aims to automate complex Data Operation tasks
3. Leverage the multimodal features of ArangoDB to make DEBy the one-stop solution for all of enterprise specific requirements of a Data Engineer

Built With

• arangodb
• langchain
• langgraph
• networkx
• openai
• python