About the Project
Inspiration
The project was inspired by the need for efficient and automated data workflows to handle hybrid environments, combining on-premise databases and cloud platforms. With AdventureWorks as the on-premise source, we aimed to showcase the capabilities of Microsoft Fabric and AI-driven tools, supported by GitHub Copilot for accelerated development.
How We Built It
- Ingestion: Leveraged automated pipelines and notebooks for full and delta data extraction from AdventureWorks.
- Preparation: Used notebooks to generate DDLs, adjust data types, and populate system tables in Microsoft Fabric.
- Tools: Microsoft Fabric, Azure SQL Database, Data Factory Pipelines, notebooks, and GitHub Copilot as an AI coding assistant to streamline development and debugging.
Challenges
- Adapting on-premise schemas to Microsoft Fabric required intricate type mapping and validation.
- Ensuring cross-system consistency demanded advanced reconciliation mechanisms.
What We Learned
- The power of Microsoft Fabric in building scalable hybrid data workflows.
- The role of AI and GitHub Copilot in accelerating development and ensuring accuracy.
- Strategies for seamless integration of on-premise and cloud platforms.
Built With
- adventureworks**
- azure
- azure-data-factory-**cloud-services**:-azure-sql-database
- copilot**
- database**
- fabric
- fabric**
- fabric-lakehouse-**apis-and-connectors**:-change-data-capture-(cdc)-connectors-**tools**:-github-copilot
- github
- lakehouse**
- microsoft
- microsoft-fabric-lakehouse-**databases**:-adventureworks-(on-premise)
- particle
- python**
- python-**frameworks-and-platforms**:-microsoft-fabric
- sql
- sql**
Log in or sign up for Devpost to join the conversation.