Project Update: Canonical Metadata Model Introduced
One of the biggest architectural milestones for DE Copilot is now in place: the Canonical Metadata Model.
Instead of generating code directly from project-specific STTM formats, DE Copilot now normalizes metadata into a standard internal representation first.
STTM ↓ Canonical Metadata Model ↓ Code / DQ / Documentation / Testing Artifacts
This abstraction layer allows the same metadata to generate:
• Snowflake DDL & SQL • PySpark Code • dbt Models • Data Quality Rules • Technical Specifications • Data Dictionaries • Test Cases • Documentation
The long-term vision is platform-independent engineering automation where the business metadata remains constant and only the target templates change.
Future targets include:
- Databricks
- Azure Data Factory
- Airflow
- AWS Glue
- Informatica
- Monte Carlo
- Great Expectations
- Additional database platforms
The goal is simple:
Reduce repetitive engineering work so engineers can spend more time solving business and architecture problems instead of manually translating metadata.
Website: https://dataengineeringcopilot.com
Architecture Deep Dive: https://dataengineeringcopilot.com/blog/canonical-metadata-model-engine-behind-de-copilot
Enterprise STTM Stress Test: https://dataengineeringcopilot.com/blog/stress-testing-de-copilot-enterprise-sttm