Inspiration
Recurring CSV deliveries often look familiar while hiding breaking changes: renamed columns, type drift, duplicate identifiers, invalid dates, unexpected business values, and inconsistent numeric formats.
SchemaPilot was built to make data onboarding safer, explainable, and reviewable.
What it does
SchemaPilot compares a trusted previous dataset with a new incoming delivery and runs a complete workflow:
Upload → Diagnose → Repair → Validate → Load → AI Decision Brief
It detects schema drift, validates data quality, separates safe deterministic repairs from decisions requiring human approval, revalidates the corrected copy, and executes a transactional DuckDB pipeline.
In the included demo:
- 23 rows are staged
- 13 trusted rows are loaded
- 9 rows are rejected
- 1 row is sent for review
- Reconciliation passes
How we built it
SchemaPilot is built with Python, Streamlit, pandas, DuckDB, Pydantic, the OpenAI Responses API, GPT-5.6, and Codex.
The deterministic Python modules remain the source of truth for schema comparison, validation, repair, row classification, and reconciliation.
GPT-5.6 is used only for an advisory Decision Brief generated from aggregate evidence. It cannot change the data, repair decisions, validation results, or pipeline classifications.
Challenges
The main challenges were:
- Detecting likely column renames without hiding real schema changes
- Keeping safe automatic fixes separate from business decisions
- Preserving Streamlit state across reruns
- Guaranteeing that every row reaches exactly one final outcome
- Building transactional reconciliation and rollback
- Sending useful evidence to GPT-5.6 without exposing customer records
Accomplishments
- End-to-end working public demo
- Deterministic schema-drift and data-quality engine
- Safe-copy remediation
- Transactional DuckDB pipeline
- Exact reconciliation across target, rejected, and review tables
- Aggregate-only AI evidence
- 27 automated tests passing
- Public GitHub repository with setup, security, and testing documentation
What we learned
Reliable AI systems need clear boundaries. SchemaPilot uses deterministic engineering controls for decisions that must be reproducible, while GPT-5.6 explains the operational result and highlights decisions that still require human approval.
What's next
Future improvements include configurable data contracts, governed approval workflows, warehouse connectors, additional file formats, authentication, and pipeline-run observability.
Log in or sign up for Devpost to join the conversation.