SchemaSync

Inspiration

Modern product companies like :contentReference[oaicite:0]{index=0} scale by acquiring and integrating other products. The hardest part of every integration is not infrastructure — it's data model reconciliation.

Two systems represent the same concepts in completely different ways:

• users vs accounts
• created_at vs registration_date
• normalized vs denormalized structures

Today, this process requires senior engineers manually mapping schemas — a slow, error-prone task that can take days to weeks per integration.
We asked a simple question: what if this could be automated in seconds?

---

What it does

SchemaSync takes two database schemas in any format (SQL, Prisma, JSON, ORM models) and automatically:

• Generates a confidence-scored mapping between tables and columns
  
• Builds a visual equivalence graph showing how entities relate across systems
  
• Detects type mismatches and structural conflicts
  
• Produces a migration scaffold with transformation logic
  

It turns a 2-day senior engineering task into a ~12-second automated analysis.

---

How we built it

SchemaSync is built as a three-layer analysis pipeline:

1. Structural Fingerprinting

We extract format-agnostic signals from each table:

• Entity type classification (entity, junction, config)
• Primary key strategy
• Foreign key density and relationships
• Audit column patterns (created_at, updated_at)

This creates a structural signature independent of naming.

---

2. Semantic Embedding

Each column is embedded using sentence-transformers with full context:

• Column name
  
• Table name
  
• Sibling columns
  
• Data type
  

This allows SchemaSync to detect matches like:

• created_at ↔ registration_date
  
• account_status ↔ subscription_state
  

Even when naming is completely different.

---

3. Global Optimal Matching

We construct a similarity matrix and apply the Hungarian algorithm (scipy.optimize.linear_sum_assignment) to compute the globally optimal one-to-one mapping across schemas.

This avoids greedy mismatches and ensures consistency at scale.

---

Challenges we ran into

• Ambiguity in real schemas: multiple columns can appear equally similar; resolving this required combining structural + semantic signals
  
• Context encoding: naive embeddings failed — including table-level context significantly improved accuracy
  
• Global vs local optimization: greedy matching produced inconsistent mappings; switching to Hungarian algorithm fixed this
  
• Format normalization: supporting SQL, ORM, and JSON schemas required building a unified intermediate representation
  

---

Accomplishments that we're proud of

• Turning a high-skill, manual engineering task into an automated pipeline
  
• Achieving robust matching even with completely different naming conventions
  
• Designing a system that is:
  • reusable across any schema format
    
  • understandable by non-experts (via visual graph + confidence scores)
    
• Delivering a working prototype within hackathon constraints (36 hours)
  

---

What we learned

• Structural signals are just as important as semantic ones
  
• Context-aware embeddings dramatically outperform naive text similarity
  
• Global optimization techniques (like Hungarian matching) are critical in real-world mapping problems
  
• The biggest leverage comes from solving boring but universal problems extremely well
  

---

What's next for SchemaSync

• Live database connectors (PostgreSQL, MySQL, MongoDB)
  
• Interactive mapping editor for human-in-the-loop corrections
  
• Auto-generated migration execution engine (not just scaffold)
  
• Version-aware schema diffing for continuous integration workflows
  
• Integration into acquisition pipelines as a reusable internal platform
  

---

GitHub

https://github.com/raghibhaque/SchemaSync

Built With

• python
• typescript