Migration Oracle

Most tools warn you about a migration. We run it somewhere safe first, score the forecast, and remember the lesson so the next one is not a cold guess.

Comment

Inspiration

Database migrations are one of the few things every team depends on and almost every team fears. In development, a change can look harmless. In production, the same change can lock tables, slow critical workflows, or fail at the worst possible moment.

That's the problem Migration Oracle is built to solve. We wanted to build a system that could predict the impact of a migration, verify those predictions against real execution, and learn from every outcome so the next decision is a little smarter than the last.

The bigger idea is simple. Migrations should not be blind leaps of faith. They should be informed, observable, and backed by evidence, so teams can ship schema changes with confidence instead of anxiety.

What it does

Migration Oracle is an agentic database migration advisor that predicts the impact of schema changes, validates those predictions on disposable CockroachDB Cloud shadow clusters, and learns from every migration it executes.

The platform begins by analyzing an incoming migration and estimating its runtime, storage impact, rollback risk, and confidence. It then provisions an isolated CockroachDB Cloud shadow cluster, executes the migration safely, measures the actual outcome, compares those results with its predictions, and stores what it learns for future migrations.

  • Predict: Policy checks and Amazon Bedrock estimate migration duration, storage impact, rollback risk, and confidence using similar previously graded migrations.
  • Approve: Predictions are presented alongside supporting evidence so the user can decide whether to proceed with shadow execution.
  • Verify: AWS Step Functions and AWS Lambda create a disposable BASIC CockroachDB Cloud cluster via the Cloud REST API, load schema and seed, execute the migration, and measure actual runtime, storage change, and success or failure. The UI follows SHOW JOB by id, then a SHOW JOBS snapshot. Changefeeds write row events to S3 for the tables the migration touches; they do not drive the spinner. Cleanup holds the cluster briefly; an EventBridge sweeper tears it down.
  • Investigate: After execution, a Bedrock tool-use agent in the execute Lambda connects to CockroachDB's hosted Managed MCP Server against that shadow cluster. Known write tools are filtered out so the investigation cannot issue schema or insert calls.
  • Grade & Remember: Migration Oracle compares predictions against actual execution, scores its own accuracy, generates embeddings with Amazon Titan, and stores the graded outcome in CockroachDB, indexed for similarity search with Distributed Vector Indexing.
  • Improve: Future migrations retrieve the most relevant graded execution histories. This helps the system learn from real outcomes instead of relying only on documentation or static rules.

The dashboard brings the entire workflow into a single workspace. Users can connect a read-only database, generate predictions, launch shadow executions, monitor progress, inspect prediction accuracy alongside execution evidence, and explore the AI's learned memories and reasoning. Slack notifications can optionally deliver migration updates directly to the connected user, so they don't have to keep the dashboard open while migrations are running.

Additionally, a linked GitHub repo can open a pull request that contains a migration file; Migration Oracle predicts it and comments on the PR. A human still approves in the app before any shadow cluster is created.

How we built it

Migration Oracle is built around CockroachDB as both its operational database and long-term memory store. The FastAPI backend manages migration runs, predictions, approvals, shadow cluster metadata, grades, and learned memories, while transaction retries ensure concurrent operations remain safe under CockroachDB's Serializable isolation.

The prediction pipeline combines policy analysis, hybrid memory retrieval using CockroachDB Distributed Vector Indexing, and Amazon Bedrock to forecast migration duration, storage impact, and rollback risk. Every prediction pauses for explicit user approval before execution begins.

Memory is also seeded with a small, fully sourced set of open-source migration incidents (each with a source URL). Those rows are labeled not a graded shadow run and are excluded from accuracy metrics. What the closed loop learns from is shadow-verified grades we actually ran.

Verification is orchestrated with AWS Step Functions and AWS Lambda. The workflow provisions a disposable CockroachDB Cloud cluster through the Cloud REST API, loads schema and seed, executes the migration, and collects metrics. Cleanup holds the cluster briefly; an EventBridge sweeper tears it down. Secrets live in AWS Secrets Manager. Artifacts, reports, and changefeed output go to Amazon S3.

After execution, a Bedrock tool use agent connects to the CockroachDB Managed MCP Server in read-only mode to investigate the migration's blast radius. Prediction accuracy is then graded, embedded with Amazon Titan, and stored back in CockroachDB as searchable memory. Those memories are indexed with Distributed Vector Indexing so future migrations can retrieve similar execution histories and learn from real outcomes.

Challenges we ran into

  • Closing the prediction loop. Generating AI predictions was the easy part. The real challenge was comparing those predictions with actual migration results, grading the model's accuracy, embedding the outcome with Amazon Titan, and storing it in CockroachDB so future predictions could learn from real execution instead of static documentation.

  • Managing disposable shadow clusters. Every verification run creates a temporary CockroachDB Cloud cluster, loads a representative schema, executes the migration, collects metrics, and tears the cluster down. Making that process reliable meant handling provisioning delays, limiting concurrent clusters, guaranteeing cleanup, and recovering gracefully from failures or aborted runs. Cleanup always runs, holds the cluster about five minutes for inspection, then an EventBridge sweeper tears it down and reaps orphans (every fifteen minutes).

  • Building trust in the predictions. We needed a way to verify whether our predictions matched what actually happened during a migration. That meant comparing every prediction against the execution results and presenting the evidence clearly enough for developers to make a decision.

  • Getting MCP running in production. Calling CockroachDB's hosted Managed MCP Server from an AWS Lambda client was very different from using it locally. Packaging the client, granting Bedrock InvokeModel on ExecuteMigration, and treating a failed MCP handshake as best effort so it does not fail the SQL took several iterations. The Execute Lambda still has a timeout, so a hang can still fail that step.

Accomplishments that we're proud of

  • Watching the feedback loop work end-to-end. Seeing a migration move from prediction to shadow execution, grading, and finally becoming a new memory for future predictions made months of engineering come together.

  • Turning every migration into a learning opportunity. Each completed shadow execution is graded, embedded, and stored as searchable memory. Future predictions can then retrieve similar migrations and learn from real execution history instead of starting from scratch.

  • Bringing so many moving pieces together. Integrating CockroachDB, AWS Step Functions, Lambda, Amazon Bedrock, Titan, MCP, Clerk, and our frontend into a single workflow took a lot of iteration, and seeing everything work together was incredibly rewarding.

  • Sharing our journey with the community. Alongside building Migration Oracle, we documented our progress through engineering blogs, sharing the architecture, design decisions, trade-offs, challenges, and lessons we learned. One of these articles was accepted by Stackademic, a Medium publication followed by 86K+ readers, which gave us a chance to share our experience building with CockroachDB and AWS beyond the hackathon.

What we learned

  • Building the AI was only part of the challenge. We went into this project thinking the hard part would be building the AI. It turned out the harder part was building everything around it. A prediction only becomes useful when you can verify it, compare it with what actually happened, and learn from the outcome. That idea ended up shaping almost every part of Migration Oracle.

  • Cockroach University became one of our most valuable resources. As undergraduate students, its self-paced courses and documentation helped us explore distributed databases far beyond what we'd covered in class. We learned the fundamentals of ACID transactions, Serializable isolation, transaction retries, and query planning with EXPLAIN, and we also learned how to use CockroachDB features while building Migration Oracle. There were plenty of times when we weren't sure how to approach something, and we'd end up back in Cockroach University or the documentation. Looking back, a lot of the project wouldn't have come together without them.

  • Trust matters as much as accuracy. We realized people shouldn't have to take an AI's word for it, especially when production databases are involved. That pushed us to build features like human approval before execution, guaranteed cleanup, evidence from shadow runs, and model traces so users can understand why the system made a recommendation instead of simply accepting it.

  • Honest memory beats a big fake corpus. The Memory browser includes a fully sourced set of open-source migration incidents (each with a source URL) so early predictions are not cold guesses. Those rows are labeled not a graded shadow run and are excluded from accuracy metrics. What the closed loop actually learns from is shadow-verified grades we ran ourselves and stored back in CockroachDB. That distinction mattered: we did not want retrieval to look like fake production history.

What's next for Migration Oracle

  • Expand the learning loop. We want to run a much larger and more diverse set of migrations through the shadow execution engine, covering different schema changes, data volumes, workload patterns, and failure modes. We also plan to expand our sourced migration corpus to improve cold-start predictions while keeping external incidents clearly separated from shadow-verified outcomes.

  • Integrate deeper into developer workflows. Our GitHub integration is an early step toward making migration analysis part of the development lifecycle. We want to extend this into CI/CD workflows where migrations can be automatically analyzed during pull requests, with predictions and supporting evidence surfaced for review before a human approves shadow execution.

  • Scale toward a production platform. As usage grows, we want to make disposable cluster provisioning and verification more cost-efficient while adding team capabilities such as shared migration history, approval workflows, and audit trails. Eventually, this could evolve into hosted plans with higher execution limits and collaboration features for larger teams.

Built With

Share this project:

Updates