A migration can preserve every row and every dollar while changing who gets billed. Sales may need a parent-company view of a customer, while Finance needs invoices tied to the correct subsidiary's legal identity. Those requirements are compatible, but a migration that collapses reporting and billing into one identity can silently break the business.

Migration Review makes that failure visible before a migration is approved. The demonstration uses synthetic customer and invoice data and a supplied hypothetical migration script. No real customer incident or financial loss is claimed.

What it does

Migration Review is a local workspace for rehearsing and comparing CRM migration repairs. An operator reviews business requirements, inspects concrete positive and negative examples, freezes the check suite, and launches three repair strategies: a minimal patch, a mapping rewrite, and a lineage-first repair.

Each repair executes in its own Daytona sandbox. The comparison shows every mandatory check, the affected invoice count and amount, and invoice-level relationships between the reporting parent, billable legal entity, and billing account. A repair qualifies only when every mandatory check runs and passes. Execution failures remain visible as unevaluated results.

The app also exports a review package containing the mapping, source, available output, frozen checks, relationship differences, checksums, and an offline replay script.

How it was built

The application uses Python for orchestration and evaluation, a JavaScript/HTML/CSS interface, and a local Node.js bridge to the OpenAI Codex SDK. The current workflow uses an agent to draft four bounded check specifications for the approved business rules. A trusted compiler adds them to eight independent baseline checks. The operator reviews ten fixed examples before freezing the suite and generating fresh repairs.

Daytona provides separate execution environments for the baseline, corrected control, and three generated repairs, with bounded concurrency and recorded cleanup. Source data and frozen check identities accompany the evidence. Independent answer keys grade the results and are withheld from repair prompts.

SurrealDB stores customer/invoice relationships and versioned candidate and run snapshots. The full-data cloud persistence was verified through a fresh connection: 5,000 customers, 50,000 invoices, 50,000 source graph edges, five candidate/run snapshots, and 200,000 result relationships matched the saved evidence.

Arize AX records five experiments for the full-data comparison on one shared scenario/example, with eleven evaluator results per experiment. The experiment values were independently read back through the API, including the failed candidate's error and unevaluated checks. The local interface displays saved evidence and dated integration receipts.

Challenges

The central challenge was separating reporting hierarchy from billing identity. Synthetic cases include legacy billing identifiers, non-billable branches, billable parents with billable children, and similar names that must never be used as identity joins.

Another challenge was making the evaluation trustworthy. Generated repairs must not redefine the expected answer, and a model-generated check needs independent examples before it is accepted. The implementation freezes the source and checks, keeps mandatory invariants, and preserves failed runs rather than treating missing output as success.

Accomplishments

The saved full-data Daytona comparison executed on 5,000 synthetic customers and 50,000 synthetic invoices. The supplied flawed proposal preserved all invoice rows and the exact $1,220,645,348.86 synthetic source total, but assigned 45,840 invoices, representing $1,118,564,737.37 in synthetic invoice value, to the wrong billable entities.

The corrected control and generated repairs A and C passed all eight frozen checks with zero misbilled invoices on this fixture. Repair B encountered a transformation error; its checks were not run. All five sandbox deletions were verified. These amounts describe synthetic test data, not money saved or actual losses prevented.

A separate executed 24-invoice development run demonstrated the complete agent-draft, example-review, freeze, generate, and Daytona execution workflow. Its new control and all three new repairs passed twelve checks. This is a different suite with different repair artifacts; its results do not replace the full-data comparison. SurrealDB and Arize receipts apply to the original full-data run, not to this newer development run.

What was learned

Row counts and monetary totals are necessary checks, but they cannot establish business correctness. Invoice ownership needs its own invariant and traceable evidence. Multiple repairs may qualify, and a real execution failure is useful evidence. The operator needs to see both outcomes before deciding what to adopt.

What's next

Extend the reviewed scenario library to more migration patterns, add connectors for real CRM exports, and evaluate repairs on independently held-out datasets. Production adoption would require broader coverage, operational review, and a separately approved deployment process. The current prototype rehearses and exports evidence; it does not perform a production migration.

Built With

  • arize
  • codex
  • daytona
  • surrealdb
Share this project:

Updates

Submission history