Isotope — Devpost Project Story

Inspiration

Every team that depends on a third-party API has lived through the same nightmare: a provider ships a "minor" contract change, nothing looks wrong in review, CI is green, and then a webhook silently stops writing to the database in production. The bug isn't a crash — it's a silent behavioral drift, the kind that's nearly impossible to catch with normal unit tests because both the old and new code paths return a 200.

We wanted to build something that treats a dependency bump not as a diff to review, but as a hypothesis to test: run the old handler and the new handler side by side, in isolation, under identical conditions, and mechanically prove whether they behave the same. That idea became Isotope — named for the fact that two versions of the same code, like two isotopes of an element, can look identical on the surface while decaying completely differently underneath.

What it does

Isotope traces a provider contract change (e.g. a Stripe or other API/library update) through your application code, then proves — mechanically, not by inspection — whether your handlers still behave the same way against it.

Given a code change, Isotope:

Scans the codebase to build a bounded provider dataflow graph (BDG), identifying exactly which execution paths actually touch the changed provider.
Executes the old and new versions of each affected handler in isolated, sandboxed child processes, with mocked provider/DB boundaries, fixed clocks/randomness/UUIDs, and blocked network egress, so every run is deterministic and repeatable.
Diffs the captured side effects with a pure structural differ. If the two runs provably diverge, that's a mechanical, unappealable FAIL — no LLM, no heuristics, no judgment call.
Escalates only genuinely ambiguous cases to a bounded reasoning layer, which requires two independent semantic votes before returning PASS_REASONED, FAIL_REASONED, or ESCALATE.
Proposes repairs (deterministic first, model-assisted later) against the BDG, applies them in an isolated worktree, and independently re-verifies the candidate against the original immutable baseline plus a held-out fixture — so a "fix" is never reported until it's been proven to restore the original behavior.

All of this runs inside the customer's own CI runner via a bundled GitHub Action — there's no hosted service, no autonomous Git operations, and no auto-merge. Isotope tells you, with proof, whether an update broke you, and only ever suggests a fix after independently verifying it.

How we built it

Isotope is a TypeScript/Node monorepo (pnpm workspaces) built up in deliberate, audited phases:

core owns the shared schemas, artifact I/O, and the L6 verdict resolver — the single source of truth for what counts as PASS, FAIL, or ESCALATE.
resolver-ts / resolver-py statically construct a bounded provider dataflow graph (BDG) from the code, so we know which execution paths actually touch the changed provider before we ever run anything.
harness-ts / harness-py execute the old and new handlers as isolated, sandboxed child processes — mocked Stripe/DB boundaries, fixed clocks and randomness, blocked egress, repeatable runs.
differ performs pure structural comparison of captured side effects — no heuristics, no LLM calls, just mechanical classification.
reasoner steps in only when the mechanical differ can't resolve things cleanly, building a bounded evidence packet and requiring two independent semantic votes before returning PASS_REASONED, FAIL_REASONED, or ESCALATE.
repair + verifier generate deterministic (and, later, model-assisted) repair candidates against the BDG, apply them in an isolated worktree, and re-verify against the original immutable baseline and a held-out fixture before anything is reported as fixed.
reporter, fleet, and cli turn all of that into PR annotations, a GitHub Action, and a self-contained fleet dashboard — with a typed CLI that runs the same verification logic locally or in CI.

We built this in twelve tracked phases — from a walking skeleton that could run a single mocked handler, up through a five-case detection matrix (mechanical FAIL, migrated PASS, provider no-op PASS, provenance-protected SKIP, and ambiguity ESCALATE) proven against isolated Git histories, and finally a bundled GitHub Action with trust-separated workflow templates for real repos.

Challenges we ran into

Keeping "mechanical" mechanical. It was tempting to let the differ get "smart" about ambiguous cases. Instead we pushed all interpretation into a clearly separated, explicitly bounded reasoning layer, so a FAIL from the pure differ stays unappealable and auditable.
Determinism across process boundaries. Getting truly reproducible runs — stable serialization, fixed UUIDs/dates, blocked network egress — inside a real Node child process is fussier than it sounds, and we hit more than one flaky-test rabbit hole before nailing repeat-run isolation.
Resisting scope creep toward autonomy. It would have been easy to let Isotope auto-merge fixes or open-endedly "agent" its way through a repair. We deliberately froze the system against that: no hosted service, no auto-merge, no open-ended repair agent — every repair is proposed and independently re-verified, never silently applied.
Honesty about gaps. We didn't fake our way past missing real-world fixtures. Rather than fabricate a passing "real" Stripe integration test, we left the missing fixtures/normalized/sub-updated-single pair as an explicitly documented, honest blocker — synthetic tests prove the plumbing works, not that we've validated against production traffic.

Accomplishments that we're proud of

Shipping a five-case detection matrix — mechanical FAIL, migrated PASS, provider no-op PASS, provenance-protected SKIP, and ambiguity ESCALATE — each reproduced against isolated Git histories and artifacts, not just asserted in a slide deck.
Building a real, deterministic execution harness (not a mock of one) that isolates side effects, fixes non-determinism at the source, and blocks egress, so "PASS" and "FAIL" actually mean something.
Keeping the system honest under pressure: rather than quietly relaxing scope to hit a demo, we froze Phase 12 against hosted services, auto-merge, new languages, and open-ended repair agents, and documented the real remaining gap (a missing real-world fixture pair) instead of papering over it.
Getting all the way from a walking skeleton to a bundled, working GitHub Action — with a typed reporter, bounded PR annotations, bot-owned comment upserts, and job summaries — across twelve tracked, audited phases in a single hackathon.
Designing a verdict pipeline where a mechanical FAIL is genuinely unappealable, and a proposed repair is never trusted until it's been independently re-verified against a held-out fixture — a level of rigor we didn't originally expect to reach in the time we had.

What we learned

Mechanical verdicts beat vibes. We learned to distinguish a structural, provably-correct "this changed" from a probabilistic "this looks different," and to keep those two categories from ever bleeding into each other in the verdict resolver.
Isolation is the whole game. Getting deterministic replay (fixed Date/random/UUID, egress blocking, side-effect capture) right in a real child-process harness took far more care than the actual diffing logic.
Bounded autonomy is a design constraint, not an afterthought. We repeatedly had to resist the urge to let the tool "just fix it" — every repair only ever gets proposed, and only after an independently re-executed, held-out verification run confirms it behaves like the original.
Provenance is a first-class citizen. Distinguishing a human-verified ChangeSpec from a model-drafted one (verified_by: draft) so drafts can never silently enter the trusted verification pipeline turned out to be one of the most important lines we drew in the whole system.

What's next for Isotope

Expand language coverage beyond TypeScript/JavaScript and Python, starting with the languages our early users actually run in production.
Harden the repair planner with a broader library of deterministic repair patterns before leaning further on model-assisted candidates, keeping the "propose, never auto-apply" guarantee intact as capability grows.
Grow a fleet dashboard into a real multi-repo, multi-provider view teams can use to see which of their dependencies are quietly drifting before an update ever lands.
Publish the accuracy methodology. Our honesty-bounded accuracy command is a start; we want to open up the historical accuracy corpus so the mechanical-vs-semantic split can be independently audited by others, not just us.
Customers. Especially small startups that need to scale rapidly and can't afford dependency downtime (for themselves, or their customers). We are the protection layer for API hygene.