Inspiration Outside services like OpenAI and Anthropic ship SDK releases that break code. A bot opens a one-line bump PR, CI may or may not catch it, and the broken calls sit in files nobody has touched in months. Someone ends up reading a changelog line by line and hunting every call site by hand.
We wanted that bump PR to arrive with the answer already in it: which lines break, a fix for the ones that are safe, and a plain reason for the ones that aren't. And we wanted it to be honest. A tool that says "fixed" when it isn't is worse than no tool at all.
What we learned An open model does this well when you give it real evidence. Nemotron got much better once we fed it the new release's actual signatures, every remaining use of the removed name, and the exact dict keys the receiving function reads. Most of the quality comes from checks outside the model. The model writes fixes. Plain code decides whether a fix counts as verified. "Tests pass" isn't the same as "proven." A fix is only proven if a test actually runs the changed lines, and only new failures count against it. Honest flags beat more fixes. "Needs review: drops temperature" or "needs a person: Stream has no to_dict" is more useful than a confident wrong fix. General beats tuned. Every time we were tempted to special-case the demo, it made the tool worse. So Conduit has no hard-coded vendors, methods or types, and every check is tested on made-up libraries. How we built it Conduit isa Python CLI that runs in GitHub Actions on a dependency-bump PR.
Impact. It installs the old and new release and compares them using Python's ast module. Then it finds every line in the repo that the change breaks, including calls that go through wrapper functions and aliases. Apply. For each broken line, Nemotron 3 Ultra writes a rewrite through Nebius Token Factory's OpenAI-compatible API. Thinking is off, so the token budget goes to the answer. The prompt includes types from the new release, helpers that already exist in the repo, and the old value of any removed constant. Nemotron 3 Nano writes the one-line "why" for each fix. Verify. The repo's full test suite runs twice: once on the old code with the old release, and once after the fixes. Only new failures count. A failure is traced back to the fix that caused it, that fix gets one retry with the error fed back, and it's reverted if it still fails. Honesty checks. These run on top of the tests: A coverage gate: no test ran the changed lines, so the status is "not covered by tests". A dict-key check: the fix passes a dict missing keys the function reads, so it's sent back to the model. A drop check: the fix stops passing something the old call passed, so the status is "needs review". Undefined-name and None checks. Anything Conduit can't fix safely becomes "needs a person", with a one-sentence reason. Report. Conduit posts one PR comment with an impact table, a status for each line, the test results and a proposed diff. It never pushes and never merges. On a real open-source repo (council, anthropic 0.34.2 → 1.0.0), one live run took about 11 minutes and used 15 Token Factory calls and 26,250 tokens. It found 9 broken lines. 3 fixes were verified by the repo's own tests, 1 fix wasn't covered by tests, 1 fix was flagged for review, 1 line went to a person, and 3 were skipped as duplicates. Across council's 320-test suite there were 0 new failures.
Challenges we ran into "Verified" kept meaning too little. An early run marked a fix verified even though it would have raised a KeyError, because no test ran that line. That led to the coverage gate and the dict-key check. Silent drops. A rewrite moved from the old completions call to the new messages call and quietly dropped temperature, top_k and top_p. The tests couldn't see it. Now any dropped argument is named in plain words, and the fix is marked "needs review" instead of verified. Measuring coverage broke a test. Our first coverage recorder slowed one timing-sensitive test enough to fail it, and that failure got blamed on every fix, so they were all reverted. We split it up: pass/fail now comes from a normal test run, and coverage comes from a separate traced run. Tests that were already failing. Council had 4 tests failing before any change. Without a baseline run, Conduit would have blamed itself for them. Order matters. Fixing a caller before the function it depends on produced fixes that didn't fit together. Fixing producers first solved it. Thinking vs. the token cap. Nemotron's reasoning counts toward max_tokens, so we run with thinking off, and we give the model better evidence instead of more room. Runs vary. The same commit can produce slightly different fixes and token counts from run to run. The statuses stayed the same across our runs, and no run has marked broken code as verified.
Log in or sign up for Devpost to join the conversation.