OpenPort

An agent that moves your app's OpenAI and Anthropic calls onto NVIDIA Nemotron on Nebius Token Factory, proves each one still works, and opens the PR. It patches vulnerable dependencies too.

Comment

Inspiration

Most AI apps run on someone else's model: their prices, their roadmap, their outages. Moving to an open model like NVIDIA Nemotron on Nebius Token Factory is, on paper, a three-line change: a base URL, a key and a model name. Almost nobody makes it, because nobody can prove it won't break the product. A prompt tuned for one model quietly fails on another, and the failures show up in production, not in the diff.

So I built OpenPort: an agent that proves the switch, then makes it.

What it does

Point OpenPort at a repo, local or on GitHub:

  1. It finds every LLM call. OpenAI and Anthropic SDK calls, and plain HTTP calls to OpenAI-style endpoints, in Python and TypeScript. It works out what each one needs: JSON mode or schemas, tools, images, reasoning.
  2. It builds an eval for each call. The cases come from hand-labelled data or recorded logs. When a repo has neither, Nemotron Super invents realistic inputs and Nemotron Ultra labels them. The grader fits the output: field checks for JSON and tool calls, an Ultra judge for free text.
  3. It tests each call, the app's own code, on Nemotron. The call runs as written, with its SDK routed to Token Factory. Nothing is mocked.
  4. It tunes what fails: the prompt and the call's settings, never the model's weights. Tavily pulls NVIDIA's own guidance for the exact model. Nemotron Ultra reads the failures and rewrites the prompt and settings: reasoning on or off, token limits, temperature. Each call starts on Nano and only climbs to Super or Ultra when it must. A fix counts only if it also passes cases Ultra never saw (about 30% are held out).
  5. It writes the code and opens the pull request. The edits put the tuned prompt and settings where they live in the code. A small runtime runs Anthropic SDK code on Nemotron, since there is no Anthropic-compatible endpoint. The PR shows each call's model, its score before and after, its held-out score and its cost per 1,000 calls, plus the sources it cites.
  6. It keeps you honest. It adds a GitHub Action that reruns the evals on every pull request and fails the build if a call drops below its bar.

The same engine makes a second kind of PR: security upgrades.

  • openport upgrade reads every lockfile and asks OSV.dev what's vulnerable.
  • It picks fixes that have been out at least a week, so a fresh malicious release can't be chosen.
  • It runs the repo's own tests in a throwaway container, before and after.
  • When the upgrade breaks something, Nemotron Super reads the changelog Tavily found and fixes the application code, never the tests. If Super's fix fails, it's undone and Ultra takes over.

Results

  • SupportDesk, my demo app, has 5 calls on GPT and Claude, and all 5 move.
    • A plain swap passed 70% of their cases. After OpenPort: 96%, and 98% on held-out cases.
    • Cost per 1,000 runs of each call: $11.70 → $2.27.
    • OpenPort's own Token Factory bill for the run: $0.70.
  • jsliapark/preflight is an open-source AI code reviewer I didn't write, with 4 Claude Opus calls and 119 hand-labelled cases.
    • All 4 calls move, each passing 96% or more of its held-out cases.
    • Cost per 1,000 runs: $22.08 → $1.32, 94% cheaper.
  • receipts is a gpt-4o-mini vision call that reads store receipts. It moves to Nemotron 3.5 Super VL, passing 96% of its cases and 100% of the held-out ones.
  • shareAI-lab/learn-claude-code has 79k stars.
    • openport upgrade closed 48 advisories across 6 packages, 38 of them in Next.js.
    • The proof is the repo's own CI: npm ci, tsc --noEmit and next build.
    • The bump broke the build. Super's fix failed and was undone. Ultra's seven-line root layout passed.
  • The field study covers the 20 most-starred open-source AI apps on GitHub.
    • Their own code makes 207 LLM calls, in 14 of the apps. OpenPort moves 78 of them to Nemotron as they are. (The video's 79 is from Oct 28: one of lobehub's calls now stays, because moving it would send its Anthropic key to Token Factory.)
    • 19 of the 20 lock a dependency with a known vulnerability.
  • Real pull requests: every recorded run opened one on GitHub. They're in a sandbox organization, narralogy-labs, so no maintainer was pinged.

How I built it

  • Every model inside OpenPort is Nemotron on Nebius Token Factory, each tier on the job it fits:
    • Nano reads pricing pages.
    • Super writes test inputs and fixes code.
    • Ultra tunes prompts, judges free-text answers and takes over the code fixes Super can't make.
    • Nemotron 3.5 Super VL takes the calls that read images.
    • Tiers and per-token prices come from Token Factory's live /v1/models catalog, so every cost number is real.
  • Tavily brings in what no model was trained on: NVIDIA's guidance for the exact model, each package's changelog cut to the releases in between, and providers' pricing pages. Every quote OpenPort uses is checked word for word against the page.
  • The stack:
    • The agent and CLI are Python: FastAPI, Typer, tree-sitter for TypeScript, and the OpenAI and Anthropic SDKs.
    • The dashboard is React, Vite and Tailwind, and streams each run live over server-sent events.
    • Each repo's tests run in Docker, with resource limits and only the upgraded copy mounted.
    • The hosted dashboard runs the same Docker image in a Cloudflare Container behind a Worker. Keys are secrets, spending has a daily cap, and live runs are gated for the judges. Recorded runs replay for everyone.
    • 444 offline tests, which need no API keys, run in GitHub Actions.
  • Built, not run live: these need Nebius AI Cloud credit, which the hackathon doesn't provide.
    • Tuning on Nebius Serverless Jobs, one container job per call.
    • The dashboard on a Nebius Serverless Endpoint.
    • Serving a distilled student model, which needs a Dedicated Endpoint.

Challenges I ran into

  • Reasoning tokens count against max_tokens. A call sized for a closed model's direct answer came back cut off on Nemotron. The obvious fix, turning reasoning off, cost 17 points of accuracy on preflight's review call. OpenPort now tunes both settings per call.
  • Anthropic SDK code has no drop-in path. 70 of the 207 calls in the field study use Anthropic's Messages API. OpenPort ships a small runtime, in Python and JavaScript, that translates Messages calls, tools and content blocks into Nemotron chat completions at the call boundary.
  • Proving a fix, not just making one. On the first real-repo upgrades, the fixer guessed at files it hadn't seen and ran out of tokens while reasoning. Now:
    • it sees the files beside the failing one and every path in the project;
    • a fix that doesn't pass is undone;
    • the next attempt climbs from Super to Ultra.
  • Running untrusted code safely. The public dashboard only tests code it has vetted, at pinned commits, and repo tests run in throwaway containers.

Accomplishments that I'm proud of

  • Real numbers on code I didn't write: preflight is 94% cheaper with every call passing its held-out cases, and a 79k-star repo's security upgrade is proven by its own CI.
  • Real pull requests on GitHub, opened by the product, not mocked.
  • A field study that became a feature list. It taught OpenPort to handle plain HTTP calls, class methods, monorepos and images, and to scan about 40 times faster.

What I learned

  • The tier ladder pays off. Once their prompts are tuned, most calls land on Nano or Super. Ultra earns its cost as the engineer and the judge, not as the default.
  • Held-out cases keep tuning honest. A prompt tuned only on the cases in front of the tuner can fit those cases and miss the rest.
  • Fetched sources beat model memory for anything newer than the training data: changelogs, prices and model guidance.

What's next for OpenPort

  • Run the weekly upgrade Action on real repos, and tuning on Serverless Jobs at scale.
  • An Anthropic-compatible endpoint on Token Factory would let a third of the calls I found move with no translation. It's at the top of my feedback.
  • More frameworks: the Vercel AI SDK, then LangChain.

Built With

Share this project:

Updates

Submission history