SpecFork

• 

• 

• 

• 

• 

• 

• 

• 

Inspiration

Give a one-sentence ticket such as "split a restaurant bill evenly among n people, rounded to cents" to several developers, and the programs you get back disagree:

• who pays the leftover cent;
• what happens with zero people;
• how negative totals round.

A coding agent hits the same fork, picks one branch silently, and you find out in production. Asking the model "what's ambiguous here?" produces speculative questions, and nothing shows which ones actually change behaviour. We wanted questions that are backed by executed evidence, plus a way to measure whether answering them helped.

What it does

You describe a Python function in plain English. SpecFork then runs these steps:

1. Frame. NVIDIA Nemotron 3.5 Lightning (reasoning off) proposes a typed signature and 14–18 edge-case probe inputs. Deterministic, type-driven fuzz probes are added to them.
2. Fan out. Lightning writes N (default 24) independent implementations: reasoning off, temperature 1.0, distinct seeds.
3. Execute. Each implementation passes an AST allow-list. It then runs on every probe in an isolated sandbox subprocess.
4. Cluster. Implementations with identical canonical outputs on every probe form one behaviour. Code picks witness inputs that separate the behaviours. No model judges the code; the outputs decide.
5. Explain. NVIDIA Nemotron 3 Ultra (reasoning on, medium_effort) turns each disagreement into a plain-English question. Each answer option shows the executed witness input and what each behaviour returned. Ultra also flags splits that look like bugs.
6. Silent-assumption review. Ultra (reasoning on) looks for consequential decisions that the spec never states but that all or most implementations made the same way. Example: a date spec where every implementation assumes MM/DD/YYYY. These are offered as Keep or Change to.
7. Resolve.
   • Your answers select the surviving behaviour, with compatibility checks between answers.
   • Code, not a model, generates pytest tests from that behaviour's executed outputs. An overridden assumption unlocks the cases it affects.
   • Ultra rewrites the spec so that it encodes only the points you decided.
8. Re-fork. N fresh implementations of the clarified spec run against the locked tests. SpecFork reports agreement with your intent before and after.

You can export the clarified spec, the reference implementation and tests.py. The Under the hood panel shows every Nemotron call made on Nebius Token Factory: model ID, reasoning on or off, latency, tokens, cost and a reasoning excerpt.

The live run shown in the demo video (run AjBp-qf98d19, shipped as the bill-split sample run, so you can open it in the app):

• 24 implementations, 20 probe inputs, 10 distinct behaviours, 2 clarifying questions and 3 silent assumptions;
• 52 Nebius Token Factory calls, $0.037 in total, 53 s of pipeline wall time;
• re-fork agreement with the chosen behaviour went from 17% (4 of 24 original implementations) to 77% (17 of 22 fresh ones).

This is one run, not a statistical claim; the benchmark results are below.

How we built it

NVIDIA open models, all served by Nebius Token Factory through its OpenAI-compatible chat completions API at https://api.tokenfactory.nebius.com/v1:

Role	Model ID	Settings
framing, fan-out, re-fork	nvidia/Nemotron-3_5-Lightning	reasoning off (chat_template_kwargs.enable_thinking: false); fan-out at temperature 1.0 with distinct seeds
explain, silent-assumption review, clarified spec	nvidia/Nemotron-3-Ultra-550b-a55b	reasoning on with chat_template_kwargs: {enable_thinking: true, medium_effort: true}; one retry with reasoning off

Architecture.

• Backend: FastAPI on Python 3.12, with a small async httpx client (specfork/llm.py, a provider-neutral router whose default profiles are NVIDIA Nemotron on Nebius Token Factory).
  • The client handles bounded concurrency and retries with exponential backoff that honours Retry-After.
  • It records per-call telemetry and separates reasoning_content from the answer.
• Orchestration is in specfork/pipeline.py. The sandbox is specfork/sandbox.py plus specfork/_harness.py. Clustering and witness selection are in specfork/cluster.py.
• Ways to use it: the web app; a specfork CLI (fork, resolve, doctor, ci); an MCP server, so coding agents can call it before they write code; a GitHub Action; and a Python API (from specfork import Client).
• Live progress streams to the browser over Server-Sent Events.
• Frontend: React, TypeScript, Tailwind CSS and Vite.
• Hosting: Google Cloud Run, running a container built from the repo Dockerfile. All inference runs on Nebius Token Factory.

What decides and what writes. Models propose inputs, write code and phrase questions. Code clusters, picks witnesses, generates the tests and measures agreement, so a model's prose cannot fake agreement.

Security. The spec is untrusted user input and the generated code is untrusted model output, so the design treats the code as hostile:

• AST allow-list, proxied stdlib modules and restricted builtins;
• python -I -S subprocess with a stripped environment;
• rlimits for CPU, memory, file size and file descriptors, plus per-call and per-process timeouts;
• a fresh network namespace where the kernel allows it;
• a non-dumpable server process and a non-root container;
• per-visitor rate limit and a daily USD budget cap.

Details and the tests behind them: docs/SECURITY.md.

Challenges we ran into

• Ultra with reasoning on used up its whole output budget.
  • On our explain prompt, Ultra spent all 12,000 completion tokens (32.7 s) on reasoning and returned no final content, so the JSON was empty.
  • Fix: medium_effort: true from the Ultra model card, plus a retry with reasoning off. Explain then took about 10.6 s and about 2.5k completion tokens.
• Reasoning cost in framing. Lightning with reasoning on took 64.3 s (5,470 completion tokens) to frame one spec. With reasoning off it took 4.9 s (1,153 tokens) at acceptable quality, so framing now runs with reasoning off.
• Shared model bias. If every sample resolves an ambiguity the same way, there is no disagreement to execute. We added the silent-assumption review for this case. It is judged by the model, not executed, and the UI labels it differently.
• Sandboxing model-written code in a public demo.
  • We needed isolation strong enough for prompt-injected specs without breaking legitimate stdlib use, which meant several layers.
  • Some legitimate code is still rejected. In the recorded run, 2 of 24 implementations failed to load because of from typing import list.
• Re-fork results vary from run to run. Agreement after clarification depends on how well the clarified spec pins the chosen behaviour. Our recorded runs differ, so we report the single run above only as an example and rely on the benchmark for claims.
• Evaluation findings: Clarification roughly doubles intent alignment. SpecFork ties a well-prompted ask-first baseline: it won 8 items, lost 9 and tied the rest. That is one run of 26 items, so we do not claim it is better on alignment.
• Where SpecFork wins outright: the deciding behaviour only shows up when code runs (page_count, cart_discounts: 12/12 vs 0/12).
• Where it loses: the owner wants a behaviour no sampled implementation produced (moving_average). A free-text "other" answer is the next step.
• What only SpecFork provides: an executed witness for every question, deterministic locked tests and a measured re-fork.
• Caveats: 8 items were used during development and the product owner is simulated by an LLM. Full details are in docs/EVALUATION.md.

Accomplishments that we're proud of

• Every clarifying question is backed by an executed witness input and the concrete output of each option, so no question is speculative.
• Tests are generated deterministically from executed behaviour, not written by a model.
• Measurement is built into the product loop: agreement is shown before and after the re-fork.
• Cost: about $0.02 per full run in the recorded example, which makes "sample dozens of programs per request" viable as a product.
• Transparency: every Nebius call is visible in the UI with its model, reasoning flag, latency, tokens and cost.
• A side contribution to NVIDIA tooling, kept separate from this project:
  • We found and fixed a NeMo Agent Toolkit bug where thinking fails validation for Nemotron 3 / 3.5 model IDs, including the Nebius IDs.
  • It is a patch with tests. A PR is not opened yet.
• Offline test suites: 53 backend tests and 30 frontend tests, passing on 2026-10-30. They use a scripted fake LLM and no network.

What we learned

• Disagreement between many cheap samples, once executed, is a much stronger ambiguity signal than asking one model what is ambiguous. It is still blind to assumptions that every sample shares.
• Reasoning budgets matter as much as model choice:
  • reasoning off for high-volume generation;
  • medium-effort reasoning for the few calls where judgement matters;
  • always a fallback path.
• Token Factory's OpenAI-compatible API made the integration a single small httpx client. chat_template_kwargs passes straight through, and reasoning arrives in a separate reasoning_content field.

What's next

• Languages beyond Python, and multi-function or class-level specs.
• Executing the silent-assumption alternatives too (generate the "Change to" variant and run it), instead of relying only on model judgement.
• Direct hand-off to coding agents: a CLI or MCP tool that returns the clarified spec and tests.py before the agent writes code.
• A larger benchmark that is not written by us, and repeated runs with confidence intervals.

Built With

• docker
• fastapi
• google-cloud-run
• httpx
• nebius-token-factory
• nemotron-3-ultra
• nemotron-3.5-lightning
• nvidia-nemotron
• pydantic
• pytest
• python
• react
• server-sent-events
• tailwind
• typescript
• uvicorn
• vite