Shipwright

The problem

Writing code is now the cheap part. What decides whether software can actually ship is everything after the code exists: review, security analysis, test coverage, verification — and a human who stays accountable for the release. Teams either skip these steps or drown in them.

What Shipwright does

Shipwright runs the post-code lifecycle as a nine-stage agent pipeline on Nebius Token Factory, with each stage using the NVIDIA Nemotron model tier that fits its job:

• intake (deterministic) — inventories the target service
• plan (Nemotron 3 Nano) — classifies the service, names its risk areas
• review (Nemotron 3 Super) — correctness and design review with file-level evidence
• security_screen (Nemotron 3 Super as a prompted guard classifier + Nano analytics) — first-pass hazard screening
• security_deep (Nemotron 3 Ultra) — evidence-based deep security review that traces data flow
• tests (Nemotron 3 Super) — specifies tests for uncovered behavior
• fix (Nemotron 3 Super) — minimal fix proposals for every finding
• verify (deterministic) — actually runs the target's pytest suite
• release_gate (Nemotron 3 Ultra) — synthesizes a ship/hold recommendation for a human approver

The last gate is human by design: the pipeline recommends, a person approves. Every analytic stage must return strict JSON matching its schema — a malformed answer fails the stage loudly instead of being silently coerced.

How it was built

• Python, OpenAI-compatible Token Factory client with per-tier model routing (TF_MODEL_* env overrides)
• The demo target is NoteForge, a small FastAPI notes service with deliberately seeded weaknesses (broken access control, injection-prone queries), so the pipeline has real problems to find and evidence to cite
• Model discipline was validated by a smoke gate against the live catalog before any pipeline work: every tier must answer in its expected format — which is how we caught that Token Factory serves no dedicated guard model, so the guard role became a prompted classification on Nemotron 3 Super, gated by the same smoke suite

Challenges

• Token Factory serves no dedicated guard model (verified against the live /v1/models catalog), so the security screen's guard check was redesigned as a prompted one-line classification on Nemotron 3 Super, beside a JSON hazard pass on Nano
• Reasoning models intermittently return empty content when the reasoning pass consumes the token budget — the client treats an empty answer as a failed call and retries, and token budgets were raised where outputs run long (deep review)
• Provider variance in JSON discipline required a tolerant-but-strict client parser (fences/preamble tolerated, schema enforced)

Accomplishments

• A full lifecycle pipeline that ends in evidence a human can actually review: findings cite files and code, fixes are minimal, and the release recommendation lists its reasons
• Live end-to-end run on real Token Factory models (2026-10-24): the pipeline blocked the deliberately vulnerable sample at review, flagged it unsafe at the guard screen, evidenced 4 security issues in the deep pass, proposed 7 fixes, ran the real test suite (5 passed), and held the release for a human
• Deterministic stages (intake, verify) do not spend model calls; model calls are spent only where judgment is needed

What's next

• SARIF export of findings, PR-comment integration, and a Duo-style flow definition for teams that want the same lifecycle inside their existing platform

Built With

• fastapi
• nebius-token-factory
• nvidia-nemotron
• pytest
• python