Shippies Agent

Inspiration

We are Shippies, and our name says what we care about: shipping. Code that is not delivered, and not trusted, is just cargo waiting in the harbor.

Coding agents are great at loading the boat. The trouble is the moment they announce "ready to sail": too often "ready" is only the model's own opinion, and nobody has inspected the cargo. We wanted an agent with a harbor inspector it cannot bribe: the test suite.

So we built Shippies Agent around one rule: the tests decide, not the model. The ship only leaves port when every test passes, and the agent cannot cheat its way past the inspection. It cannot rewrite the original tests, cannot slip in placeholder tests, and cannot hide a failure: when it cannot finish the job, it says so and keeps the ship in port for a human to look.

What it does

Shippies Agent is an autonomous coding agent running on Nebius Token Factory with NVIDIA Nemotron 3 Super. It has five modes:

Bug fix: reads failing tests, finds the root cause and edits the code until the suite passes.
Feature: test-first. It writes a new failing test, confirms it fails, then implements the feature.
Scaffold: builds a whole project from a natural-language spec (plan, dependencies, tests first, code, README). Our example is a FastAPI app with JWT auth.
Optimize: fixes bugs, removes dead code and reduces algorithmic complexity, proving each change with a benchmark.
Explain: a read-only guide that explains a repository and answers questions about it.

How we built it

The model works through tools (read, search, edit, run tests), and the agent controls what each tool may do:

Original tests and pytest config are protected and fingerprinted with SHA-256.
Placeholder tests (assert True, tests that import nothing from the project, skipped tests) are rejected.
An edit that breaks the syntax or increases the number of failing tests is undone automatically, and the model is told why.
If a run ends unsolved, the agent returns to the best state it reached.
Every run ends with SUCCESS, REVIEW (needs a human look) or ERROR, never a silent pass.
Optional model routing: a stronger model can take over when the agent is stuck.

We evaluated it on 15 cases: 7 hand-written and 8 real bugs reproduced from open-source projects (more-itertools, boltons, toolz, requests, click, marshmallow, sortedcontainers, tabulate). Each real case is built from the commit just before the upstream fix plus the test added by that fix.

Result: 13 of 15 solved (7 of 7 hand-written, 6 of 8 real bugs), with the original tests intact in all runs. The two failures (click and sortedcontainers) hit the 60-iteration limit and ended as REVIEW. These are single runs, and the model is not deterministic.

Challenges we ran into

Reasoning tokens eat the budget. Nemotron counts reasoning tokens against max_tokens, and a long chain of thought could cut a tool call in half. We raised the limit and made the agent ask the model to repeat a truncated call.
Agents find shortcuts. On an empty benchmark case, the agent wrote its own assert True test and almost declared victory. That led us to reject trivial tests and to stop with an ERROR when a case has no tests.
Stale line numbers. Edits by line range broke code when numbers shifted, so we added a "you must have read these lines since the last edit" rule plus automatic undo.
Hidden environment effects. Editing a copy of a src/-layout project had no effect when the library was installed, so tests ran the old code. We fixed it by controlling PYTHONPATH.

Accomplishments that we're proud of

Successfully Navigated Massive Real-World Codebases: Our agent successfully resolved complex bugs in production-grade repositories with files exceeding $5600$ lines of code.
Robust Automated Testing Loop: We engineered a resilient evaluation loop that reliably runs tests, validates fixes, and achieves definitive SUCCESS states autonomously.
Rapid Iterative Progress: We maintained relentless momentum, iterating across fifteen distinct demo milestones to continuously refine prompts and error-handling logic.

What we learned

Verification matters more than generation. Most of our progress came from making the agent harder to fool (including by itself), not from making it write more code. We also learned to report failures honestly: a REVIEW is more useful than a fake success.

What's next for Shippies Agent

Run tests in a clean container sandbox (today, a missing dependency in scaffold mode can go unnoticed), repeat runs to measure variance, and use the stronger-model escalation on the cases that still fail.