Mjolnir

Runs broken code against a locked test on fast Nemotron Nano first, then selectively pulls 2026 docs via Tavily for API changes, or escalates to Nemotron Super/Ultra only when math gets deep.

Comment

Inspiration

Recent research into autonomous coding agents reveals an expensive paradox: throwing every task at a massive 550B frontier model burns up to 15x more money, yet still fails more than 57% of the time whenever an SDK changed its API after the model's training cutoff. No amount of parameter scaling can hallucinate a 2026 function signature that didn't exist in 2024.

At the same time, naive coding agents that search the web on every single error suffer an accuracy penalty of up to 39% on pure mathematical and algorithmic problems due to noisy HTML context pollution.

We asked ourselves: Why are we routing models based on prompt length or guessing when to search? In real software engineering, you don't guess—you run the test suite. We built Mjolnir around a first-principles rule: Test first, escalate second.

What It Does

Mjolnir is an autonomous, cost-governed coding agent on Nebius Token Factory that solves code defects through a two-dimensional escalation matrix:

  1. The Un-Bribable Judge: Every task begins by running against a locked, read-only unit test with a strict SHA-256 integrity hash (os.chmod read-only sandbox). The agent cannot alter assertions or fake passes.
  2. The Fast Probe: The task runs first on high-velocity NVIDIA Nemotron-3-Nano. If the sandbox exits with code 0, the verified git diff is returned immediately at sub-cent cost.
  3. Deterministic Error Distillation: When a test fails, Mjolnir parses the traceback into a structured failure classification:
    • KNOWLEDGE_GAP (AttributeError, ImportError, ModuleNotFoundError): The failure is an API or SDK change. Mjolnir triggers a 3-stage Tavily documentation pipeline (POST /search $\to$ POST /extract $\to$ POST /map) to inject authoritative 2026 signatures, and patches the file using Nemotron Nano.
    • REASONING_GAP (AssertionError, algorithmic edge cases): The failure is pure logic. Mjolnir skips web search completely (zero context pollution) and escalates compute directly to NVIDIA Nemotron-3-Super (120B) or Nemotron-3-Ultra (550B).

$$ \text{Cost per Verified Fix} = \frac{\sum (\text{Inference Tokens} + \text{Tavily API Calls})}{\text{Tasks with Exit Code } 0} $$

Across our 10-task benchmark, Mjolnir achieved a 100.0% verified solve rate while slashing cost per verified fix by 69.8% compared to an always-on frontier baseline.

How We Built It

  • Inference Fabric (Nebius Token Factory): We implemented an OpenAI-compatible async inference client in mjolnir/nebius_client.py with live /v1/models discovery, real-time token tracking, and sub-second hardware receipts.
  • Model Hierarchy (NVIDIA Nemotron 3):
    • nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B: Used as the initial velocity probe and grounded API patcher.
    • nvidia/NVIDIA-Nemotron-3-Super-120B-A12B: Invoked for structural type coercion and dynamic programming.
    • nvidia/NVIDIA-Nemotron-3-Ultra-550B: Reserved exclusively for deep graph and logic invariants.
  • Live Grounding Engine (Tavily Partner Integration): Built a 3-stage progressive retrieval pipeline:
    1. POST /search: Queries official developer documentation domains.
    2. POST /extract: Pulls clean markdown function signatures, stripping raw HTML boilerplate.
    3. POST /map: Crawls parent changelogs to capture adjacent breaking changes across package trees.
  • Interactive Split-Screen Workbench: Built a live browser workspace in vanilla HTML/CSS/JS (static/app.html) providing live side-by-side execution assays, terminal telemetry, and git diff inspection.
  • Motion Graphics & Demo Production: Produced a 66-second 60fps launch film using deterministic Canvas animation, ElevenLabs voiceover, and subframe motion blur (ffmpeg tmix).

Challenges We Faced

  1. Subprocess Pipe Deadlocks on Windows: When streaming thousands of raw 1080p frames through FFmpeg subprocess pipes, the Windows OS 4KB stderr buffer deadlocked synchronous pipes. We re-engineered the rendering harness with non-blocking file descriptors and 4 parallel worker processes.
  2. Evaluating Without Cherry-Picking: Single-prompt anecdotes are misleading. We constructed a strict 10-task benchmark (6 post-cutoff 2026 API shifts + 4 complex algorithmic invariants) and measured 5 experimental arms live against real endpoints to obtain verifiable telemetry.
  3. Preventing Agent Self-Deception: Language models frequently attempt to bypass broken code by commenting out test assertions. Enforcing filesystem-level read-only permissions and SHA-256 pre- and post-execution checks was essential to keep the agent honest.

Accomplishments We're Proud Of

  • 100% Verified Solve Rate: Solved 10/10 tasks in live testing, including post-cutoff 2026 API breaking changes where ungrounded Nemotron-Ultra scored 0/6.
  • Deterministic Receipts: Zero mock data, zero dead buttons, and zero hand-waving. Every metric on the dashboard is backed by actual unit test exit codes.
  • True Cost Efficiency: Proved that selective grounding with a 30B model can beat a raw 550B model at less than a third of the operating cost.

What We Learned

We proved empirically that bigger models cannot invent unknown facts, and more retrieval is not always better. The most effective agentic architecture isn't a complex swarm of autonomous personas—it is deterministic routing governed by locked tests and high-signal documentation.

What's Next for Mjolnir

  • Multi-file repository bisection across Git commit history.
  • Integration with GitHub Actions and GitLab CI to automatically generate cost-optimized PR remediations.
  • Expanding the 2D escalation grid to support local quantized Nemotron edge instances.

Built With

Share this project:

Updates

Submission history