Release Gate
Minimizer Ledger
Job Ledger
Comparison Curves
Video Simulator
Original vs Minimized Scenario
Failure Atlas
Test Lab

Inspiration

We had recently competed in the ABB Hackathon 2026, and won second place! The concept involved building a machine learning predictive maintenance platform for manufacturing operations. We were team Odin then, and our product was Thorbolt! This cemented our theming, but also something more.

While we were in Fort Smith, Arkansas, we had a chance to tour the ABB factory there and attend their AI Accelerator conference. It was exciting to see how the motors themselves got made: pressers, cutters, conveyors all working in perfect harmony with person and machine. We also saw these gigantic robotic arms grab and pick up parts in a deterministic way. These were heavy motor component pieces, and we saw first-hand the application of robots firsthand.

The conference was all about AI solutions. The concept of "digital twin" and "smart factory" was brought up more than a few times. Reflecting on the week, we saw a business case. You have a robot performing a task in a simulated world, and you can change the physics of that world: the object's mass, the friction of the table, how late the sensor data arrives.

Within the ranges a real robot would actually face, which of those changes breaks the policy, and how small can that change be?

Nobody can test every combination, and nothing in a typical pipeline stops a release when one is found. Skuld searches that bounded envelope for the smallest change that breaks the policy, minimizes it to the variables that matter, stores it as a replayable regression test, and blocks the release until it passes. The envelope itself is kept honest: the perturbation ranges are locked before any search and checked against the manufacturer's documented limits, with a human approving every constraint.

What it does

Skuld is a release gate for robot policies. You give it a policy, a simulated task and a locked set of world variables, in our case object offset, object mass, table friction and observation delay, each bounded by the manufacturer's documented limits. Within a fixed budget of episodes, Skuld searches for the smallest change to that world that makes the policy break a rule. A deterministic oracle reading simulator state decides every pass and fail; no model output touches that path.

When it finds a failure, Skuld minimizes it: each perturbed variable is reverted to nominal and only what the failure needs is kept. Our nearest failure reduced to a single variable, 140 ms of observation delay, which sent the fingers into the floor. That minimized failure is stored as a regression test with its scenario, seed, simulator version, policy hash, trajectory and video, replayable byte for byte.

From then on every candidate release must pay it. The gate runs the nominal, stored-regression and held-out suites and declares a candidate eligible only when all three pass. Anything missing, stale, corrupt or unknown fails closed. Our first controller was blocked; a repaired version passed all ten tests; the next search round broke it again at 182 ms of delay, and that failure is now the next test.

Around the loop, AI proposes and humans approve. Tavily fetches the manufacturer's limits, Nemotron on Nebius Token Factory proposes hypotheses and extracts constraints, a human signs off, and physics decides. Everything is in simulation, on one scene and one classical controller, and the results claim nothing beyond that tested envelope.

How we built it

Spec-driven development for Claude Code, with sequential gate tasks. Each gate task acts like a stepping stone to the next, building out features one at a time, and then allowing us to test/re-test between each gate. Once successful, we go to the next gate task.

Challenges we ran into

We did this on a Windows laptop with no NVIDIA GPU and decided early that we would not spend money on cloud compute. So no Isaac Lab, no GR00T, no GPU rollouts. Everything runs in CPU MuJoCo inside WSL2. Honestly it was fine. A 15-second episode without rendering takes about half a second, so a budget of 25 episodes is cheap and anyone can rerun our results on a laptop.

The thing we worried about most was fooling ourselves on the random-vs-active comparison. It is very easy to make a search method look good against a weak baseline. We ran and archived the random baseline before the search code existed, pinned the protocol and seeds, and made the scripts refuse to run if the p changed. The final rerun matched both archives byte for byte.

The first live Nemotron call got the payload number right but paraphrased threw it out. We fixed the prompt and the second call went through. We kept the reject because it is a better demo of the design than the one that worked. We also learned that you get the same answer twice.

Scope was a constant fight. One robot, one task, one four variables, whole loop ran end to end. Every day there was a good reason to add something.

Two of us shared a keyboard while the other reviewed, and the long runs went in the background. "Eligible" means it passed our test suite. It does not mean safe, and we never say safe.

Accomplishments that we're proud of

The loop actually closes. Search finds a failure, the minimizer strips it to one variable, 140 ms of observation delay with the fingers dragging on the floor, three replays identical to the byte. That failure becomes a stored test, the gate blocks the release, a fix passes, and the next round breaks the fix again at 182 ms. We watched that happen on real runs, and the negative result at the end is the project working, not failing.

The comparison is honest. Random found 3 failures in 25 episodes, first at episode 16. Our search found 6, first at episode 6, and closer to nominal. Same domain, same budget, same oracle, baseline archived before the search code existed, and the final frozen rerun reproduced both archives 25 of 25, byte for byte. Every number in the README and video comes from a file in the repo.

The gate fails closed and we tried hard to break it. Nine injected faults, including a truncated recording, a stale policy hash, and a verdict edited to PASS with the violations still attached. None passed. The same discipline runs through the AI integrations: the Nemotron and Tavily calls are real, and the model's first wrong answer sits in the evidence folder next to the right one because the validator caught it.

What we learned

The boring parts carry the interesting parts. The deterministic oracle, the hashes on every artifact, and a gate that fails closed are what made it safe to let a language model propose anything at all. Without them a Nemotron hypothesis is a guess. With them it is a guess that gets tested and either becomes evidence or gets thrown out. The first live call proved the point: it paraphrased a quote, the validator rejected it, and temperature zero turned out not to mean reproducible. Archive what the model returned and hash it.

A failure is only useful once it is small. A four-variable failure is a shrug. The same failure reduced to 140 ms of delay is a bug report an engineer can act on and a regression test a release can be held to.

Passing the suite is a checkpoint, not a finish line. The repaired controller passed all ten tests and was broken again within minutes by the next search round. That is the argument for running this continuously instead of as a one-time audit, and it is why we froze scope early, archived baselines before building what we compared against them, and wrote down the words we were not allowed to use.

What's next for Skuld

Put a learned policy under the gate. The adapter boundary takes any policy that maps observations to actions each step, so a GR00T checkpoint on a GPU box is the obvious next candidate. The regression tests we already found against the classical controller apply to it unchanged, which is the point: the debt belongs to the task, not the policy.

Make the comparison statistical. Right now it is one run per strategy at a budget of 25. Repeated runs at larger budgets, plus a second simulator behind the same adapter, Isaac Lab first, would turn "active search found closer failures than random" into a claim with error bars, and would let us test whether a counterexample found in one simulator survives in another.

Run it as CI. Hook the gate to a policy repository so every candidate pays the stored regression suite before merge, with the search running on a schedule to keep adding to the debt. Cosmos as a disagreement signal between predicted and simulated futures is the stretch version of the same idea: find the places where the world model and the physics disagree, and test there first.