Live site · Shove the G1 in your browser · npm: golem-spec · Code · Evidence · Kiro Crew app

GOLEM: requirement 1.3 written around a Unitree G1; 36 recorded shoves, the ones from its left fall and the floor cracks

Inspiration

In the legend, a golem is clay brought to life by a written word. Robots now get brought to life by words too: a spec, and code an AI agent writes from it. Kiro writes acceptance criteria in EARS, and its docs call them "testable". For a robot, nobody actually tests them. An agent hands back a video of the robot walking, and the spec says it survives a 250 N shove. Nothing connects the two.

What it does

GOLEM is an npm package, golem-spec, plus a Kiro pack.

  1. Compile. It reads EARS lines straight from .kiro/specs/*/requirements.md. "WHEN the pelvis is pushed with up to 100 N for 0.2 s from any direction at any point in its stride THE robot SHALL stay upright for 3 s" becomes a search space (force 0–100 N, angle 0–360°, stride phase 0–1) and a monitor with a signed margin in metres.
  2. Falsify. MuJoCo 4 compiled to WebAssembly runs the same trials in Node and in the browser. GOLEM replays pinned seeds, then Latin-hypercube trials, then a cross-entropy search toward the lowest margin, then shrinks a failure to the mildest push that still breaks the line. Every trial reproduces from its seed.
  3. Gate. A Kiro Stop hook runs golem gate. Lines marked Expected_Hold must hold and Expected_Break lines must break; while any line misses, the hook blocks the agent and hands back the counterexample. A preToolUse guard stops the tuning agent from editing the requirement instead of the robot.

It runs in CI too: a reusable GitHub Action (uses: RohanGlitched/golem/.github/actions/golem-gate@main) gates every push and writes a per-requirement job summary; this repo's own CI runs it on the G1 both ways. It ships as one npm package: the compiler and runner, a golem CLI (init, check, test, falsify, replay, gate, mcp), an MCP server with 8 tools, and a Kiro pack (hooks, three custom agents, two skills, steering, a workflow), also packaged as a Kiro Power and a Kiro Crew app. npx golem-spec init installs all of it into .kiro/, with a starter cart-pole robot when the project has none.

Three apps share the skeleton (Skeleton Crew):

  • Stagger: a Unitree G1 humanoid on Unitree's published walking policy, ported to TypeScript and matching Unitree's own Python reference run to 2.5 µm.
  • Candy: a Franka Panda with a suction cup sorting Halloween candy while trick-or-treaters reach in.
  • Hover: a Skydio X2 quadrotor (MuJoCo Menagerie, unmodified) holding a point 1 m up in gusty wind, on a cascaded PD controller. Kiro built the whole app on the same package in one session: lexicon, controller, six EARS lines, tests and its studio.

What GOLEM found (all raw data in the repo)

  • The G1 on Unitree's policy falls from a 62.5 N push from its left early in the stride, yet holds 240.6 N from the front right: 36 directions × 5 phases × 3 seeds, 4,860 trials.
  • Kiro made it stronger, and GOLEM measured it. Kiro's golem-tuner agent, told to change only the reflex file and gated by GOLEM, wrote a step-into-the-push reflex. Re-measured over the same 4,860 trials: weakest push 62.5 → 101.6 N; directions that hold requirement 1.3's 100 N: 34 → 36 of 36. It gave up 23 N at the front right; we say so.
  • Candy: 57 of 228 trials break "keep every link at least 0.10 m from the hand". The safety controller stops the arm by the cup's distance and stops too late for the wrist: link 7 is closest in 48 of the breaks.
  • Hover: the drone's two lines meant to break do, at every search seed tried: a gust from about 10 N for 0.5 s (75% of its weight) blows it out of its 1 m ball, and a position estimate about 0.27 s late turns the hover into a growing oscillation. Its four lines meant to hold do.
  • Does the Stop gate stop false "done" claims? Two pre-registered pilots, 6 runs per arm. With a reachable target, no agent claimed success falsely in either arm (0/6 and 0/6) and 11 of 12 made the gate green. With an unreachable one (1.3 at 200 N), 3 of 6 runs without the Stop hook and 1 of 6 with it claimed success while an independent gate was red (p = 0.55, a pilot). The false claims were not lies: the agents trusted their own quick 24-trial green, which another search seed broke. So we changed the gate: since golem-spec 0.1.2, GREEN has to survive a second, independent search seed. Re-gated with no agent involved, 3 of those 4 controllers now come out red; one still passes, so a second seed lowers the odds of a lucky green without removing them. Both pilots and the re-check, run by run.
  • Honest negative: on the G1's real lines, guided search found counterexamples no more often than random sampling at budget 96. It wins on a synthetic 0.11% region (30/30 against 7/30). Both are on the evidence page.

How we built it

See "How Kiro was used" below. Stack: TypeScript, @mujoco/mujoco 4 (WASM), three.js, fast-check, Vite, Node worker_threads and browser Web Workers, AWS CloudFront + S3 (CloudFormation).

Challenges

  • An invalid map. The first stability map was identical in every direction. A Kiro-written audit found the push direction never reached the simulator. We quarantined the data, fixed it through a bugfix spec, and re-ran.
  • Flat robustness defeated the search. A fall scored a fixed −0.01 m, so cross-entropy search had no slope and lost to random sampling. The fix was a graded margin (height and tilt headroom), found by the falsify audit.
  • The tuning agent caught our convention bug. Kiro's tuner reported that "from behind" pushed the robot from the front. It was right; we flipped the push-angle convention and re-ran everything.
  • Kiro's Stop hook fires once per user turn (checked on kiro-cli 3.5.0), so the repair loop also lives in the tuner agent and a workflow.

Accomplishments

  • A reproducible weak side, in simulation, of a robot people buy.
  • An AI agent made a robot measurably stronger under a gate it could not talk its way past, and the whole session log is committed.
  • One package that turns any Kiro spec with numbers in it into tests, in the browser and in CI.

What we learned

EARS is already a test language; the missing piece was the physics and the search. And a gate is only as good as the semantics of "done": the gate had to learn Expected_Break before an agent could ever finish green.

What's next

Python/ONNX policies over stdio, a perturbation library (payload, latency, friction, motor weakness) so a new robot needs only its MJCF and a controller, and a larger agent-gate study.

How Kiro was used

GOLEM was built in Kiro CLI 3.5, headless (kiro-cli chat --no-interactive --output-format stream-json), one prompt per session. The raw log of every session is committed in evidence/kiro-sessions/ (the 40-plus headless runs we kept, about 2,700 credits recorded in their own logs), with its prompt, so every claim below can be checked.

Specs, and when we used which mode. 16 specs (4 of them bugfix specs, 3 of them the robots' own requirement files), 217 requirements, 2,157 EARS acceptance criteria, 273 correctness properties, 728 of 736 tasks done (a 17th spec, golem-cloud, is written and not started). Spec mode wrote each spec's requirements, design and tasks. For implementation we switched to Kiro's default mode, one long sequential session per spec: spec-mode task dispatch died under rate limits when three sessions ran at once, and the sequential session was cheaper (38 credits for the whole simulator spec and its 204 tests, against 139 for the EARS compiler in spec-mode dispatch).

Audits that became bugfix specs. The EARS compiler, the simulator and the falsifier each got an independent review with probe scripts against real inputs (docs/reviews/), and each review became a Kiro bugfix spec with current, expected and unchanged behaviour. Since then a custom Kiro agent, golem-auditor (Sonnet 5.5; it may write only to docs/reviews/), does this job: its audit of the Expectation-aware gate (docs/reviews/gate-audit.md, 7 probe scripts, 22.8 credits) became a fourth bugfix spec, golem-gate-hardening. They caught real bugs: the push direction never reached the simulator (our first stability map was identical in every direction and is quarantined), the simulator reset a fallen robot and hid the fall, and the cross-entropy search lost to random sampling on a flat margin.

Steering. Nine steering files: product, tech, structure, interfaces (the cross-module contracts, frozen), ears-for-robots, mujoco-wasm, testing, design-language and experience (the binding signature moments of the studio).

Hooks. In this repo: a Stop hook that blocks the agent until the typecheck passes, and PostFileSave hooks that run the unit tests of a saved engine file and compile robot requirements on save. In the shipped pack: check requirements on save, falsify the controller on save, the Stop gate (golem gate --kiro), a test run after every spec task (PostTaskExec), and a manual "Summon the ghosts". We measured how Kiro's Stop hook behaves headless: a block continues the agent, once per user turn, so the repair loop also lives in the tuner agent and a workflow.

Custom agents. golem-tuner may edit only controller files; its own preToolUse guard stops it editing a requirement, and its shell allow-list is pinned. golem-skeptic is read-only and proposes harsher believable requirements. golem-scribe turns plain words into EARS lines that compile. All are validated with kiro-cli agent validate.

MCP. GOLEM ships an MCP server with 8 tools (list, check, run, falsify, get_counterexample, replay, explain_margin, propose). This repo's .kiro/settings/mcp.json runs one server per robot, so Kiro talks to the G1 and the Panda directly.

Skills, workflow, Power, Crew app. Two skills (robot-ears, falsify-and-fix), a falsify-and-fix workflow that repeats tuner then gate, the pack as a Kiro Power, and a Kiro Crew app ("night watch") that runs the skeptic on a cron and writes its proposals.

Property-based testing. 543 fast-check property assertions in 415 test files, each tied to a numbered correctness property in a design.

The third robot, in one session. One Kiro CLI session (default mode, Sonnet 5.5, 107.8 credits, 9 commits) built apps/hover on the package: it fetched the Skydio X2 from MuJoCo Menagerie at a pinned commit, wrote the lexicon, a cascaded PD controller, six EARS lines with an Expectation table, the tests and the studio page, and measured the lines with the real CLI over six search seeds before choosing the numbers. A later session (55.5 credits) added the gate's confirmation seed and re-gated the experiment's controllers.

The session we are proudest of. golem-tuner (Opus 5.5, 13.6 minutes, 11.3 credits) was told to make requirement 1.3 hold, touching only the reflex file. It used GOLEM's MCP tools before writing code. Its first reflex made 1.3 hold and broke the slope and latency lines; GOLEM caught both. A new search seed broke the next version at 79.9 N; it lowered the gain to 0.8 and the line held under four search seeds. It then stopped honestly with the gate still red, explaining that the gate did not read the Expectation table yet, that the MCP server was serving a stale controller, and that our push-angle convention looked backwards. All three were real; all three are fixed. Re-measured afterwards over 4,860 trials, its reflex raised the weakest push from 62.5 N to 101.6 N.

What went wrong with Kiro, honestly. One kit edit emptied a 161-line test file; review caught it and the file was restored. Spec-mode runs were throttled above three parallel sessions. /usage lags, so we tracked credits from the stream logs.

Third-party assets

Unitree G1 model and walking policy (BSD-3-Clause, unitree_rl_gym), Franka Panda model (MuJoCo Menagerie, Apache-2.0), MuJoCo (Apache-2.0), B612 and Gloock fonts (SIL OFL). Everything else was written during the hackathon.

Built With

Share this project:

Updates

Submission history