GIF
real run!
overview

Inspiration

The most capable robot behaviors in the world come from a small number of excellent reinforcement learning researchers. That's how it should be: they push the frontier.

But most of what people want robots to do isn't frontier work. Walk backwards. Push this ball into that zone. Cross rough ground without falling. Each of these still needs someone to hand-write a reward function, train, watch the robot do something strange, adjust the weights, and train again, often for days.

That bottleneck shuts out the people who could get the most from robots. A biologist studying gait, a farm that wants a robot patrolling its rows, a logistics team that has never thought of robots as an option: they can all describe exactly what they want, but none of them can write PPO reward shaping. Meanwhile, the RL engineers who can are spending their talent on the thousandth "walk to the ball" script instead of the frontier.

So we asked whether the reward engineer could be replaced by a loop. AI agents would write rewards, train them, watch the result, and fix what went wrong, the way a developer fixes code.

What it does

Rewardsmith takes a request like "push the red ball into the green zone" and returns a simulated Unitree Go1 quadruped that does it. You get a video, measurements, and the reward code that produced the behavior.

For each request, Rewardsmith:

Builds the world. Claude turns the request into a validated scene: the terrain, plus any balls, boxes and target zones, with their sizes, colors and placement.
Writes the test first. Before any training, Claude writes a pass/fail success check over measured rollout metrics. An example is "backward velocity faster than 0.3 m/s, without falling," or "the ball ends in the zone in most episodes." The check is locked for the whole run, so the system can't move the goalposts later.
Drafts competing rewards. Claude writes several reward functions in Python, each with a one-line idea. Every draft is compiled and smoke-tested in the simulator before it's allowed to train.
Trains them in parallel with PPO on a single GPU. Every candidate builds on a pretrained walker, so it can already walk from the start.
Coaches live. A fast Claude model reads each training curve as it comes in and comments on the board. It can also stop candidates that are going nowhere.
Judges. The success check runs on the measured metrics. A Claude critic looks at frames of the rollout and scores it 0–10. It writes what to change and flags reward hacking: a robot that satisfies the math without doing the task. A fall or a hacking flag caps the score, and that cap is enforced in code.
Revises like a developer. Factory Droid, an AI coding agent, opens the best rewards alongside the critic's notes, the measurements and the history of earlier attempts. It makes targeted edits instead of starting from scratch. Each change comes back as a diff with a one-line reason, and the revised rewards train again.

It's conversational. After a run you can say "lift your feet higher" or "now add a box." Rewardsmith either revises the reward and continues training from the saved policy, or rebuilds the scene if the world changed. Instant mode aims for a first video in under two minutes. Effort mode trains four competing rewards written by Claude Opus, capped to finish in under five.

Everything happens on a live board, which shows:

what's happening, in plain words
training curves
the success check, clause by clause
the reward code, with a "What changed" view of every diff and the reason behind it
the coach's feed
a hall of fame of reward hacks the critic caught

It's built for researchers in other fields who want robots in their work, for industries that have never considered robotics, and for RL experts whose time belongs on harder problems.

How we built it

Separate Claude agents build the scene, write the success check, write rewards, judge rollouts and coach training. Opus 5.5 writes and critiques, and Haiku 5.5 runs the live coach. Replies stream back as JSON, so reward ideas show up on the board while Claude is still writing.
Factory Droid edits the rewards. Each revision runs in its own workspace holding only the reward files, the critique, the metrics and earlier attempts. Droid runs at low autonomy with the shell turned off, and we diff every file it changes.
Training uses MuJoCo Playground's Go1 environment on MuJoCo Warp and JAX, with Brax PPO, on one Runpod RTX 4090. Every reward returns named terms, so we can check each one for NaNs, chart it, and show the critic which term the policy exploited.
A walker we pretrained once (213M steps in 330 seconds) moves the legs. Each task's policy only learns where to walk.
To keep runs short, trainers start warm, all evaluation runs in one batched compile, and we patched out a duplicate Brax compile.
A FastAPI server and a worker share a SQLite database, and the board is React and TypeScript. A fake simulator and a fake Claude let the whole loop run on a laptop, and a GPU regression suite checks that known rewards still train.

Challenges we ran into

Our first warm start fine-tuned the pretrained walker directly, and on "move the ball to a target" every candidate learned to stand still. They reached the ball in 0% of episodes, while training from scratch reached it in 28–100%. Moving exploration into command space fixed it: the policy picks velocities and the frozen walker moves the legs. That steer mode is now the default.

Success checks were harder to get right than rewards. Ours couldn't tell standing on the hind legs from sitting back, didn't read "move the ball to the destination" as a push, and included "stop and hold" conditions no policy could ever meet. Each needed its own fix.

Droid needed enough context to make a real fix but no way to touch the rest of the project. Each revision gets its own workspace with only the files it needs. The shell is disabled, and any request for more permission is refused.

Compile time dominated. Each round paid about 90 seconds of PPO compile and about 50 seconds of Opus writing before training started. Pre-warmed trainers and a one-round Effort mode cut that down. So did removing a duplicate compile that cost 21 seconds per run.

Training isn't deterministic. The same code and seed could end 10% lower or collapse entirely, so our regression suite compares the median of three runs.

Two policy bugs were hard to spot. Scene policies saw the episode time in their observation, but training resets every 6.4 seconds, so any later time was one they had never trained on. Deterministic rollouts also made the spin policy fall, so videos now sample actions the way Brax's own evaluation does.

MuJoCo Warp printed a solver warning on every physics step and filled 13 GB of disk with logs in 21 minutes. We now clear only those warning bits.

Accomplishments that we're proud of

The full loop runs end to end on a real GPU with real Claude calls. Plain English goes in, and a scored video of a trained robot comes out, with no human touching reward code.
A run passes only on a test written before training, and code enforces the critic's score caps. Reward hacks the critic catches end up on the board, not in a shipped policy.
Every reward revision can be reviewed. Droid returns each change as a diff with a one-line reason, so you can see why the robot's behavior changed.
The pretrained walker didn't fall once across 64 randomized robots, and it tracks a 1.0 m/s forward command at 0.97 m/s.
Walk backwards, walk forward and spin in place each train in about four minutes with no falls, reaching −0.50 m/s, +0.59 m/s and 1.48 rad/s.
Follow-ups continue training from the saved policy instead of starting over.
Over the weekend we made 373 commits and wrote about 11,200 lines of Python and 386 automated tests.

What we learned

The space a policy explores in matters as much as its reward. Steer mode turned 0% success into a working default.

Writing the test first works for RL too. Locking the success check before training was our best defense against an optimizer that finds shortcuts.

Tuning a reward is code editing: read the critique, make a targeted change, show the diff. That's how a human engineer does it, and it's what a coding agent like Droid already does well.

A vision critic needs numbers. Still frames can't show speed or leg jitter, so the critic also gets the metrics and the reward curve. Jitter is still on our list, because no single frame shows it.

Fixed costs dominate short RL loops. Compile time and process startup mattered more than training itself.

Structured outputs still need defensive code. Opus sometimes put code in the "idea" field and prose in "source," so the orchestrator swaps them back.

What's next for Rewardsmith

A fallback editor. When a Droid revision fails, hand it to Claude.
Bring your own robot. The reward contract only reads robot state, so the method is robot-agnostic by design. Next we want to accept any MuJoCo robot model, whether an arm, a humanoid or a brand-new form factor, and generate its state description automatically.
Sim to real on a physical Go1, starting with behaviors the walker already handles robustly.
A skill library that saves learned behaviors and composes them into bigger tasks.
Public benchmarks: success rates, rounds to success, GPU cost per behavior, and how often the critic agrees with the check.
Multi-user support: access control, multiple workers, and Claude API cost tracking.