MicroDuck Lab

Teach a real robot RL in your browser. An AI agent drives the same controls, trains on your GPU, and brings the weights back. 53 WebMCP tools, real MuJoCo, real Microduck.

Comment

Learn reinforcement learning by teaching robots. Live: microducklab.com · Repo: github.com/SynthReality-Labs/microduck-lab · Video: youtu.be/eZY9nkB0mYE · by SynthReality Labs

Inspiration

Ten thousand people just bought a reinforcement-learning robot.

Microduck — the $399 open-source biped from Pollen Robotics and Hugging Face — passed 10,000 units and $5M within days of launch, taking $2.6M in the first 24 hours. Demand was heavy enough that the Christmas delivery promise is gone; new orders now quote four to six months.

So there are thousands of people who have paid for a robot whose entire premise is train in simulation → deploy → refine → retrain — and who will spend months waiting, many having never trained a policy in their lives.

RL is taught badly because it is taught abstractly. The loop that actually produces understanding — change a thing, watch the robot fail differently, work out why — is buried under CUDA installs, YAML files and wandb dashboards. Newcomers read about reward hacking. They rarely watch it happen to something they configured.

MicroDuck Lab is that loop, in a browser tab, with an AI agent beside you. The real robot model, the real published policies, real MuJoCo physics, and 53 WebMCP tools. No install, no GPU, no account.

Why this use case is a strong fit for WebMCP

Because a human and an agent are good at opposite halves of RL diagnosis, and until now there was no way to put them in the same room.

You look at a rollout and know instantly that that gait is ugly, I don't like that fall, it should look more deliberate. That judgement is pre-verbal, immediate, and nearly impossible to write down as a metric.

An agent can read thirty metrics, a reward decomposition, a curriculum state and a config diff at once, and cross-reference them against a documented catalogue of known failure modes. That is tedious and error-prone for a human.

Neither does the other's half well. Without WebMCP, the agent is guessing at a screenshot and you are transcribing numbers into a chat box. With it, both of you act on the same live simulation — and the simulation is the kind of state a screenshot cannot carry: 61 observation values at 50 Hz, a reward decomposition per term, a selection that means this joint, this window, this rollout.

Reward design is a search over a space no human can hold in their head. The run that produced base-walk logged 22 reward terms — velocity tracking, uprightness, foot clearance, foot slip, swing height, angular momentum, action rate, torque rate, self-collision, a fallen tax, and a dozen more — each with a weight, each interacting with the others across 4,000 iterations of 4,096 parallel environments. Change one weight and the policy does not get a little better or worse; it finds a different way to satisfy the objective, and the only way to know which way is to look. So every real experiment branches: this weight or that, this curriculum or that, and each branch produces its own curves, its own per-term breakdown, its own set of failures. Inspecting all of it — cross-referencing a suspicious reward curve against a config diff against a catalogue of known failure modes — is exactly the work a person finds tedious and gets wrong, and exactly the work an agent finds natural. What the agent cannot do is look at the duck and know the gait is ugly. MicroDuck Lab is built so that neither has to do the other's job: the human supplies the judgement and the intent, the agent supplies the reading and the bookkeeping, and WebMCP is what lets them do it on the same live state.

What people and agents can do together that was difficult or impossible before

Your pointing gesture becomes the query.

Drag a range on the rollout timeline. Click the duck that fell. Type five words:

"What went wrong here?"

The agent does not ask what "here" means. It calls get_selected_timeline_range and get_selected_joint, learns you mean episode roulade, 1.80s–3.12s, pulls the telemetry for exactly that window, and answers:

Lost upright orientation in this window — the trunk went past horizontal. Trunk dropped to 0.049 m, well below the nominal 0.12 m stance. Large angular velocity peak (10.01 rad/s) — the trunk is being thrown, not steered. Action-rate spike at t=2.38 s — the policy is correcting hard.

No transcription. No screenshots. No "at about two seconds in". And it points back: name a joint and it lights up on the robot you are already looking at.

The agent arrives already knowing this robot. Not because the model was trained on it — it launched five weeks ago — but because the page hands over Pollen's own reward-design playbook through get_rl_playbook. Ask why a policy keeps falling and you get the maintainers' hard-won rules, not textbook RL.

You can break a reward function on purpose. Reward is a function of a trajectory, so stored rollouts can be re-scored under any objective instantly — no retraining, no GPU. Give a penalty a negative weight and the ranking reorders: a forward roll overtakes a clean walk, 7.31 to 7.04. The breakdown shows why in one line:

Angular-velocity penalty   w=-3   mean=-1.538   contrib=+4.613

A penalty paying out. That is Pollen's own documented sign-convention bug — where butt-hopping and crash-sits come from — reproduced in a browser in one slider drag. check_reward_signs implements their "infallible check" (every penalty must contribute ≤ 0) so an agent can catch it before GPU time is spent.

And the loop closes off the page. Training a 61→14 locomotion policy needs CUDA and hours, so the studio does not pretend to do it. Instead, compose_training_job emits a prompt — not just a command — that you paste to the assistant on your own GPU machine, or hand to Hugging Face Jobs. It clones microduck_rl, runs Pollen's 64×5 smoke test first, launches the real run, watches that every penalty stays ≤ 0, and exports to ONNX. Training continues the same conversation instead of becoming shell archaeology.

Then you bring the weights back. import_policy validates the 61→14 contract on load — a wrong-shaped file fails there, not as a flailing robot — and the imported policy becomes first-class: in the list, drivable, and A/B-able. We did this for real: base-walk, trained overnight on an RTX 5080 (velstand, 4096 envs, 4000 iterations), ships with the site. compare_policies runs it against the published alpha_walking over identical scenarios and seeds. It survives on ice where the baseline does not — 50% vs 0%, on 2 seeds × 5 s, which is a small sample and is stated as one.

How it creates a better user experience

Six guided lessons, each a real action with a real outcome, each ending by handing you a question worth asking the agent. The simulator is the lesson; the agent is the tutor that can read live state.

The studio is built to be operated by either party. Everything the agent can do, you can do by hand — drop obstacles in the duck's path, make the floor slippery, shove it with the mouse, drag a reward slider — and everything you do, the agent sees. A policy trained elsewhere gets the same treatment as one that shipped.

And when the duck goes down it says so, from a speech bubble anchored to its actual jaw. The bubbles are the studio's way of turning every moment into an invitation: a fall suggests a question to ask, an idle duck suggests something to try, a new obstacle suggests a comparison. One of the things it says is true and teaches something: "I have fallen and I cannot get up — there is genuinely no get-up policy."

How we implemented WebMCP

One command layer. No second path for the agent.

Every state change goes through a single core module (src/core/commands.ts). The UI calls it; each WebMCP tool (src/webmcp/registerTools.ts) is a thin schema wrapper over the same function. So when the agent sets a velocity command, the slider physically moves. Human state and agent state cannot drift, because they are the same state. A parity test asserts it.

  • 53 tools registered on document.modelContext (not navigator — deprecated in Chromium 150), grouped as knowledge, state, selection, attention, mutation, execution, analysis and escalation.
  • Security annotations per Chrome's tool-security guidance: 25 tools marked readOnlyHint, 3 marked untrustedContentHint — get_rl_playbook returns third-party markdown verbatim, import_policy echoes a user-supplied URL. They are nested under annotations; passed flat, Chrome silently drops them.
  • Within the published budgets: names ≤30 chars (max 27), descriptions ≤500 (max 373), parameter descriptions ≤150.
  • Errors are values — {ok:false, reason, suggestion} — and invalid input names the valid options, because an agent can recover from that. No enum freezes the policy list, so a policy you imported is as nameable as one we shipped.
  • A dev panel lists every registered tool and logs every call, so the difference between structured tool calls and computer-use clicking is visible on screen.

Under the hood: MuJoCo compiled to WASM (DeepMind's official build) running Pollen's real robot_allcollisions.xml; the nine published ONNX policies via onnxruntime-web at the trained 50 Hz; rendering from MuJoCo's own compiled mesh arrays, in the robot's real livery. Client-only — no backend, no telemetry, nothing to authenticate. Deployed on Cloudflare Workers.

The observation contract was ported deliberately literally from upstream, which calls it "the highest-risk code in the crate" — a wrong offset does not throw, it produces a robot that flails. obs[1,61] → actions[1,14], joint positions relative to the home pose, and a command block ordered z, roll, pitch.

Challenges we ran into

  • The MJCF sets no timestep, so MuJoCo defaulted to 0.002 s while mjlab trains at 0.005 s. Control ran at 125 Hz instead of 50, and the duck walked at a third of its trained speed. Found by measuring, fixed by setting the timestep explicitly.
  • Three evaluation-modelling errors, each silently wrong: a slope whose sign was inverted, a floor rotation the solver never saw, and a friction override MuJoCo ignored because it takes the element-wise maximum of the two surfaces — fixed with geom_priority.
  • Chrome drops security annotations passed flat, without a warning. They must be nested under annotations. We only found out by reading the tool objects back.
  • Imported policies were second-class. Import worked, load worked, and then the dropdown, the eval suite and the post-eval restore each resolved ids against a frozen list of the nine published policies. The one thing the import existed for — A/B against the baseline — was the one thing it could not do. Fixed with a single policy registry that both sides use.
  • Auto-wake was corrupting eval numbers. During evaluation the duck falls by design; auto-wake was resetting it mid-episode and inflating survival rates. Now suspended for the duration of a suite.
  • An RTX 5080 is Blackwell (sm_120), which stock PyTorch wheels do not support. The overnight training run needed a CUDA 12.8 build first.

Accomplishments that we're proud of

  • The premise we started with — "inflate forward speed and a pathological rollout wins" — turned out to be false. alpha_walking beats everything on distance. We replaced it with Pollen's two documented failure modes rather than ship a demo that lies.
  • The training loop is real. A policy trained on our GPU, from a command the page composed, is on the site right now and beats the shipped one on ice.
  • The pointing gesture as query. Nothing about it is a trick; the selection is genuinely agent-readable state.

What we learned

That the honest version of a demo is usually the sharper one. "Change a weight and watch the robot change" would be a lie; "change a weight and watch what your reward function already prefers" is both true and a better lesson about reward hacking. That MuJoCo's defaults will silently disagree with a training rig's. And that a WebMCP tool surface is only as good as the one command layer beneath it — the moment there is a second path for the agent, the two drift.

What's next for MicroDuck Lab

  • A deeper policy-and-reward designer, working with the agent. Explore and design policies and rewards inside MicroDuck Lab; the agent runs the branch experiments on a local GPU or in the cloud (Hugging Face Jobs), collects and analyses every result, picks the winning policies, and you iterate again in MicroDuck Lab.
  • Closing the loop tighter. Today the handoff to a GPU is a prompt you paste; next it is a trainer bridge — a small local daemon the agent can talk to directly, so a training run is launched, watched and brought back without leaving the conversation at all. Alongside it, an experiment journal: every recipe, every eval, every A/B kept as agent-readable history, so the question becomes what did we try last Tuesday and why did it lose rather than what is the state right now. And the training curves themselves come into the studio, next to the A/B table, so the whole loop — hypothesis, run, result — is visible in one place.
  • Fine-tune in the browser — not train from scratch. From-scratch locomotion is impractical in a tab by two to three orders of magnitude, and the bottleneck is physics, not the network: MuJoCo Warp is CUDA-only and no WebGPU engine does contact-rich articulated bodies. But a .pt base checkpoint changes the question. Fine-tuning a few hundred iterations at 16–64 environments is a tiny fraction of the 390M steps a full run takes — plausibly minutes to an hour — and the 61→14 network itself trains happily in TF.js or ORT-web. Whether PPO at that env count learns anything useful is an empirical question, and the honest answer may be "only for small nudges". But nudge a policy in the browser and A/B it against the base is a real feature, and the base checkpoint that makes it possible now exists.
  • Train the hierarchical top layer in the browser. The Arena idea already solves the compute problem: locomotion stays frozen (alpha_walking, ball_kick_*), and what gets trained is the small policy that decides where to walk and when to kick. Tiny state, tiny action space, and the physics cost per decision is amortised over hundreds of frozen low-level steps. That trains in a browser today, without WebGPU — and it is the pedagogically right place for browser training to live, because it teaches the hierarchical decomposition itself.
  • The rest of the sensors, especially the RGB camera and the ToF, to train more complex policies.
  • Arena — several ducks, one world. The feature that was scoped first and cut first. Two ducks and a ball, because football makes multi-agent reward design self-evident: both ducks chase the ball, and the reason is visible at a glance — both are rewarded for ball proximity and nothing rewards positional diversity. It is hierarchical by construction, which is what makes it tractable: the low-level locomotion is already trained (Pollen's alpha_walking and ball_kick_*), so what gets designed in the browser is the high-level policy that decides where to walk and when to kick. That is the honest, cheap, and pedagogically right introduction to hierarchical RL, and it gives the agent a new class of question to answer: not why did this duck fall but why do these two ducks do the same thing.
  • A get-up policy, so the duck can fix what it says it cannot.
  • Contextual tool registration — fewer tools exposed until the state makes them relevant.
  • More of the robot's own catalogue: the ball kick, the roller mode, the tasks mjlab already knows how to train.

Honest limits. Changing a reward weight does not change the robot's behaviour — that needs retraining. Pollen publish ONNX only, with no optimizer state, so fine-tuning from the published policies requires training your own base checkpoint first. The simulation lacks mjlab's BAM actuator model, so this duck tracks roughly 43% of commanded velocity; the robustness numbers are a property of this simulation, not of the real robot.

Try it. Open the live URL in ChatGPT's in-app browser (or Chrome with chrome://flags/#enable-webmcp-testing), then ask: "What tools does this page expose?" — 53. "Load the walking policy and make it walk forward." — watch the sliders move by themselves. Click a joint. "What am I looking at?" Drag body_ang_vel to −3 or below. "Why did the roulade suddenly win?"

It works with no agent at all, too. WebMCP absence is stated, not fatal.

MicroDuck Lab is an independent open-source project (Apache-2.0) and is not affiliated with or endorsed by Pollen Robotics or Hugging Face.

Built With

Share this project:

Updates

Submission history