MotionArena

• 

  2 human motion examples - one happening due to RL env agentic training of nerural surrogate in rollouts while other is the ground truth.

Inspiration

AI benchmarks are approaching a ceiling. Most are static: a model receives a problem, produces an answer, and gets a score. That works until models begin saturating the task distribution. At that point, another fixed set of harder questions only moves the ceiling temporarily.

Real scientific discovery does not look like a standardized test. The hard part is often not solving a narrowly posed problem with a known answer, but searching an enormous space of trajectories, designs, hypotheses, and even frameworks i.e deciding which experiment to run next, which question to ask, and which assumption to discard.

Systems like AlphaGo showed the power of pairing an enormous search space with a cheap, unambiguous verifier. Other breakthroughs such as AlphaFold similarly benefited from unusually strong computational supervision and dense ground truth. We wanted to ask what an equivalent substrate could look like for computational science.

MotionArena is our small working answer: replace narrow static benchmarks with dynamic simulation environments that offer large continuous search spaces, cheap verification, and a much steeper hill for increasingly capable agents to climb.

Physics and physical simulation are especially attractive substrates because their environments can also evolve toward progressively higher fidelity - spatially, temporally, and physically - rather than remaining a fixed collection of questions.

What MotionArena does

TL;DR: MotionArena frames human-motion reconstruction as a continuous search problem.

The environment withholds a ground-truth physical trajectory. The agent receives a natural-language prior and must reconstruct the hidden motion using a learned motion model. It can run repeated rollouts, inspect aggregate feedback, refine specific temporal spans, and iteratively improve its best candidate.

The browser makes that process visible in real time: the left avatar shows the agent's mutable reconstruction, while the right shows the immutable reference for the human spectator.

Instead of revealing the correct answer, the environment returns deterministic reward based on physical trajectory agreement — including pose alignment, root motion, velocity, and contact behavior.

The paradigm shifts from:

question → answer → score

to:

observe → propose → simulate → verify → improve

Why WebMCP matters

WebMCP turns the browser from a visualization into an agent-operable research environment.

Without WebMCP, an agent has to reverse-engineer buttons, sliders, canvases, and DOM state. MotionArena instead exposes the meaningful actions of the environment directly:

• inspect_episode
• run_rollout
• inspect_reward
• refine_span
• submit_best

These tools operate on the exact same state rendered in the browser. When the agent launches a rollout, the motion on screen changes. When its score improves, the rollout history changes.

The browser becomes both the laboratory and the observable audit trail of the search.

This is the WebMCP capability we find most exciting: websites can expose not merely UI controls, but the action space of an environment.

Why human motion?

Open-ended motion generation normally has no unique ground truth. If two people are told to "climb upward," many different motions may be valid.

So we changed the problem.

MotionArena uses withheld trajectory reconstruction. The environment starts from a captured human motion and hides part of it. The agent therefore faces a huge space of possible reconstructions, while the environment retains an objective reference trajectory.

Generation is hard. Verification is cheap.

Once a candidate exists, comparing it against the withheld trajectory is inexpensive and deterministic.

That gives us the structure we were looking for:

large search space + learned world model + cheap verifier.

How we built it

We use MoMask as the learned human-motion prior. It generates candidate trajectories represented as BVH motion, which we retarget onto matching VRM characters for visualization.

Three.js renders two synchronized avatars: the mutable reconstruction and the immutable reference. Both use the same character, camera, timing, and environment so differences in motion are immediately visible.

The scientific reward is calculated in canonical motion space rather than from the rendered avatar. This matters because BVH-to-VRM retargeting can introduce visual differences that should not affect the underlying score.

A shared environment state connects the renderer, verifier, rollout history, and WebMCP tools. Frontier agents can repeatedly search the MoMask inference space, while smaller browser-local policies can optimize residual motion corrections directly.

The problem we are really exploring

MotionArena is not primarily about producing one better animation.

It is a prototype for a different way of measuring model capability.

A capable scientific agent should increasingly be able to:

• choose useful experiments,
• search large continuous or combinatorial spaces,
• learn from failed attempts,
• allocate expensive computation intelligently,
• discover weaknesses in its current world model,
• and improve solutions under objective feedback.

Static benchmarks mostly measure the terminal answer.

Dynamic environments can measure the trajectory of improvement.

They can also become harder without being replaced: hide longer intervals, introduce unfamiliar motions, reduce rollout budgets, perturb initial conditions, increase spatial or temporal fidelity, or adversarially select episodes where the current agent performs poorly.

The benchmark becomes a moving frontier instead of a fixed question bank.

Why this matters economically

The same architecture appears across bottlenecked scientific and engineering industries.

A high-fidelity simulation may take minutes, GPU-hours, or much more compute, while a learned surrogate can evaluate candidate states far more cheaply. A capable agent can search inside the surrogate, identify promising or suspicious regions, spend expensive simulation only where it is most informative, and use those results to improve the next search.

The economically meaningful capability metric becomes something like:

verified improvement per expensive experiment.

In robotics, that can mean fewer physical robot trials. In aerospace, fewer high-fidelity CFD evaluations. In materials, fewer expensive quantum calculations. In thermal engineering, fewer full simulation runs. Eventually, in experimental science, fewer wet-lab experiments required to reach a verified result.

MotionArena demonstrates the smallest visible version of this architecture: the model proposes a physical trajectory, the hidden environment judges it, and the agent gets another chance.

Challenges and what we learned

The hardest problem was designing the verifier.

Our first instinct was to evaluate free-form text-to-motion generation, but natural-language instructions do not define a unique correct trajectory. That would make reward partly subjective.

Reframing the task as hidden trajectory reconstruction gave us objective ground truth while preserving an enormous search space.

We also had to separate what the human can see from what the agent is allowed to know. The spectator can compare both animated characters, while the agent receives only the instruction, environment state, and aggregate verifier feedback.

Finally, we separated visualization from evaluation. VRM retargeting makes the search understandable to humans; the underlying trajectory representation determines the score.

That distinction — between a human-readable world and a machine-verifiable one — became one of the most important ideas in the project.

Where we want to take it

MotionArena is a small example of a larger idea: scientific dry labs for agents.

Instead of giving increasingly capable models another frozen collection of questions, we want environments where they can continuously search, simulate, fail, recover, and improve against computational laws that provide objective feedback.

Physics simulation is particularly powerful here because the environment itself can scale with the model: more dimensions, finer spatial resolution, longer temporal horizons, more coupled physics, and stricter verification.

WebMCP provides a natural interface for these environments because the browser can simultaneously be the visual world for a person and the structured action space for an agent.

Our Vision

Playbook for building science benchmarks now needs to be redefined. Astra is a phenomenal model but with its release containerized, static computational workflow benchmarks will hit a definite wall.

But, until these models have the capability to either solve or pose Millennium-class problems, we must build dynamic environments that give them a much steeper hill to climb.

Think of it as a research problem on its own but instead training AlphaGo we'll be building a new Game of Go: where each move distinctly alters the win-lose state, the environment dynamically shifts through multiple configs, and yet the final win condition remains extremely cheap to verify.

Built With

• 3d-visualization
• agentic-ai
• ai
• bvh
• cmu-motion-capture
• codex
• humanml3d
• modal
• momask
• motion-capture
• motion-generation
• neural-simulation
• openai
• react
• reinforcement-learning
• rl-environment
• scientific-computing
• three.js
• typescript
• vercel
• vite
• vrm
• vrma
• webgpu
• webmcp