Inspiration
Robots need demonstrations to learn practical skills, but collecting them often requires specialized equipment, technical setup, and operator training. We wanted to turn a Meta Quest headset into a robot demonstration collection tool, making this work easier for more people to participate in.
What it does
RoboTrainerXR lets users control simulated robots with their hands, turning human actions into demonstration data for robot learning. Users collect demonstrations in mixed reality on a standalone VR headset, without setting up and operating a physical robot for each recording session.
Guided tasks, progress tracking, and game-like rewards help users learn the controls and keep contributing. Each completed task produces more than an interactive experience: it contributes a recorded demonstration for subsequent model training and evaluation.
How we built it
We built RoboTrainerXR with Unity, Meta Quest hand tracking, and MuJoCo physics. Robot simulation and demonstration recording run locally on the headset, bringing interaction and data collection together in one application.
Challenges we ran into
The core challenge was not just making robots easy to control, but ensuring that the collected data had practical training value.
We selected established robotics benchmark tasks for our initial collection workflows and used the collected demonstrations to train robot models, including ACT and π0.5. Across the tasks and settings we tested, models trained on our collected data achieved varying improvements in task success rates over models trained on the original benchmark demonstrations. These results provide experimental support for the training value of our data.
Accomplishments that we're proud of
We ported MuJoCo to Quest, connected guided interaction, robot control, and local demonstration recording into a complete workflow, and validated the value of the collected data through model training experiments. This establishes consumer XR hardware as a viable tool for robot demonstration collection.
What we learned
Accessible interaction and reliable recording are equally important. Clear guidance and feedback help users contribute demonstrations more consistently, while downstream model training and evaluation help us determine whether those demonstrations are useful.
What's next for RoboTrainerXR
We plan to develop the platform further so robotics researchers and companies can post real data collection needs, and users can take on tasks, contribute demonstrations, and receive compensation. Our long-term goal is to connect these organizations with a broader community of headset users, making robot demonstration collection a form of productive work that more people can participate in.

