Arrival

Home robots arrive in kitchens they've never seen. Arrival turns one iPhone scan into a physics sim where the robot practices first, making 1,000 GR00T demos in under an hour.

Comment

Inspiration

Home robots are coming, and each one will arrive in a kitchen it has never seen. General policies like NVIDIA GR00T learn in generic simulated kitchens, and teaching a robot your own kitchen by remote control took us 61.5 seconds per demo. We wanted the robot to practice in your exact room before it gets there.

What it does

Arrival turns one iPhone RoomPlan scan into a MuJoCo twin of the room, where every drawer, door and appliance door is a working joint. A robot arm practices kitchen tasks there while simulated people walk around it.

  • Nemotron on Nebius Token Factory plans each task
  • Tavily supplies facts about each appliance
  • Every demo that passes the physics check is saved as a GR00T-ready LeRobot dataset, at 3.3 seconds a demo on one laptop

How we built it

  • Core: Python 3.11 and MuJoCo, with a LeRobot v2 writer and validator
  • App: a web app with an upload queue and workers
  • Evaluation: a frozen-trial evaluation harness with paired comparisons
  • Models:
    • Nemotron 3 Super on Token Factory plans the tasks
    • Nemotron Lightning writes varied training and held-out instructions
    • Tavily researches each scanned appliance
    • We fine-tuned GR00T N1.7-3B on one room's data

We built everything in the open: 274 merged pull requests, 1,727 tests, and CI on Linux, macOS and Windows.

Challenges we ran into

  • Noisy demos taught bad habits. Our first noisy demonstrations taught GR00T to stop drawers short, because 102 of 339 drawer demos ended under 0.28 m. Removing those brought drawers back to 21 of 24.
  • RoomPlan describes surfaces, not cabinet internals. We generate drawers, doors and handles from each fixture's type and size and keep them from overlapping.
  • Physics tuning. Getting the arm to hold a bowl by its rim and open doors without touching walkers took many rounds of tuning.

Accomplishments that we're proud of

  • GR00T N1.7 fine-tuned on one room's simulated data passed 39 of 48 paired trials, against 26 of 48 for the clean-data baseline (sign test p = 0.007).
  • On indirect requests like "let me check if we're out of milk," the Nemotron planner chose the right task 13 times in 15. A keyword parser managed 6.
  • A real kitchen scanned on an iPhone ran end to end into a physics-checked dataset.
  • The project is open source under Apache 2.0.

What we learned

  • Data quality decides what the policy learns. Nearly a third of our drawer demos were marginal, and the policy copied them.
  • Measure the baseline first. Timing a human teleoperation baseline first kept our speed claims honest.
  • LLM planners earn their place on indirect requests. They matter for requests that never name the task, which is where a keyword parser fails.

Built With

Share this project:

Updates

Submission history