Inspiration
We don't have an SO-101 every day of the week. When we do, there's no time to teleoperate for hours, collect hundreds of demos, and learn from training mistakes on real hardware. We wanted to see the outcome of our code BEFORE training, and train in sim without losing our minds re-configuring the VM, simulator and environment every single time.
So we built a platform for it: one that asks us intelligent questions and recursively gets better with every answer.
What it does
- Video → sim twin. SAM 3 segments every object instance in each frame. SAM 3D Objects reconstructs each one as a 3D model with pose. Nemotron (Nebius Token Factory) estimates physics as RANGES (mass, friction, size), and Tavily pulls real-world physics priors to ground them.
- One demo, in sim. You show the SO-101 the push once. No teleop, no real robot needed.
- Predicted action first. Before anything trains, you see what the robot is going to do, so you know what training will produce.
- Hundreds of what-ifs. Isaac Sim on a Nebius L40S replays and randomizes your demo across the physics it's unsure about.
- It asks, then improves. It finds which unknown actually breaks the task and asks ONE question (e.g. "How narrow is your cup?"). Nemotron Super phrases it, Nemotron Nano parses your answer, and Nemotron VL explains the failures. It re-plans (contact height, speed, stroke), re-runs, and asks again only if it's still worth it.
How we built it
- Segmentation + 3D: SAM 3 (per-frame instance segmentation), SAM 3D Objects (object meshes + pose)
- Reasoning: Nemotron Super / Nano / VL on Nebius Token Factory
- Simulation: NVIDIA Isaac Sim 6.2 on a Nebius L40S VM, set up once and reused; MuJoCo on a laptop for the demo and fast statistics
- Physics priors + prior work: Tavily
- Platform: FastAPI + web app
Results
- Success with no question: [FILL]%
- After sim-chosen questions: [FILL]% (random question: [FILL]%)
- Demo replay in Isaac Sim matched to 0.3 cm
- Total GPU spend: ~$[FILL]
Challenges
- Our first baseline was 92%, so there was nothing for questions to fix. We widened the physics ranges and made the robot's push depend on the answers.
- Some failures (tall cups tipping) can't be fixed by ANY answer: a paddle hardware limit. The platform learns not to ask useless questions.
- Isaac ran ~20–55 rollouts/min, so we split the work: MuJoCo for bulk statistics, Isaac for validation and video.
What we learned
A question is only worth asking if the robot's action changes with the answer. Simulation is the right judge of which question matters.
What's next
GR00T fine-tuning on the generated rollouts, more tasks beyond pushing, and closing the loop on the real SO-101 the days we have it.
Log in or sign up for Devpost to join the conversation.