Inspiration
The diffusion of robotics into the economy can only scale as fast as we can build robots. If robots are going to make an impact, it follows that we will need a lot of them. In the 36 hours of this hackathon, we wanted to build a scoped version of this thesis: could two SO-101 arms assemble parts of another robot? We explored two approaches:
- Training a manipulation policy in simulation.
- Using a camera-guided agent to control the physical arms.
And we focused on understanding what it would take to get robots building robots.
The Build
First, we worked on curating the parts. We disassembled an SO-101 arm (which was much harder than it seemed) to have a set of physical pieces. Also, we separated the CAD into components and built a shared environment in MuJoCo for full simulation. Then, we combined a few pieces into a scoped assembly segment with 5 tasks for both physical and sim in motor, horn, screw, retainer, and yoke.
Around this environment, we put together a (1) RL setup with Success Guided Sampling and an actor-critic policy that we trained from scratch. In parallel, we built a (2) camera and lidar guided control harness for Astra/Sol to stream commands to our arms. We show that Robots Buildings Robots is feasible both in sim and real, but there is still a long way to go.
- ### Reinforcement learning with SGS
In starting this project, our initial goal was to train a small policy from scratch to build the robot arm end to end. However, reward signals for tasks of such a horizon are hard to come by. And with a sparse completion reward, a randomly initialized policy may almost never succeed. Our question was: what is the lightest amount of task-specific guidance we can give the policy so it starts getting a useful reward signal?
We decided to implement Success Guided Sampling, or SGS. Instead of supplying a complete trajectory, we prepared reset families of tasks that would serve as the seed state for different trials. For example, to calibrate our early bi-arm experiments, we trained a small model to pass a cube between the two arms. Once such a “reset Family” may be a state where both arms have their hand on the cube with one arm about to release. This is a state that will exist in nearly every successful sim. And once the policy learns to finish from this point, attempts further up the learning tree may now be successful when it reaches this task it already knows how to handle. SGS does not change the policy, only the starting points.
We train a 1.53 mn parameter SimBa policy with a separate 1.53 mn critic during training using PPO. It receives ~500 modeled surface points and ~250 robot-state history values, then produces 12 action commands at 25 Hz for the two arms. We prepared 50 reset states across the 5 tasks and 5 reset families, with separate training and evaluation starts, and train on MuJuCo sim. Then, we would have distilled this policy which takes actions on a point cloud into a policy that takes action based on RGB images from camera stream and adding noise in Isaac Sim.
- ### Camera-guided physical control
In parallel, we tried a more direct path of using IPhones as lidar sensors alongside physical cameras to give Astra/Sol more direct access to a vision workspace and then use code/write a DSL to move the physical arms. The core loop here is observed from the 2 unique data streams, plan, execute, and repeat. We implemented separate execution from streaming such that the next movement could be planned as the current segment executed.
Challenges and what we learned
- Training within 36 hours is quite hard. We think that this is a given, but trying to train a 2 mil param model + get over sim2real is much harder than software.
- Time-bound with simulation. For such a small model, having more than a few thousand CPUs doesn’t decrease time-to-performance.
- Getting the simulation right. Two arms and multiple mating interfaces made small geometry errors consequential. Even in simulation we had a hard time with the precision of the objects.
- Working with the physical hardware. Funnily enough, disassembling the SO-101 took longer than expected, and some pieces were difficult to fit even by hand.
- Slow model-guided control. Running Astra/Sol to control an arm is just quite hard and slow.
What’s next
- Getting the full project to work thoughtfully
- Screws
- World models instead of simulation because are bounded by CPUs