Inspiration
Before the hackathon, Krish Shah shared 52 ideas for 52 robot arms that you can build these at the hackathon this weekend:. Harsha Goli responded saying:
- Robot arm to fold laundry the hardest problem in robots is number 16
That was all the motivation we needed to pick our hackathon idea!
We decided the most interesting laundry folding subject was shirts because 1) it seemed to require more intricate operations than other garments and 2) it necessitated simultaneous left+right arm movement.
Furthermore we’d never seen a robot implement the ninja shirt fold technique which looks basically magical when done correctly so we had a novel target to aim for!
What it does
Two pairs of SO-100 arms collaborate to fold t-shirts incredibly quickly using an ACT model trained on demonstrations.
How we built it
We started by validating our data collection and policy process on a much simpler task — autonomous handshaking. We used the LeRobot library in a leader-follower configuration to collect 50 demonstrations of one SO100 giving us a handshake. We trained an Action Chunking Transformer (ACT) policy using the LeRobot library and RunPod GPUs (H100). We called this task a success when Krish, Krish, Nicole, and many others lined up to shake the robot’s hand.
After validating our pipeline, it was time to tackle shirt folding! One of our team members owned an SO100 and was gracious enough to let us use it for the hackathon, so we could create a bimanual setup. In order to have the setup interface with LeRobot, we had to modify the library to allow two SO100 arms to be controlled simultaneously instead of just one. We did this by creating a new configuration class for the bimanual setup and modifying the repository where necessary.
After that we only needed to collect 85 meticulously and artfully performed demonstrations of the ninja folding technique on our unique bimanual LeRobot SO100 arm setup. Afterwards we trained an ACT to perform the task and relieve ourselves of the burden of folding our clothes.
Challenges we ran into
On the hardware side:
- We struggled for 12 hours with stripped screws, smoking motors (3), and over eager power supplies (too powerful for the servos to handle). The solution was to render the burnt servos the status of leader arm, as they still had working encoders, and swap the gears between the arms to continue folding on.
- This bimanual setup also required a crazy number of cables to support 3 cameras (2 arm cams and 1 overhead webcam), 2 leader arms, and 2 follower arms and had us running all over the city to get enough dongles to pipe everything into 1 computer.
- The grabbers had difficulty snagging the fabric until we were able to find a working foam tip configuration that had enough grip to pick up the shirt but also smoothly drop it.
On the software side, (on top of the usual integration hell that every team experienced) we found that bimanual tasks were way less supported by LeRobot than single arm use cases and we were constantly running into issues with the bandwidth of the triple camera setup or the training architecture.
And for generating the dataset, recording 85 high quality shirt folds via teleoperation was a colossal task. We worried that we were going to fall into a trap of garbage in -> garbage out where bad training examples lead to bad results. In practice we got better and better at folding the shirts so that the final behavior only slightly resembles its sleepless 7am folding example parentage!
Accomplishments that we're proud of
- Autonomous handshaking checkpoint on the first day,
- Autonomous bimanual shirt folding on the second day with secret technique.
- Modifying the LeRobot library to fit our bimanual configuration.
What's next for Ninja folding technique dual arm robot
In order to get our bimanual setup to work with LeRobot, we had to create our own configuration class and modify their existing code. We plan to introduce a pull request to LeRobot to enable the ability for others to easily teleoperate and record data with the bimanual SO100 arm setup as well.
Log in or sign up for Devpost to join the conversation.