Pickerupper

Pickerupper environment

Inspiration

We wanted to see how physical AI is able to help robots perform simple everyday tasks that humans can do easily but would be much harder for machines. At the hackathon, we chose to have our LeRobot SO-101 robotic arm pick up an object and move it to a designated drop-off location (which in this case was a napkin). Our goal was to use an open-source dataset and pretrained model rather than have to start from scratch with our own model and our own datasets.

What it does

Pickerupper focuses on grabbing objects and placing them in a designated area. It uses a LeRobot SO-101 arm, an open-source dataset, and the SmolVLA vision-language-action model / the ACT model as the basic structure of the project. We try to have the robot use camera observations to guide its movements, grasp the object, and place it on the napkin.

How we built it

We started by setting up the environment and installing packages needed to run LeRobot and PyTorch. We initially spent a lot of time on an NVIDIA-related package but realized it wasn't needed for this project.

Next, we calibrated the SO-101 leader and follower arms, allowing the follower to mirror the leader's arm movements. We then downloaded the open-source pickplace dataset from Hugging Face, which matched our goal.

Finally, we integrated the SMOLVLA pretrained model. The original dataset setup used 3 camera views, while our physical setup had only two. We adapted the inputs by just selecting the 2 views most accurate to our camera views, allowing us to work with the configurations needed to us.

The SmolVLA model was bugging a lot so we decided to find another model with more steps. We decided to switch to the ACT model which is basically a simplified Smolvla model and we ran 1k steps with it initially. It was not bad and was able to get to the object, but wasn't able to pick it up as well. We decided to bump it up to 10k steps and it became much better after this.

Challenges we ran into

One of our biggest challenges was the dependencies we had to download and managing out time. We initially spent a long time installing an NVIDIA-related package that was not needed for our hardware. Getting the PyTorch and ROCm environment working also required a lot of troubleshooting.

We also had another package needed for our training model to even run, but it moved at an incredibly low pace as low as 1 kb/s. It took a lot of time to get the existing data set and pre trained model to work, which made us reconsider throughout the process if we should have made our own data instead.

Another challenge was adapting the existing model and dataset to our physical setup. The mismatch between the 3rd camera view and our 2 available views meant we had to consider which data was most relevant and how to make the model work for our setup specifically.

Accomplishments that we're proud of

We are proud of getting our robotic arm to complete teleoperation with calibration. Our follower arm was able to copy the lead arm perfectly. I'm also proud of our robotic arm being able to pick up the object and drop it off on the napkin, but it ran into a lot of errors and we're still debugging it. I'm also proud of how hard we worked throughout this hackathon as this was our team's first hackathon during our freshman year.

What we learned

We learned the importance of open-source robotics with reusing datasets that are relevant to us and pretrained models that we can use easily.

We learned a lot such as using another computer and connecting it through terminal to run all of our training on that. We also learned about hugging face open source datasets and the smolvla model that we used to run our data on. Lastly, we learned how to think more efficiently and develop better problem-solving skills for future hackathons and tasks.

What's next for Pickerupper

Our initial idea was to be able to differentiate between different pieces of trash and be able to put them into different bins such as trash, recycle, and compost. We were only able to get the robot arm to pick up an object and place it in a designated location, but we would need to introduce multiple pieces of trash and be able to differentiate between the different items to place into each bin for our mission 2 idea.