bot pong

Inspiration

We wanted Bracket Bot to be a robot that people could actually play with, just like how you would (cough, get drunk, cough) play beer pong.

What it does

A Bracket Bot would be able to first see the ball, pick it up, aim, and then take a shot. The bot combines camera feedback and robotic arm control to perform a pick-and-throw sequence:

Detect an orange ping-pong ball through cameras
Move the gripper toward it, using the ball's image position and size as feedback
Close the gripper while moving to a recorded throwing pose.
Execute a throwing motion and open the gripper when the wrist crosses a selected angle.
Monitor all feeds (camera, detections, joint positions, current, temp, etc.) through a dashboard.

How we built it

Most of the hardware came pre-packaged with Bracket Bot (NVIDIA Jetson Orin Nano, head camera, wrist cameras, IMU and robot arms). For control, we used BBOS motor interfaces and native inverse kinematics.

We used Python, NumPy, OpenCV as well as wiring up camera/telemetry services as well as a browser dashboard. For early experimentation, we used BracketBot's MuJoCo simulator to envision what the final version in real life would look like. For demonstrations (arguably the most critical part of this project), we used built in Mimic recordings supplied examples and reference poses for starting, grasping and throwing.

Challenges we ran into

We ran into a whole bunch of challenges! We had to pivot a lot of times to make our eventual idea work. None of us had robotics backgrounds, and our whole idea at the beginning was to allow Astra to handle most of the motion control, together with the motor control, etc. We had belief that Astra was able to do this because of several Twitter videos of Astra controlling robotic arms to generate great paintings.

What we soon realized however was that even though Astra was able to control each individual motor and move the bot, interfacing with the bot via ssh was non-trivial. Max tps for Astra plateaued around 60 tps; we were further plagued by weak WiFi issues while hacking on this project. As such, we had to abort our initial idea due to extremely high latency and long feedback loops. We were rather discouraged after the first night of hacking because little to no progress was made.

While we were sleeping, we let (our collective subscriptions of) Astra overnight to produce good simulation trajectories in MuJuCo. This yielded ~some success (video attached), even though it was unclear how we could use such success we would have in translating it to an actual environment. Simulation results did prove that our initial idea was feasible though, since the Bracket Bot (in sim) managed to chuck the ball into the pong cup.

We soon figured that it would make more sense for our team to operate over a wired connection (USB C to C) to the bot, since the bot would be operating on a fixed "leaning" position. This simplified the challenge, eliminating the need of operating the travel motions of the bot autonomously. At this point, we had split the hack into two challenges: 1. picking up the ball and 2. throwing the ball once the ball has been picked up.

As a team, we found it hard to agree on what data types we should use from the bot. We converged on needing to use Mimic to a greater extent to record full trajectories, even though the parameter mappings did not reliably generalize to new ball locations (for challenge 1 of picking up the ball). Eventually, after much help from the Bracket Bot team, we managed to agree on a visual-servo approach. Even though we first started with a "bang bang" control method, we soon realized its shortcomings. This approach caused the movement towards the ball to be extremely erratic. So we faced many challenges until then!

It was also extremely hard to parallelize things since this was not a software project.

Accomplishments that we're proud of

Being able to do complete physical pickups without intervention was a huge milestone
Recording our first autonomous throw from the bot into the cup was awesome! (see video)
Representing trajectories in simulation
Creating a data pipeline / learning flywheel that allows us to fit a line of best fit

What we learned

Robotics is really non-trivial! Even though there were instances where LLMs showed its brilliance, such as being able to generalize a good function for all failures and successes, they are definitely not token efficient in the way that they are trained for robotics tasks.