RICO: Your Tool-Carrying Companion

4-DOF robotic arm
Elevated webcam used for CV
Screwdriver model results
perspective transformation example with homography matrix

Inspiration

From having worked in a robotics team, we always enjoyed building robots, but hated being at the workshop alone; half the time would be spent on the robot, while the other half would be spent rummaging through toolboxes to find a tiny-but-essential Allen wrench. These lonely, tedious searching sessions are what inspired our project. We named our robot RICO, after the penguin of Madagascar who could pull out tools for any job from his tiny beak.

What it does

RICO is object-grabbing system that consists of a webcam and 4-degree-of-freedom robotic arm. It first receives a voice command from the user (ex. "Grab the screwdriver"), then looks for the tool that was requested in the camera view, grabs the tool, then locates the user's hand, where it will then drop off the tool.

How we built it

The camera stand and robotic arm parts were built custom 3D printed parts which were designed in Onshape, as well as some salvaged parts from past projects. Servo motors were powered by a Servo2040 microcontroller, which communicates with our laptop via Serial. We use Groq as our LLM to parse the voice command into performable actions. The tool-detection CV models were trained using open-source datasets from Roboflow. We used OpenCV to manage the camera feed, and used a prospective transformation matrix to turn our angled webcam POV into a bird-eye-view camera feed. The robotic arm's movement was precisely calculated using Inverse Kinematics, taking a 3D coordinate point as an input and returning the required servo turn angles as the output.

Challenges we ran into

Computer-microcontroller communication: In order to command the arm's movements, we needed to send servo angles through a USB-C cable to reach the Servo2040 microcontroller. At first, we experimented with Serial communication, but we found that this method bottlenecked our productivity; each time we needed to push code to the Servo2040, we had to manually disconnect and reconnect the microcontroller from our computer, wasting precious minutes.
Datasets yielded mediocre results: Since gathering our own dataset during a hackathon was not feasible, we turned to open-source datasets on Roboflow to train our tool-detection CV model. The downside of this was that the resultant models provided inconsistent detection, only working when the objects were oriented in a certain way (screwdriver was only detected when placed diagonally), making our tool pickups less consistent.