CHOPSTIX

3D-Printed "Chopstick" food gripper
The full Chopstix Bot in all its glory

Inspiration

We we're hungry.

What it does

Our project features a silly, but useful robot.

For anyone with disabilities making meal time difficult (or for anyone too lazy to use chopsticks with their own hands) our robot comes equipped with a pair of 'chopsticks', and is capable of assisting people in putting food in their mouth.

With the help of AI, the robot will recognize what food you desire, properly pick up the food object, and gently place it in your mouth.

In future implementations, we plan on adding spoon functionality so that the robot will pick up a spoon with its chopsticks and use it to scoop up water/soup based foods. The goal of this prototype, however, was simply to attempt to work with solid and easy-to-grab foods (such as chocolates) as a proof of concept.

How we built it

The robot is comprised of 3D printed parts, random junk we found lying around, zip ties, and LOTS of duct tape. Its controlled by three NEMA 17 and one 28-BYJ48 stepper motors connected through their respective controllers to an Arduino for low level control code.

The Arduino communicates via serial with a python control interface run on the main control computer. This control interface contains methods that can be called by the feeding routine program (or any other for that matter) to make the robot move around.

Technical Details

We use dlib's face detector with a 68-point landmark model to track facial features, focusing on the mouth in real-time video. It calculates the Mouth Aspect Ratio (MAR) to detect when the mouth is open and displays this information using OpenCV. We process video frames with OpenCV by converting them to HSV color space to accurately detect Snickers Minis and chopsticks through color masking and contour analysis.

The system filters contours by size, calculates their centers, and identifies the closest targets to align the robotic chopsticks. By integrating with our custom communication interface module, we send serial commands to an Arduino, enabling the base and arm to autonomously position the chopsticks and pick up the detected Snickers. Together, the these programs allow the robot to autonomously retrieves and delivers Snickers to the users in response to their facial movements.

Challenges we ran into

Some challenged included actually making the vision detection code work, debugging the serial communication code, building the hardware in such a way that the motors could stand the (admittedly low but not insignificant) load we placed upon them, and integrating all of the parts together.

Unfortunately, this last challenge-commonly referred to as 'integration hell' in the robotics field-proved to be our downfall. Although we got every portion of our robot working individually-from using OpenCV to detect chocolates and mouths and output the direction of travel needed to place the chocolate in said mouth to the physical robot and all of its low-level control systems and interfaces-we could not get it to all work together. Eventually, the 'jank' nature of our robot-and our own human exhaustion-caught up to us, and it was clear that both varieties of hardware simply couldn't proceed with testing any longer. With too little time remaining to bring Chopstix to its fullest state, we humbly present a prototype-the beginnings of an idea which may one-day change the world.