Surgical Assistant

Inspiration

The inspiration for our project is Tony Stark's Dum-E robot, an intelligent robotic arm that is also comically clumsy.

What it does

Our robot is controlled from a web UI that uses the web camera to track the user's head position. The user can turn their head left, up, or right to command the arm to pick up the object from the user's hand and and place it beside a yellow, green and red marker respectively. This mimics the behavior of a surgical assistant, where the surgeon can use a tool, and have a "third arm" controlled by his head position to grab instruments from his hand and place them in their desired spot.

How we built it

We used Javascript to create the UI and Mediapipe + OpenCV to for head position tracking.

The robotic arm grabbing + placing capability was built by training 3 separate models for each end destination (yellow, green, red). We created 30 training episodes using the teleorchestration arm and trained the models using AMD Radeon and Brev across 100K steps each with ACT policy.

Challenges we ran into

The biggest challenge we ran into was inconsistent data collection which affected the accuracy of the robot arm. We noticed that a few things varied too much for the small dataset of 90 total episodes. These include the starting wrist position, item orientation, overhead camera angle, and the speed of the teleorchestrated movements. In an attempt to compensate for this, we increased the number of training steps..

Accomplishments that we're proud of

We are proud of having a fully working end to end pipeline from head position recognition to grabbing and placing the object in the correct position.

What we learned

Consistent data is highly important. With only 30 episodes, slow and consistent movements across joints because there isn't enough data to generalize accurate movement.