-
-
Controlling robot arm with webcam
Inspiration
Telerobots can be used for dangerous and strenuous activities, such as nuclear decommisioning, bomb disposal, and construction. It can also be used in our daily lifes. For instance, when we are overseas or busy at work but want to check on our parents or feed our pets.
However, haptics gloves and virtual reality technologies are commonly used in telerobots. They are expensive and not commonly found in households. We want to democratize this telerobots technology by using RGB cameras to remotely control the robotic arm.
What it does
We demonstrate how the robotic arm is controlled with a webcam by performing pose estimation on the images. The robotic arm can be controlled to some extend using the webcam. A GUI is also developed to remotely control the robotic arm and a pick and place is demonstrated.
How I built it
We remotely control the robotic arm in simulation using our webcams. The simulation is setup with Unity3D. The body and hand poses are estimated using images from the webcam. The pose estimators are imported from Mediapipe.
The wirst position (x, y, and z) and orientation (yaw, pitch, and row) are send via AWS SQS. Using the initial pose as reference, the hand's displacement is translated to the robot end effector's displacement. ROS Moveit is used to plan the trajectory and move the robotic arm to the target pose.
Challenges I ran into
The body pose estimator has some issues with detecting the depth (z coordinate) of the joints. The end effector has some trouble moving front and back.
The hand pose estimator has some issues with detecting the pose because the image of the hand is too small. An additional object detection of hand might help to crop and enlarge the image of the hand only.
Accomplishments that I'm proud of
We are proud to integrate the pose estimation of the hand and upper body to the control the robotic arm to some extend.
What I learned
We learned how the body and hand pose estimators work in mediapipe. We learned the limitations of using single camera for pose estimators.
What's next for telerobots
A more accurate pose estimators for the body and hand is needed to control the robot steadily.
Log in or sign up for Devpost to join the conversation.