Real-Time Vision-Guided Teleoperated Mechanical Hand

Your hand's movements, replicated in real time by a cardboard robotic hand built from scratch.

Comment

Goal

My goal was to build a robotic hand that could mirror my own hand movements in real time without relying on expensive sensors or specialized equipment. I wanted to see how cheaply I could build a functional robotic hand using materials I already had, while developing the software myself.

Inspiration

I started by following a YouTube tutorial that used flex sensors to control a robotic hand. Since flex sensors were too expensive for me, I initially replaced them with simple push buttons. However, controlling the hand with buttons felt clunky and wasn't very natural.

One day, I saw a TikTok video of someone controlling smart lights using camera-based hand tracking. That gave me a different idea: why buy additional hardware when I already had a webcam?

I replaced the buttons with a Python hand-tracking script, allowing me to control the hand in real time without needing any additional sensors.

What it does

This project lets you control a mechanical hand in real time using just your laptop's webcam. When you hold your hand in front of the camera, a Python computer vision script tracks your finger movements and determines which fingers are open or closed. It instantly converts these gestures into servo motor angles and streams the data over a serial connection to an Arduino. The Arduino drives five micro-servo motors, pulling fishing-line tendons routed through straws on a cardboard frame, making the robotic hand mirror your physical hand movements live.

How i built it

Software: Built a Python script using OpenCV and cvzone to track hand landmarks from a live webcam feed in real time. The script converts finger positions into target angles (0° or 180°), flips inverted channels in code to fix motor orientation, and streams comma-separated string payloads over USB serial using pySerial.

Hardware & Electronics: Programmed an Arduino Uno to read the live serial data and control five SG90 micro-servo motors attached to Arduino's control pins.

Mechanical Design: Built the physical hand frame using cardboard, using plastic drinking straws as guide loops for tendons. Threaded fishing line from the fingertips through the straws and connected them directly to the servo horns, using custom crossed routing at the wrist to prevent the thumb tendon from snagging.

Challenges i ran into

  1. Mechanical Blueprinting & Tendon Alignment: Building my first model without detailed sketches caused fingers to overlap and jam during flexion. Additionally, mounting the thumb servo in the far corner created bad diagonal strain on the line. I completely rebuilt the hand frame with proper blueprints and moved the thumb servo to the center for a smoother diagonal pull.

  2. Serial Port Locking & Connection Delays: Integrating Python with the microcontroller threw unexpected bugs: pySerial could not access the serial port when the Arduino IDE Serial Monitor was open on the same port. Adding exception handling and a 2-second boot delay in Python solved the connection pipeline.

  3. Physical vs. Digital Servo Inversion: Because servos were mounted facing opposite directions inside the compact cardboard base, sending raw angles caused some fingers to move backward. Instead of rebuilding the physical mounts, I fixed this in software by inverting specific channels (1 - f) directly inside the Python payload.

Accomplishments that I'm proud of

Software & Hardware Integration: Upgrading the project from a clunky button-controlled hand to a computer vision tracking system—combining live gesture detection directly with my passion for Arduino microcontrollers.

Resourceful Engineering over Expensive Parts: Shifted my perspective on basic materials like cardboard, straws, and fishing line. What initially felt like an inferior limitation became my strength, proving that creative engineering matters more than a high budget.

Teleoperation without additional sensors: Successfully building a responsive teleoperation pipeline using only a built-in laptop webcam and open-source Python tools instead of building gloves with expensive flex sensors.

What I learned

Proper Planning & Blueprinting: I learned the hard way that jumping straight into building without preliminary sketches leads to structural jamming and wasted materials. Careful planning and layout design are mandatory when building moving mechanical parts.

Computer Vision & Serial Integration: I learned how to implement real-time hand tracking using MediaPipe (cvzone) and discovered how straightforward it is to bridge high-level Python software directly to Arduino hardware via serial communication.

The Joy of DIY Prototyping: Above all, I learned to truly enjoy and appreciate building things from scratch. Working within tight constraints showed me that limited resources don't limit your potential—they just push you to be more creative.

What's next for Real-Time Vision-Guided Teleoperated Mechanical Hand

My next goal is to make the hand more anatomically accurate and capable of more natural movement. I want to add articulated joints to better replicate the structure of human fingers, improve the grip and range of motion, and move beyond cardboard toward stronger and more flexible materials.

I also want to improve the control system so the hand can reproduce more complex movements rather than simply switching each finger between open and closed positions. Ultimately, I want to turn the current prototype into a more capable and reliable robotic hand for real-time teleoperation.

Built With

Share this project:

Updates

Submission history