Phantom Touch

Robot arm
Computer vision

Inspiration

We wanted to make controlling a robot feel as natural as moving your own arm. Phantom Touch explores that idea using a laptop webcam, turning everyday gestures into robotic motion without wearable sensors or a specialized controller.

What it does

Phantom Touch uses computer vision to track your right arm and control a five-servo robotic arm. Moving your elbow controls the base and lower pivot, moving your wrist controls the upper pivot and claw rotation, and pinching your thumb and index finger controls the claw.

A live preview shows tracking and movement targets. When detection drops, affected joints hold their position and resume when reliable tracking returns.

How we built it

We built the vision controller in Python using OpenCV and MediaPipe, with dependencies isolated in a virtual environment. Everything runs in Linux through WSL, with the webcam and microcontroller connected through USB passthrough.

Our PlatformIO firmware receives servo angles over USB serial. We normalize gestures by shoulder width, capture a comfortable neutral pose, and apply smoothing, speed limits, and configurable joint ranges.

Challenges we ran into

Getting reliable webcam capture through WSL took troubleshooting: the initial video stream produced a partially green image, which we resolved by switching capture formats.

Tracking also struggled when joints left the frame or fingers became obscured. We introduced a stronger pose model, confidence hysteresis, and a hand close-up detector with full-image recovery. Serial resets, garbled replies, and limited physical movement made hardware debugging another significant challenge.

Accomplishments that we're proud of

We connected webcam tracking, gesture interpretation, serial communication, and servo control into a complete pipeline. We also made tracking failures more predictable: missing fingers hold the claw, while missing wrist tracking allows visible elbow gestures to continue controlling the base and lower pivot.

Our automated checks now include 48 passing tests covering movement mapping, tracking recovery, joint holding, and communication with a simulated microcontroller.

What we learned

Camera framing, lighting, coordinate systems, and calibration all shape the experience. We also learned to distinguish a commanded servo angle from actual physical movement, and to test vision, communication, and hardware behavior separately.

What's next for Phantom Touch

We want to finish physical calibration, investigate power-related movement limitations, and evaluate tracking across different lighting conditions and gestures. From there, we hope to improve motion fidelity and explore practical uses such as remote manipulation and interactive robotics demonstrations.