Team PiGestures

Members

Aiden Ye
Mihir Tandon
Andrew Cheng
Aiden Shoppel

Project: Gesture-Controlled Robot

Date: 2nd March 2025

Problem

Construction worksites are often noisy and chaotic, making nonverbal communication an important challenge. Workers frequently use hand signals to communicate with those operating heavy machinery (e.g., riggers/signalmen directing the movement of a crane). Additionally, standing outside of a vehicle increases awareness of the surroundings, so off-site/indirect operation can improve efficiency and decrease human risk.

Solution

Our multi-directional rover utilizes computer vision and machine learning to recognize and interpret human hand gestures. It autonomously detects and centers the user’s hands within its field of view, ensuring smooth interaction. By simply pointing in a direction (left, right, forward, or backward), the user can guide the rover’s movement, while a closed fist instantly stops it. Additionally, the rover will stop if it detects a potential collision autonomously. This technology lays the foundation for large-scale autonomous machinery on worksites, enabling adaptive robots to interpret and respond to human actions with precision and safety.

Inspiration

Construction worksites are often noisy and chaotic, making nonverbal communication an important challenge. Workers often rely on hand signals to communicate with those operating heavy machinery (e.g., riggers/signalmen directing the movement of a crane).

What It Does

The rover autonomously detects and centers the user’s hands within its field of view, ensuring a smooth capture of the hand joints. By pointing in a direction (left, right, forward, or backward), the user can guide the rover’s movement, while a closed fist instantly stops it. Additionally, the rover will stop if it detects a potential collision autonomously.

How We Built It

We utilized a hand-gesture recognizing library in Python to read and distinguish inputs from the camera. These hand gestures are then translated into commands for the robot, such as "move left," "move right," or "move forward." In parallel, we run an algorithm for the camera servos to track the hand of the navigator.

Challenges We Ran Into

Servo broke
SSH connection issues
X11 forwarding
Setting up a virtual environment
Raspberry Pi overheating
Camera latency
Multithreading complications
Synchronizing object detection with motor and light functions

Accomplishments That We're Proud Of

In the end, we successfully integrated all of the components of our robot, which required extensive troubleshooting and on-the-fly problem-solving. After breaking a servo, we quickly adapted by replacing it with one from another kit, which required disassembling and rewiring a significant portion of the robot.

What We Learned

Writing clean code and testing early saves a lot of time debugging and troubleshooting.
Changing too many things at once without incremental testing makes debugging much more difficult.
Writing multithreaded code is harder than we initially thought.