• 

  3Dof robotics Arm

• 

  Esp32 cam

• 

  Esp8266

• 

  Full project image

• 

  esp8266 micro controller

• 

  SMPS _Power supply

Mobile Web Robotic Arm Control Using MediaPipe

Inspiration

The inspiration for this project came from the idea of making robotic arm control more natural, accessible, and suitable for remote operation.

Traditional robotic systems often require physical joysticks, dedicated controllers, or specialized software. We wanted to explore whether a robotic arm could be controlled naturally through human hand movement using only a mobile device and a web-based interface.

Our main goal was to combine computer vision, IoT, cloud communication, mobile web technology, and robotics into a single real-time teleoperation system.

The project was especially designed for remote robotic arm control, where a user can operate a robotic arm from a distant location simply by moving their hand in front of a mobile phone camera.

---

What it does

The project is a Mobile Web MediaPipe-Based 3-DOF Robotic Arm Teleoperation System.

The robotic arm currently has three controlled joints:

• Base — controls left and right rotation.
• Shoulder — controls upward and downward movement.
• Elbow — controls forward and backward arm movement.

The current prototype does not use a gripper.

The mobile phone camera captures the user's hand movement in real time. MediaPipe detects the hand landmarks and extracts motion information in the X, Y, and Z directions.

The hand position can be represented as:

$$ P=(x,y,z) $$

where:

• $x$ represents horizontal hand movement.
• $y$ represents vertical hand movement.
• $z$ represents forward and backward hand movement.

The detected hand movements are mapped to the robotic arm as:

$$ X \rightarrow \text{Base} $$

$$ Y \rightarrow \text{Shoulder} $$

$$ Z \rightarrow \text{Elbow} $$

Therefore, the user can control all three joints of the robotic arm through natural hand movement.

An ESP32-CAM is mounted on or positioned near the robotic arm to provide live video feedback.

This allows the operator to observe the robotic arm and its surrounding environment remotely while controlling the arm through the mobile web interface.

---

How we built it

The system was developed using a distributed architecture where different devices and technologies perform different tasks.

The complete system combines:

• Mobile phone camera
• MediaPipe hand tracking
• X, Y, and Z hand-position processing
• Mobile web application
• Render cloud deployment
• HiveMQ MQTT communication
• ESP8266 robotic controller
• ESP32-CAM live streaming
• Servo motor control
• 3-DOF robotic arm

Mobile Camera and MediaPipe

The mobile phone camera provides the video input for MediaPipe.

MediaPipe detects the user's hand landmarks and tracks hand movement continuously in real time.

Instead of requiring physical buttons or joysticks, the system observes how the user's hand position changes.

The changes in hand position are calculated as:

$$ \Delta x=x_{\text{current}}-x_{\text{previous}} $$

$$ \Delta y=y_{\text{current}}-y_{\text{previous}} $$

$$ \Delta z=z_{\text{current}}-z_{\text{previous}} $$

These values represent how much the user's hand has moved between two consecutive measurements.

The detected movement values are then converted into robotic joint movement commands.

A simplified joint update equation can be represented as:

$$

\theta_i^{new}

\theta_i^{current} + K_i\Delta P_i $$

where:

• $\theta_i^{new}$ is the new robotic joint angle.
• $\theta_i^{current}$ is the current joint angle.
• $K_i$ is the movement scaling factor.
• $\Delta P_i$ is the detected hand displacement.

This makes the robot respond according to the user's natural hand movements.

---

3-DOF Motion Mapping

The robotic arm currently uses three degrees of freedom:

$$ DOF=3 $$

The three joints are represented as:

$$ J_1=\text{Base} $$

$$ J_2=\text{Shoulder} $$

$$ J_3=\text{Elbow} $$

The MediaPipe motion mapping is:

$$ \Delta x \rightarrow J_1 $$

$$ \Delta y \rightarrow J_2 $$

$$ \Delta z \rightarrow J_3 $$

Therefore:

• Horizontal hand movement controls the Base.
• Vertical hand movement controls the Shoulder.
• Forward and backward hand movement controls the Elbow.

This allows the operator's three-dimensional hand movement to be translated into 3-DOF robotic arm motion.

---

Render Cloud Deployment

The web application and backend are deployed using Render.

Render hosts the web-based control system and makes the control interface accessible through the internet.

Because the system is web-based, the user does not need to install a dedicated desktop application.

The operator can open the system using a supported mobile web browser and use the phone camera for MediaPipe-based hand tracking.

This makes the robotic control system more accessible and suitable for remote operation.

---

HiveMQ MQTT Communication

The project uses HiveMQ MQTT for communication between the web application and the robotic hardware.

The main control communication path is:

$$ \text{Mobile Camera} \rightarrow \text{MediaPipe} \rightarrow (X,Y,Z) \rightarrow \text{Web Application} \rightarrow \text{HiveMQ MQTT} \rightarrow \text{ESP8266} \rightarrow \text{3-DOF Robotic Arm} $$

MediaPipe processes the user's hand movement and generates the required control information.

The web application then publishes the robotic control data through MQTT.

The ESP8266 subscribes to the required MQTT topics and receives the commands for:

• Base
• Shoulder
• Elbow

MQTT was selected because it is lightweight and well suited for IoT communication.

It also provides an effective communication bridge between the browser-based computer vision system and the physical robotic arm.

---

ESP8266 Robotic Controller

The ESP8266 acts as the main robotic arm controller.

Its main responsibilities include:

• Connecting to the network.
• Connecting to the HiveMQ MQTT broker.
• Receiving robotic control commands.
• Controlling the servo motors.
• Updating Base, Shoulder, and Elbow positions.
• Applying servo angle limits.
• Maintaining wireless IoT communication.

The servo mapping is:

$$ Servo_1 \rightarrow \text{Base} $$

$$ Servo_2 \rightarrow \text{Shoulder} $$

$$ Servo_3 \rightarrow \text{Elbow} $$

When the ESP8266 receives new joint-position commands from MQTT, it updates the corresponding servo motors.

This converts the MediaPipe-generated digital control information into actual mechanical robotic movement.

---

ESP32-CAM Live Streaming

The ESP32-CAM is used to provide live visual feedback from the robotic arm side.

The project therefore uses two cameras for two different purposes.

• Mobile Camera — used for MediaPipe hand tracking and robotic control.
• ESP32-CAM — used for monitoring the robotic arm and its surrounding environment.

The ESP32-CAM communication path is:

$$ \text{ESP32-CAM} \rightarrow \text{Wi-Fi / Internet} \rightarrow \text{Mobile Web Interface} $$

While the user moves their hand in front of the mobile phone camera, the ESP32-CAM simultaneously provides a live view of the remote robotic arm.

This is especially important for long-distance control because the operator may not be physically close to the robot.

The user can therefore:

1. Move their hand in front of the mobile camera.
2. Allow MediaPipe to detect X, Y, and Z movement.
3. Send robotic commands through the internet.
4. Move the physical robotic arm.
5. Observe the robotic movement through the ESP32-CAM stream.

This creates a complete remote visual teleoperation system.

---

Workspace Safety

Robotic safety is important because uncontrolled servo movement can damage the mechanical structure or move a joint beyond its safe range.

For this reason, each robotic joint operates within a predefined safe angle range.

For every servo joint:

$$ \theta_{\min} \leq \theta \leq \theta_{\max} $$

where:

• $\theta_{\min}$ is the minimum allowed servo angle.
• $\theta_{\max}$ is the maximum allowed servo angle.
• $\theta$ is the requested servo position.

If a MediaPipe-generated movement requests a position outside the allowed range, the movement can be limited or rejected.

This helps protect the robotic arm from excessive or unsafe movement.

---

Adaptive Motion Scaling

Human hand movements can vary greatly in speed and magnitude.

Directly converting every small hand movement into servo motion may produce unstable movement or servo jitter.

The magnitude of the detected hand displacement can be represented as:

$$ D= \sqrt{ (\Delta x)^2+ (\Delta y)^2+ (\Delta z)^2 } $$

The robotic movement can then be scaled as:

$$

M_{\text{robot}}

S(D)M_{\text{hand}} $$

where:

• $M_{\text{robot}}$ is the resulting robotic movement.
• $M_{\text{hand}}$ is the detected hand movement.
• $S(D)$ is the adaptive movement scaling factor.

With adaptive motion scaling:

• Small hand movements can produce smaller and more precise robotic movements.
• Larger hand movements can produce faster robotic movements.

This can make robotic teleoperation smoother and more natural.

---

Challenges we ran into

One of the biggest challenges was integrating several different technologies into one responsive real-time system.

The project combines:

• Mobile camera access
• MediaPipe
• X, Y, and Z hand tracking
• Mobile web application
• Render cloud deployment
• HiveMQ MQTT
• ESP8266
• ESP32-CAM
• Servo motor control
• 3-DOF robotic arm

Each component performs a different task, so communication between all parts of the system had to work correctly.

Communication Latency

One of the major challenges was latency.

Any significant delay between human hand movement and robotic response can make remote robotic control difficult.

The total system latency can be represented as:

$$

T_{\text{total}}

T_{\text{camera}} + T_{\text{MediaPipe}} + T_{\text{network}} + T_{\text{MQTT}} + T_{\text{ESP8266}} + T_{\text{servo}} $$

where:

• $T_{\text{camera}}$ is the mobile camera capture delay.
• $T_{\text{MediaPipe}}$ is the hand-tracking processing delay.
• $T_{\text{network}}$ is the internet or Wi-Fi communication delay.
• $T_{\text{MQTT}}$ is the MQTT messaging delay.
• $T_{\text{ESP8266}}$ is the microcontroller processing delay.
• $T_{\text{servo}}$ is the mechanical servo response delay.

For remote teleoperation, all of these delays affect how quickly the robotic arm responds to the user.

---

Hand Landmark Instability

Another challenge was converting small and sometimes unstable MediaPipe hand-landmark changes into smooth robotic motion.

Even when a person's hand appears stationary, the detected coordinates may change slightly between frames.

These small variations can cause unwanted servo movement or jitter.

Therefore, movement scaling, thresholds, servo limits, and smoothing techniques are important for achieving stable robotic control.

---

Remote Visual Feedback

Remote robotic operation also depends on visual feedback.

If the ESP32-CAM video has excessive delay while the robotic control responds at a different speed, controlling the arm accurately becomes difficult.

Therefore, both:

• Control latency
• Video streaming latency

are important parts of the teleoperation experience.

---

Accomplishments that we're proud of

We are proud that we successfully combined computer vision, IoT communication, cloud services, mobile web technology, and embedded robotics into one working prototype.

Our main accomplishments include:

• Controlling a real 3-DOF robotic arm using natural hand movement.
• Using MediaPipe X, Y, and Z motion information for robotic control.
• Controlling Base, Shoulder, and Elbow without a physical joystick.
• Developing a mobile web-based robotic control system.
• Deploying the web application using Render.
• Using HiveMQ MQTT for IoT communication.
• Connecting MediaPipe computer vision with ESP8266-based robotic hardware.
• Integrating ESP32-CAM live video streaming.
• Implementing robotic joint safety limits.
• Creating a platform capable of remote robotic teleoperation.

One of our most important achievements was building a complete communication bridge between:

$$ \text{Human Hand Motion} \rightarrow \text{Computer Vision} \rightarrow \text{Cloud / IoT} \rightarrow \text{Physical Robot Motion} $$

Instead of controlling the robot using only buttons or a traditional joystick, the system allows a user to interact with a physical robotic arm through natural human hand movement.

---

What we learned

This project taught us that robotic teleoperation involves much more than simply controlling servo motors.

We learned how to integrate multiple technologies into a complete working system.

These technologies include:

• MediaPipe hand tracking
• Computer vision
• X, Y, and Z motion processing
• Mobile camera access
• 3-DOF robotic control
• ESP8266 programming
• ESP32-CAM video streaming
• MQTT communication
• HiveMQ
• Render deployment
• Mobile web development
• Cloud-to-device communication
• Servo motor control
• Workspace safety
• Adaptive motion scaling
• Latency management
• Remote robotic teleoperation

We also learned the importance of distributing tasks between multiple components.

In our architecture:

• The mobile device performs computer vision and hand tracking.
• MediaPipe extracts hand-position information.
• The web application processes and publishes robotic control data.
• Render hosts the web application and backend.
• HiveMQ MQTT manages IoT communication.
• The ESP8266 controls the physical robotic arm.
• The ESP32-CAM provides remote visual feedback.

This distributed architecture makes the system more practical than attempting to perform every task on a single microcontroller.

---

What's next for Mobile Web Robotic Arm Control Using MediaPipe

The next goal is to improve the system for more accurate, smooth, and reliable long-distance robotic teleoperation.

Future developments include:

• Enhancing long-distance teleoperation so users can control the robotic arm remotely through natural hand movements detected by MediaPipe.
• Improving communication latency so robotic arm movement responds more quickly to hand movement even when the operator is far away.
• Improving ESP32-CAM streaming so the operator receives smoother real-time visual feedback.
• Improving motion smoothing to reduce servo jitter caused by small changes in MediaPipe hand coordinates.
• Improving adaptive motion scaling for better precision and more natural robotic movement.
• Improving Z-axis estimation for more accurate forward and backward control.
• Expanding the robotic arm from 3-DOF to 5-DOF or 6-DOF.
• Adding a robotic gripper for object manipulation.
• Adding inverse kinematics.
• Adding object detection and recognition.
• Adding automatic object grasping.
• Implementing collision avoidance.
• Adding motion prediction.
• Improving latency compensation for internet-based remote operation.
• Combining autonomous robotic operation with human teleoperation.

The current prototype focuses on controlling the three main joints:

Base + Shoulder + Elbow

A future version can add more joints and a robotic gripper to increase the arm's manipulation capability.

---

Long-Term Goal

The long-term goal is to create a robotic system where an operator can remain in one location while controlling a robotic arm located far away using only natural hand movements and a mobile camera.

The mobile camera detects the operator's hand movement using MediaPipe.

The control information is transmitted through the internet to the ESP8266, which controls the physical robotic arm.

The complete remote teleoperation concept can be represented as:

$$ \boxed{ \text{Remote User} \rightarrow \text{Mobile Camera} \rightarrow \text{MediaPipe} \rightarrow (X,Y,Z) \rightarrow \text{Internet / IoT} \rightarrow \text{ESP8266} \rightarrow \text{3-DOF Robotic Arm} } $$

At the same time, the remote robotic environment provides visual feedback through the ESP32-CAM:

$$ \boxed{ \text{Remote Robotic Arm} \rightarrow \text{ESP32-CAM} \rightarrow \text{Internet} \rightarrow \text{User's Mobile Web Interface} } $$

Therefore, the overall system creates a two-way teleoperation architecture:

Control Direction:

$$ \text{User} \rightarrow \text{Robot} $$

Visual Feedback Direction:

$$ \text{Robot} \rightarrow \text{User} $$

This architecture can potentially be useful for:

• Remote inspection
• Hazardous environments
• Industrial maintenance
• Research
• Education
• Rescue operations
• Remote laboratories
• Robotic experimentation
• Locations where direct human presence may be difficult or unsafe

The ultimate vision is to make robotic control more natural by allowing human hand movement to become a direct interface between the operator and a remotely located physical robot.

Built With

• 3-dof
• computer-vision
• css
• embedded-systems
• esp32-cam
• esp8266
• hand-tracking
• hivemq
• html
• iot
• javascript
• mediapipe
• mobile-web
• mqtt
• real-time-control
• render
• robotic-arm
• robotics
• servo-motors
• teleoperation
• webrtc