Wall-eVN

A shared-autonomy robotics platform that seamlessly integrates EEG brain signals and eye-tracking computer vision for intuitive, hands-free robot control.

Comment

Inspiration

The idea for Wall-eVN comes from a deeply personal experience. My childhood friend recently suffered a stroke that left him with severe motor impairments in his hands. Watching him lose his physical independence was heartbreaking. However, one thing never changed—his lifelong passion for playing chess.

For people with severe motor disabilities, traditional interfaces like keyboards, mice, or joysticks can be completely unusable. I realized that before helping him return to the chessboard, we first needed to solve a more fundamental problem: how someone with limited motor ability can interact with the physical world again.

Wall-eVN was born from this idea. While the ultimate goal is to enable him to play chess autonomously, this project focuses on the first milestone: a shared-autonomy robotic system that allows users to select objects using only their gaze and brain signals, and have a robotic arm retrieve them.

What it does

Wall-eVN is a web-based shared-autonomy dashboard that enables completely hands-free robotic control.

The system combines:

  • Eye tracking to select an object
  • EEG signals from a MUSE 2 headset to confirm actions
  • Computer vision to detect objects
  • A robotic arm to perform pick-and-place actions

The workflow follows a 5-phase interaction pipeline:

  1. Pupil Track
    The user's gaze is tracked using a face camera.

  2. Object Detection
    A table camera identifies objects that can be selected.

  3. Gaze Mapping + Dwell Lock
    The gaze position is mapped onto the table and locks onto an object after a short dwell time.

  4. EEG Blink Confirmation
    The user intentionally blinks to confirm the action.

  5. Pick & Place
    The robotic arm retrieves the selected object and brings it to the user.

This allows users with limited mobility to interact with their environment without using their hands.

How we built it

We designed a modular full-stack architecture that separates perception, control, and hardware layers.

Frontend

Built with:

  • React
  • Vite
  • TypeScript
  • TailwindCSS

The frontend provides a real-time dashboard that includes:

  • live camera feeds
  • EEG signal monitoring
  • system logs
  • calibration interface
  • robot control status

Backend

The backend is built with FastAPI and communicates with the frontend using WebSockets for low-latency streaming.

This allows the system to stream:

  • video frames
  • gaze tracking data
  • EEG signals
  • robot state updates

in real time.

Vision System

The vision pipeline uses two main components:

Face Camera

We use MediaPipe Face Landmarker to extract iris landmarks and compute stable pupil centroids for gaze tracking.

Table Camera

We run YOLOv11-nano to detect objects on the table and generate selectable targets.

EEG Processing

EEG signals are captured from the MUSE 2 headset using:

  • muselsl
  • Lab Streaming Layer (LSL)

We analyze the frontal channels AF7 and AF8 to detect intentional blinks based on peak-to-peak amplitude thresholds.

Robot Control

To support both development and deployment, we implemented:

  • MockRobotArm for simulation
  • SafetyController for enforcing workspace limits and emergency stops

This allows safe development even when the physical robot is unavailable.

The Mathematics of Gaze Mapping

To allow the robot to pick up an object, we must convert coordinates between different reference frames.

The system maps camera pixel coordinates ((u,v)) to robot workspace coordinates ((X,Y)) using a projective homography matrix.

$$ \begin{bmatrix} x' \ y' \ w'

\end{bmatrix}

\begin{bmatrix} h_{11} & h_{12} & h_{13} \ h_{21} & h_{22} & h_{23} \ h_{31} & h_{32} & h_{33} \end{bmatrix} \begin{bmatrix} u \ v \ 1 \end{bmatrix} $$

The final robot coordinates are computed as:

$$ X = \frac{x'}{w'} $$

$$ Y = \frac{y'}{w'} $$

The homography matrix (H) is calculated during a 4-point calibration phase and stored in a configuration file.

Challenges we ran into

Asynchronous Data Streams

One of the biggest challenges was synchronizing multiple real-time data streams:

  • EEG signals at 256 Hz
  • camera frames at 30 FPS
  • robot control updates

Managing these streams without latency required careful Python threading and WebSocket optimization.

EEG Signal Noise

Raw EEG data is extremely noisy. Many artifacts can resemble blink signals. After experimentation, we found that rule-based blink detection using amplitude thresholds was more reliable than machine learning models for this hardware.

Calibration Accuracy

The system requires two calibration stages:

  1. pupil-to-screen mapping
  2. screen-to-table mapping

Even small calibration errors could cause the robot to miss objects, especially small ones.

Hardware Limitations

During development we often did not have access to the physical robot or GPU acceleration. To solve this we implemented fallback components such as:

  • MockRobotArm
  • PlaceholderDetector

These allowed development to continue without full hardware access.

Accomplishments that we're proud of

Accessibility Impact

We transformed a large personal goal—helping someone play chess again—into a practical technological milestone: hands-free object retrieval.

Hardware + AI Integration

We successfully integrated:

  • EEG brain-computer interface
  • real-time computer vision
  • robotics control
  • a web dashboard

into a single cohesive system.

Safety-first design

From the beginning we implemented:

  • workspace motion limits
  • emergency stop systems
  • robot safety controllers

to ensure safe human-robot interaction.

What we learned

Technically, this project helped us gain experience in:

  • asynchronous Python systems
  • WebSocket state management
  • computer vision pipelines
  • robotic kinematics
  • brain-computer interface data processing

More importantly, we learned that engineering can bridge the gap between physical limitations and human dreams.

What's next for Wall-eVN

Chess Integration

The next step is enabling the system to recognize and manipulate chess pieces on a board, bringing us closer to the original goal of autonomous chess gameplay.

Real Robot Deployment

We plan to deploy the system on a physical robotic arm using the xarm-python-sdk.

Advanced EEG Commands

Instead of using blinks for confirmation, we aim to develop machine learning models for EEG-based command detection, such as imagined hand movements.

Automatic Calibration

Future versions will use ArUco markers to automate camera-to-robot calibration and eliminate manual configuration.

Built With

Share this project:

Updates