Surgy

Meta Quest Headset with Haptic Gloves
Haptic Gloves Close Up
Physics Simulation
QNX Standalone Monitoring Setup

Inspiration

Surgery is physical.

A surgeon does not learn precision by watching a screen. Surgical skill is developed through repetition, hand control, grip, and the physical feeling of interacting with tools and tissue.

Virtual reality creates a safe place to practise procedures, but there is still a major disconnect: the virtual world often does not feel real, and completing a task does not necessarily tell us how the trainee's hands behaved while doing it.

We asked:

What if you could physically feel a virtual surgical environment while an embedded system independently analyzed how your hands moved?

That idea became SurgY.

SurgY combines custom haptic gloves, virtual reality, and a QNX-powered edge AI monitor to make surgical training more physical and measurable.

VR teaches the procedure. The gloves make it physical. QNX watches the hands.

What it does

SurgY uses custom wearable haptic gloves based on LucidGloves.

Each finger is connected to a retractable spool and potentiometer that measures finger flexion. An ESP32 samples all five fingers and sends the data to SteamVR, allowing the user's real finger movements to control a virtual hand.

Servo mechanisms provide force feedback by resisting finger movement through guided strings. When the virtual environment detects an interaction, the glove can physically resist the corresponding fingers, making virtual objects feel more present.

At the same time, glove telemetry is mirrored over USB to a Raspberry Pi 5 running QNX.

This creates two independent paths:

Bluetooth → OpenGloves → SteamVR for VR interaction and haptic feedback.
USB → QNX for real-time hand monitoring and AI analysis.

The QNX system operates independently from the VR experience, allowing SurgY to analyze the trainee without becoming a bottleneck in the simulation.

Real-time AI monitoring

On QNX, SurgY processes both hands independently using 20-frame sliding windows.

For each hand, we extract 15 features describing:

Finger position
Frame-to-frame movement
Short-term movement variability

These features are analyzed locally using mlpack K-nearest neighbours (KNN).

SurgY compares live hand behaviour against a recorded normal baseline. Movement that is close to the baseline receives a low anomaly score, while unusual or unstable behaviour produces a higher score.

All inference runs locally on the Raspberry Pi.

No cloud connection is required.

In one QNX validation test, controlled movement scored 0.066 and was classified as NORMAL, while intentionally erratic movement scored 0.602 and was flagged as an ANOMALY.

How we built it

We combined 3D-printed mechanical glove components, potentiometer-based finger sensing, servo-driven force feedback, ESP32 firmware, SteamVR, OpenGloves, and QNX.

The QNX monitor was written in C++17 and runs natively on a Raspberry Pi 5.

Its multithreaded architecture separates:

Left glove ingestion
Right glove ingestion
Feature extraction and mlpack inference
Monitoring and dashboard output

Higher-priority threads handle glove data, while lower-priority work handles visualization and logging. This allows SurgY to prioritize recent hand movement and keep monitoring both hands independently.

Our Tools

ESP32
LucidGloves
3D printing
Potentiometers
Servo motors
Bluetooth
USB serial
OpenGloves
SteamVR
Raspberry Pi 5
QNX
C++17
POSIX threads
mlpack
Armadillo

What's next

We want to collect larger real-user datasets and expand SurgY beyond finger flexion.

Future versions could include:

Hand position and orientation
Tool trajectories
Grip consistency
Tremor characteristics
Procedure-specific movement patterns
More realistic haptic resistance for different tools and tissues

Long term, we see SurgY as a hardware and edge-intelligence platform that can integrate with professional surgical simulators and provide trainees with measurable feedback on how their hands move over time.

Feel the surgery. Measure the skill.