Pyxis Lab

Inspiration

Surgical simulation helps students build procedural skills, but access often depends on expensive simulators, VR hardware, or dedicated training labs.

We wanted to test a simpler idea: can a laptop webcam become the input system for basic surgical-skills practice?

Pyxis Lab was built around that constraint — no specialised controllers, no headset, and no additional tracking hardware.

What it does

Pyxis Lab is a browser-based surgical skills simulator controlled through hand movements captured by a webcam.

Students can:

Control virtual laparoscopic instruments using both hands
Practise tasks such as peg transfer, pattern cutting, and knot tying
Receive performance metrics after each attempt
Get AI-assisted feedback on mistakes and areas for improvement
Repeat sessions and track progress

The goal is to make procedural practice more accessible using hardware students already own.

How we built it

Pyxis uses MediaPipe for real-time hand landmark detection through a standard webcam.

We map hand coordinates to virtual surgical instruments and process the movement to reduce jitter and make instrument control more stable.

The application is built using Next.js, React, and TypeScript, with Firebase for application data and Vercel for deployment.

During a session, Pyxis records performance signals such as task completion, errors, movement efficiency, and consistency. These metrics are then used to generate structured post-session feedback.

Webcam
  ↓
Hand landmark detection
  ↓
Motion processing
  ↓
Virtual instruments
  ↓
Surgical task
  ↓
Performance metrics
  ↓
AI-assisted feedback

Challenges we ran into

The hardest problem was converting webcam hand tracking into precise instrument control.

Hand landmarks can shift because of lighting, camera position, occlusion, and depth changes. For surgical-style tasks, even small amounts of jitter can make interaction difficult.

We also had to determine how to evaluate a session beyond simply checking whether a task was completed. This meant designing metrics around movement quality, errors, efficiency, and consistency.

Another challenge was keeping the system lightweight enough to run through a normal browser without requiring dedicated hardware.

Accomplishments that we're proud of

Built two-hand webcam control for virtual surgical instruments
Created interactive surgical-skills exercises that run in the browser
Added performance-based evaluation instead of simple pass/fail scoring
Connected session metrics with AI-assisted improvement feedback
Reduced the required hardware to a laptop and webcam

The part we're most proud of is that accessibility comes directly from the technical design: the webcam replaces specialised motion-tracking hardware.

What we learned

We learned that real-time computer vision for interaction is significantly harder than using computer vision for detection alone.

Stable hand detection does not automatically produce stable controls, especially for tasks requiring precision.

We also learned that useful AI feedback needs structured data behind it. Instead of asking an AI model to guess how a student performed, Pyxis first measures the session and then uses those metrics to produce understandable feedback.

What's next for Pyxis Lab

Next, we want to:

Improve hand tracking and depth estimation
Add more procedural training exercises
Develop stronger motion-efficiency and precision metrics
Add longitudinal progress tracking
Build an instructor dashboard for reviewing student performance
Test Pyxis with medical students and compare its metrics with established surgical-training methods

Our longer-term goal is to explore whether commodity computer vision can make procedural simulation available in places where dedicated training hardware isn't practical.