Operating Room
Hand Calibration
Expert "Ghost" Mode Tutorial
Replay Review & AI Feedback

Steady

Practice surgical motion anywhere. Just a laptop, a webcam, and your hands.

Inspiration

Surgical skills are built through repetition.

The problem is that getting that repetition can mean expensive simulators, specialized hardware, limited lab time, or having an instructor available.

We kept coming back to one question:

Why can't you practice meaningful surgical motion from your own laptop?

That became Steady.

What it does

Steady turns a laptop webcam into a surgical motion trainer.

You choose a skill like needle passing, suturing, knot tying, or laparoscopic peg transfer and practice inside a 3D operating room using your hands to control virtual surgical instruments.

Before you start, an expert ghost generated from established surgical training datasets and expert-recorded motion trajectories demonstrates the movement.

Steady then compares your attempt against that reference across:

trajectory · precision · smoothness · wrist rotation · speed · timing · efficiency · coordination

During practice, you can get short voice corrections.

Afterward, Steady overlays your movement against the expert in a synchronized 3D replay, highlights the biggest deviations, and explains what to work on next.

We did not want to build something that just gives you another score.

We wanted you to actually understand how you moved differently.

What mattered most to us was making that kind of practice accessible.

Surgical simulation can depend on VR systems, physical trainers, dedicated labs, sensors, or other expensive equipment.

Steady needs:

a browser · a webcam · your hands

We are not trying to replace instructors or clinical simulation. We want to make the practice that happens between those sessions much easier to access.

That loop became the core of Steady:

watch → practice → measure → understand → repeat

How we built it

System architecture

┌─────────────────────────────────────────────────────────────────────┐
│                            Steady                                   │
│  ┌──────────────┐  ┌──────────────┐  ┌───────────────────────────┐ │
│  │Skills Library│  │3D Operating  │  │Learn / Practice /        │ │
│  │+ Progress    │  │Room          │  │Assessment                │ │
│  └──────────────┘  └──────────────┘  └───────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
┌─────────────────────────────────────────────────────────────────────┐
│                     Webcam Motion Tracking                          │
│  ┌──────────────┐  ┌──────────────┐  ┌───────────────────────────┐ │
│  │MediaPipe     │  │Natural-Pose  │  │Filtering + Hand-to-Tool │ │
│  │Hands         │  │Calibration   │  │Mapping                   │ │
│  └──────────────┘  └──────────────┘  └───────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
                                  │
                  ┌───────────────┴────────────────┐
                  ▼                                ▼
┌────────────────────────────────┐   ┌────────────────────────────────┐
│          User Motion           │   │         Expert Motion          │
│                                │   │                                │
│ ┌────────────┐ ┌─────────────┐ │   │ ┌────────────┐ ┌─────────────┐ │
│ │Virtual     │ │Recorded     │ │   │ │Expert Ghost│ │Reference    │ │
│ │Instruments │ │Trajectory   │ │   │ │            │ │Trajectory   │ │
│ └────────────┘ └─────────────┘ │   │ └────────────┘ └─────────────┘ │
└────────────────────────────────┘   └────────────────────────────────┘
                  │                                │
                  └───────────────┬────────────────┘
                                  ▼
┌─────────────────────────────────────────────────────────────────────┐
│                     Motion Evaluation                               │
│                                                                     │
│   Precision • Smoothness • Wrist Control • Timing • Coordination    │
│                                                                     │
│                   20-D Performance Profile                          │
└─────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
┌─────────────────────────────────────────────────────────────────────┐
│                         Tiger Data                                  │
│ ┌────────────┐ ┌──────────────┐ ┌──────────────┐ ┌───────────────┐ │
│ │PostgreSQL  │ │TimescaleDB   │ │Continuous    │ │Vector + BM25  │ │
│ │            │ │Hypertables   │ │Aggregates    │ │Search         │ │
│ └────────────┘ └──────────────┘ └──────────────┘ └───────────────┘ │
│                                                                     │
│   Motion History • Progress • Similar Attempts • Coaching Context   │
└─────────────────────────────────────────────────────────────────────┘
                                  │
                  ┌───────────────┴────────────────┐
                  ▼                                ▼
┌────────────────────────────────┐   ┌────────────────────────────────┐
│          AI Coaching           │   │          3D Replay             │
│                                │   │                                │
│ ┌────────────┐ ┌─────────────┐ │   │ ┌────────────┐ ┌─────────────┐ │
│ │Structured  │ │Retrieved    │ │   │ │User +      │ │Paths +      │ │
│ │Feedback    │ │Context      │ │   │ │Expert      │ │Deviations   │ │
│ └────────────┘ └─────────────┘ │   │ └────────────┘ └─────────────┘ │
└────────────────────────────────┘   └────────────────────────────────┘
                  │
                  ▼
             ElevenLabs Voice

Our core stack:

Frontend       Next.js • React • TypeScript
3D             Three.js • React Three Fiber
Vision         MediaPipe Hand Landmarker
Motion         One Euro Filter • Quaternions • Custom Evaluation
AI             Gemini 2.5 Flash • Structured Outputs • Retrieval
Voice          ElevenLabs
Data           Tiger Cloud • PostgreSQL • TimescaleDB • Hypercore
Search         pgvector • pgvectorscale • pg_textsearch

One rule mattered from the beginning:

AI explains the movement. It does not grade it.

The scores come from our own motion analysis.

How Tiger Data powers Steady

Tiger Data is what turns Steady from a one-time webcam demo into a real training platform.

Every attempt generates high-frequency instrument positions, rotations, errors, scores, exercise segments, and coaching events at roughly 20–30 samples per second.

We store that motion stream in TimescaleDB hypertables.

Then Continuous Aggregates turn all of those raw samples into progress over time:

best scores · average scores · precision trends · coordination trends · recurring weaknesses

We use Timescale hyperfunctions for p50 and p95 motion deviation, time-weighted errors, time buckets, and downsampled motion charts.

Every attempt also becomes a 20-dimensional performance vector.

Using pgvector and pgvectorscale, Steady can find previous attempts with similar movement patterns.

So instead of only saying:

“Your wrist rotation was late.”

Steady can eventually say:

“You've made this same mistake before. Here's how you're improving.”

We also use pg_textsearch with BM25 to retrieve the most relevant coaching material for the exact error our evaluator detects.

As the motion history grows, Hypercore lets us keep older telemetry efficient for analytics without throwing away the history.

Tiger gives Steady its:

motion history · progress analytics · vector similarity · coaching retrieval · long-term memory

Without Tiger, Steady can analyze one attempt.

With Tiger, it can understand how someone is developing across all of them.

How AI coaches the user

We did not want an AI model watching movement and guessing whether it looked right.

Steady calculates the motion first.

The AI receives the actual measurements, largest deviations, weakest metrics, similar past attempts, and relevant coaching material retrieved from Tiger.

It then turns that into concise feedback like:

“Your entry position was accurate, but your wrist rotation happened too late through the second half of the movement.”

ElevenLabs delivers selected feedback through voice so the user can stay focused on the surgical field.

The model explains the measurements.

It never invents them.

Challenges we ran into

The hardest part was making webcam controls actually feel natural.

At first, users had to hold their palms toward the screen.

It tracked well.

It also felt nothing like surgery.

We rebuilt calibration so users can keep their hands naturally down over a desk, with their starting posture becoming the neutral position for the instruments.

We also spent a lot of time balancing jitter against latency.

Too little filtering feels unstable.

Too much filtering feels like you are fighting the tools.

Accomplishments we're proud of

Steady works as one connected loop:

webcam → virtual instruments → expert ghost → motion analysis → Tiger Data → AI coaching → 3D replay → progress

And it does all of that with hardware people already own.

The part we are most excited about is that the system does not forget what happened when an attempt ends.

Every session becomes useful data for the next one.

What we learned

A good AI coach needs good evidence.

Instead of asking a model to guess whether someone's movement was correct, we calculate the motion first and let AI explain what the data already shows.

We also learned that one score only tells you so much.

A history of motion can tell you what keeps going wrong, what is getting better, and what you should work on next.

What's next

We want to add more surgical skills, improve our expert motion references, and work with surgical educators to understand which measurements matter most.

We also want instructors to be able to create their own exercises and eventually connect individual skills into larger procedure pathways.

Steady started with a simple idea: meaningful surgical practice should not depend on having access to expensive hardware.