scalpal.

landing
scalpal companion dashboard
docs main page
launch page on quest
operating room
simulated robot acting on training data
robot hand learning curve
scalpal in action
spacetimedb
finchnode
coaching pre operation
scalpal coaching during operation
post operation results
architecture diagram
vitals via presage

Inspiration

Surgical training depends on repetition. Residents learn an operation by doing it many times under supervision, and before they reach that point there are few ways to practice the whole sequence. Simulators exist, but the capable ones are expensive and concentrated in teaching hospitals. We wanted to see how much of a real case a Quest 3S could reproduce: taking a history from a patient, reaching a diagnosis, and then performing the operation that diagnosis calls for, with feedback at each step.

We were also interested in what that practice produces. Surgical robots are trained on recorded demonstrations of hand and tool motion, and collecting those demonstrations is slow and costly. A learner practicing in VR generates exactly that kind of motion data as a side effect, so Scalpal records each attempt and replays it on a simulated robot hand.

FinchNode's synthetic health records made the cases credible. A real chart carries constraints that a generic case doesn't: a penicillin allergy that affects antibiotic choice, a patient on apixaban and aspirin, a latex allergy that changes how the room is prepared, and a nine-year-old with asthma whose mother answers most of the questions.

What it does

Scalpal takes a learner through a single surgical case in a Meta Quest 3S, from the first conversation with the patient to the end of the operation.

It starts on a board of synthetic patients from FinchNode. Each card shows the patient's chart (medications, allergies, labs, and problems) along with the risk flags Scalpal derives from it, such as bleeding risk from apixaban or reduced kidney function from a low eGFR. The learner picks a patient and enters a doctor's office, where the patient speaks with an ElevenLabs voice. After each thing the patient says, the learner chooses the next move from four options by tapping or saying it, working through history, examination and tests before committing to a diagnosis and a plan. Findings appear on screen, and the patient responds the way the person in the chart would.

The diagnosis determines the operation. In the operating room the learner chooses full VR with a virtual patient, or mixed reality, where generic teaching anatomy is fitted to a real person lying on a table. Scalpal, the voice coach, runs the Time-Out using the risks the learner found or missed in the office, then coaches through the procedure. The open appendectomy is a physical simulation: the learner marks and incises the skin, splits the muscle, clamps and ties vessels, and can cause bleeding that shows on the patient monitor and can kill the patient. In mixed reality, Presage reads the volunteer's real pulse and breathing through a phone camera and uses them as the patient's baseline vitals.

When the case ends, a recap explains what happened and why, and a web dashboard follows the session live through SpacetimeDB. The learner's controller motion also becomes a demonstration for a simulated robot arm and hand, which learns the incision-marking step from those demonstrations and is graded with the same checks as the learner.

We also built an agent skill that turns any FinchNode chart into a new playable case.

How we built it

The headset app is Unity 6 on OpenXR with the Meta XR SDK, shipped as one APK that moves through four scenes: the patient board, the office, the operating room and the recap. Voice runs through an ElevenLabs Agents WebSocket client we wrote in C#, so the patient and the coach stream audio and call tools directly from the headset. Both agents run on Claude Sonnet through ElevenLabs. In the office, spoken answers go through ElevenLabs Scribe and then a Claude Haiku classifier that maps what the learner said to one of the four options.

The case service is TypeScript on Hono. It reads FinchNode's demo API and derives the risk flags with deterministic rules; no model touches the chart. It then combines the chart with an authored plan and a procedure to produce the case the headset plays. The same service scores the interview and runs the coach, which follows the live state of the operation and decides when a hint is worth giving. Safety warnings are pre-rendered with ElevenLabs Flash, so a warning plays immediately instead of waiting on the model.

The surgery runs on a reducer that turns measured tool actions into tissue facts: where the blade crossed the skin, how deep it went, where a clamp sits relative to an injury, how fast the learner pulled. Each step of the operation is a set of checks over those facts. The coach needs this logic on the server and the headset needs it for grading, so it exists in both TypeScript and C#, and both versions must reproduce a shared 41-action golden log exactly. The physiology model that turns blood loss into vitals exists in four places for the same reason, with parity tests between them.

SpacetimeDB holds the shared state. Our module has 25 tables and 52 reducers covering sessions, attempts, the coach transcript, commands to the headset, WebRTC signaling for the dashboard and robot results. A scheduled reducer ticks the patient's physiology inside the database, so every client sees the same patient at the same moment.

The robotics side is Python. During the incision-marking step the headset records both controllers' poses and grip and posts them as demonstrations. A worker retargets them onto a simulated Franka Panda arm with a Shadow hand in MuJoCo, trains a behavior-cloning policy, grades each rollout with the checks the learner faced, and returns a replay video to the recap. Presage SmartSpectra runs as a separate small service that reads pulse and breathing from a phone pointed at the volunteer.

Challenges we ran into

Keeping two engines in agreement was the hardest constraint. The coach needs the state of the body to talk about it, and the headset needs the same state to grade the learner, so the surgery logic exists in both TypeScript and C#. Small numeric differences broke parity in ways that were hard to spot: a bleed rate of 0.3 ml/s stored as a float drifted from the TypeScript value, and a 0.8 threshold compared differently once a float was widened to a double. We now quantize every measurement to whole millimeters and degrees where it enters the engine, keep rates as doubles, and require both engines to replay a 41-action golden log with identical results.

The voice coach was hard to make useful rather than noisy. Our first version routed every mistake through the language model, which took one and a half to three seconds and turned an unsteady hand into a running commentary. Safety warnings now come from pre-rendered clips that interrupt whatever is playing, rough handling is capped at one mistake per tool and tissue every three seconds, and the model only speaks for explanations and hints.

The Quest surfaced problems the editor never showed. Tools trailed behind the controllers on the first headset build. The VR patient rendered as a translucent ghost at 13% opacity. The pause menu ignored the pointer, because physics doesn't step while time is frozen, so the new buttons had no colliders yet. And the app wouldn't start over adb until we launched it through the launcher intent.

Presage gave us 50 minutes of live measurement for the whole event. After a first test spent 45 seconds without a single reading, we built a guard around it: demo mode by default, sessions that end themselves after five minutes, a cutoff when no face is visible for 15 seconds, and a log of every live second. The guard earned its keep. The next run failed because the terminal had no camera permission, and the one after that returned black frames because the phone was face down. We used about 11 minutes in total.

Accomplishments that we're proud of

Getting the voice agents to work in real time was the first big milestone. The patient holds a conversation in character, answering from what's actually in their chart and reacting to whatever the learner does. The coach knows where the learner is in the operation and can be interrupted mid-sentence. Its tool calls also act on the scene: it can highlight a structure or hand over an instrument while it talks.

The surgery became something you can actually get wrong. Cutting off the marked line, skipping a tie or nicking the bowel all have consequences, and bleeding you don't control shows up on the monitor and can end the case. The headset and the coach run the same model of the body, so what the coach says always matches what the learner sees.

The chart carries through the whole case. A risk found in FinchNode data shows up in the office as something to ask about. At the Time-Out it becomes a check, and in the operating room it changes the coaching at the step it affects.

We got real vitals into a simulated operating room. Presage read a volunteer's pulse and breathing from a phone, and Scalpal used them as the patient's starting point before the simulated blood loss takes over.

The robot learns from the same practice the learner does. In simulation, a policy trained on demonstrations clearly outperformed replaying them, including when we moved the instrument mid-task, so the recording turned into something more than a playback.

Anyone can build on it. The case-authoring skill let an agent with no context write a new, valid case from a FinchNode chart on its first try.

What we learned

Language models are the wrong tool for anything that has to be fast or exactly right. We started by sending the coach every event and letting it decide what to say, and it was too slow for a warning and too inconsistent for a grade. The version that works keeps the model for conversation and explanation and puts everything else in deterministic code: the risk flags, the scoring, the safety warnings. The model got better once it had less to do.

Two codebases that have to agree need a shared test, not shared intentions. The TypeScript and C# engines drifted apart several times before the golden log, and the same lesson applied to the JSON passing between the service and Unity: a field one side forgets about doesn't throw an error, it just disappears. Contract tests that check both directions caught bugs no one would have spotted by reading the code.

Being honest about what's real made the product better, not weaker. Every value in Scalpal is labeled as chart data, authored teaching content, a measured reading or a simulation, and keeping those apart forced clearer decisions. The volunteer's real heart rate, for example, can't respond to a virtual bleed, so it became the baseline and the simulation adds to it.

And a headset changes the problem more than we expected. Things that worked in the editor failed on the Quest for reasons we couldn't have guessed from a desk, so the earlier something ran on the device, the cheaper it was to fix.

What's next for scalpal.

The robot needs real data. Right now it learns the incision-marking step from scripted demonstrations, and the pipeline from the headset to the training worker is in place. The next step is to collect demonstrations from actual learners and see whether a policy trained on people practicing outperforms one trained on scripts. That's the claim the whole project rests on, and we haven't tested it yet.

We want more operations than the appendectomy. The cholecystectomy already exists as a procedure the coach can run, with the anatomy modeled. Making it playable in the headset means building the gallbladder region of the operating room and treating the critical view of safety as a hard gate before any clip or cut, the way current guidelines recommend.

Hand tracking would replace the controller-driven gloves. A port of MediaPipe's hand model that runs on the Quest already exists on a branch, and real hand motion would make both the practice and the robot demonstrations closer to surgery.

Presage could do more than set a baseline. The volunteer's breathing could move the teaching anatomy with each breath, which is the same respiratory motion problem image-guided surgery has to solve.

The case library should grow without us. The authoring skill already turns a FinchNode chart into a valid case. What's missing is clinician review, so the cases that reach learners have been checked by people who actually perform these operations.