Inspiration

Our team lead is a spine surgeon who builds surgical simulators. In spine surgery done through a small camera and keyhole incisions (endoscopic spine surgery), the hardest thing to teach is how far to drill. Take an operation that drills away part of the bony roof over the spinal canal to take pressure off the nerves (a laminectomy), done from the left: the drill takes a completely different path depending on whether the surgeon frees only the side of approach (ipsilateral) or continues across the midline to the far side (contralateral). That decision is hard to put into words; today it is learned by watching over someone's shoulder. So we asked: if a surgeon's hand motion is recorded, can a robot learn that decision and follow it from a single sentence? This is imitation learning (IL): the robot learns by copying demonstrations instead of hand-written rules.

What it does

Demonstration rig: the surgeon(Donghyun Won) holds a replica endoscope and a replica drill (trigger button), each carrying a marker block with four retro-reflective spheres, and performs a unilateral biportal endoscopic (UBE) laminectomy over a hollow skin phantom. Registration uses five skin dimples. Everything under the skin is simulated; no patient data is used.
Tracking and simulation: an Intel RealSense D435 in infrared (848 × 480, 90 fps) tracks both tools (pose noise 0.03–0.09 mm). The tracked poses drive a virtual lumbar spine (L3 to sacrum) that renders the underwater endoscope view and removes bone wherever the 2 mm burr touches, on a 0.8 mm voxel grid.
Policy: NVIDIA GR00T N1.7, fine-tuned. Input: one 256 × 256 virtual endoscope image, the position, orientation and trigger of both tool tips, and an instruction sentence. Output: the next 40 steps of tool poses plus a "procedure progress" value (0 → 1); when progress passes 0.85 the policy ends drilling by itself.
Robot: two SO-101 arms carry the same marker blocks and close the loop through the same tracker and the same virtual spine, so on the robot the policy sees the same kind of image it saw in training. The approach to the start pose is scripted; during drilling, every motion comes from the policy. A safety gate limits speed and range and can stop the arms.

How we built it

90 demonstrations: one surgeon(Donghyun Won), left side, 30 each at L3, L4 and L5 (the 3rd, 4th and 5th lumbar vertebrae). Drilling took 34–218 s (mean 103 s) and removed on average 1,403 mm³ of bone.
Labels and instructions: each recording is labelled by whether the drilling crossed the midline (ipsilateral 31 · contralateral 34 · unspecified 25). Each extent has 15 instruction wordings (10 for unspecified), re-drawn every frame.
Dataset: LeRobot v3, 417 episodes, 178,068 frames at 15 fps.
Training: GR00T N1.7 (3B, vision-language backbone Cosmos-Reason2-2B) fine-tuned with LeRobot 0.6.1 on one NVIDIA RTX PRO 6000 Blackwell GPU (preemptible Nebius AI Cloud VM): 20,000 steps, batch 32, 2 h 12 min (2.6 steps/s). The top 4 language-model layers, the projector and the action head were trained; the rest was frozen. Before that, our first pilot fine-tunes (on an early set of 24 demonstrations) and a held-out evaluation ran as Nebius Serverless AI Jobs on the same GPU type.
Serving: during the 105 robot runs the policy server ran on the Nebius GPU VM. For judges, the policy server (HTTP JSON + bearer token) and the render server run in one container on a Nebius Serverless Endpoint (RTX PRO 6000). The fine-tuned weights are baked into the private image, so the policy is ready about a minute after the container starts.
Robot loop: asynchronous execution — every 8 frames an observation goes to the policy server on Nebius and the answer is blended into the running plan. Per-frame limits of 2 mm and 3° at the tip; the run stops if the two tips come within 5 mm.

Challenges we ran into

Human hands vs. low-cost arms: we mounted the surgeon's own marker blocks on the arms and kept the tool tips virtual, so the policy input on the robot matches the human demonstrations.
Timing: we built a recording format that puts camera exposure time, tracking and rendered frames on one clock.
Remote inference latency: from Seoul to the Nebius GPU, sending an observation and receiving a checked answer took a median 1.8 s (model inference 117–134 ms). Asynchronous execution — fetching the next chunk while executing the current one — keeps the arms moving.
Knowing when to stop: the robot stopped earlier than the surgeon. Knowing when the procedure is complete is still open.

Accomplishments that we're proud of

105 robot runs over three levels (per level: 16 contralateral, 16 ipsilateral and 3 neutral instructions, from four start poses).
Over the three levels, the 48 runs told to go contralateral removed on average 9.9 % of their bone across the midline, against 0.4 % for the 48 told to stay ipsilateral (p < 0.001).
With wordings never seen in training: 9.8 % vs 0.2 % over the three levels. Per level: L4 18.1 % vs 0.6 % and L5 10.6 % vs 0.0 % (p < 0.001 each; at every start pose every contralateral run was above every ipsilateral run). At L3 the difference was small: 0.6 % vs 0.1 % (p = 0.053).
In the video: L4 run 17, told to go contralateral, 26.8 % (the furthest of all contralateral runs); run 19 from the same start pose, told to stay ipsilateral, 0.5 % (the run closest to its group mean).
The analysis (exact permutation tests within start pose) was fixed after the first 31 runs; every later run followed that design. The fraction was cross-checked with an independent estimate from the tracked tip positions (ρ = 0.90).

What we learned

With only 90 demonstrations, a single instruction sentence made the policy choose a different surgical strategy from the same start pose.
Giving the policy the same simulated "eyes" for hand and robot makes the human-to-robot transfer much easier.
Progress and stopping are much harder to learn than motion.
We fixed the analysis after the first 31 runs, kept it for the remaining 74, and cross-checked the measurement two ways; the L4 estimate may still be slightly optimistic.

What's next for Spine IL: a surgeon's hands teach robot arms where to drill

Where to drill and when to stop: contralateral reach was at most 5.2 mm, against a demonstrated mean of 10–12 mm, and 14 of the 48 contralateral runs were ended by a supervisory timeout (counted as failures). Both remain to be learned.
Stronger evidence: more surgeons and anatomies, several fine-tuning seeds, execution closer to demonstration speed, and a check on real bone or tissue.

Limitations. The simulator that rendered the training images also scored the robot runs; nothing was validated on real bone or tissue. One surgeon, one virtual anatomy, start poses inside the demonstrated range, a single fine-tuning run (one seed) and execution at one third of demonstration speed are all favourable conditions. The unseen wordings still share cue words such as "only" and "opposite side" with the trained ones, so whether the policy understands the meaning is an open question. A closed loop run entirely in the simulator does not reproduce the robot result: for the same tool path the simulator's step mode removes about 60 % of the bone the robot's recording page removed, so a policy-driven simulated drill stops short of the midline whatever the sentence. The robot results are the 105 recorded runs, whose camera videos are released.

Release.

Code: GitHub, MPL-2.0. Fine-tuned weights: Hugging Face, NVIDIA Open Model License, research and evaluation use. Data (LeRobot dataset, the 90 raw hand recordings, level meshes, the recorded robot runs used by the judge scripts, camera videos of all 105 robot runs): Hugging Face, CC BY-NC 4.0.
The rendering engine is closed source; it is offered free of charge as a hosted API on the Nebius Serverless Endpoint through the judging period (2026-12-08 to 2026-12-23, Pacific Time).
Research use only. Not a medical device.