Inspiration

Deep brain stimulation works — but only if the electrode lands in the right few millimetres of the subthalamic nucleus, and only if it never nicks a vessel on the way in. Surgeons build that intuition over hundreds of cases. Robots have no way to practise at all.

We wanted a world where a robot could practise: a brain you can fly a probe through, where stimulating the wrong nucleus visibly makes the patient worse, and where "did the tremor stop?" is a number, not an opinion. A world model in the real sense — an internal, functional model of cause and effect that an agent can act and plan inside — with a video model as the window into it.

What it does

A closed agentic loop for DBS electrode placement:

A 3D labelled brain. A 64³ voxel volume with grey and white matter, STN, VIM, GPi, internal capsule, vessels, cerebellum and brainstem, jittered per episode seed. One hidden target coordinate.
A randomised surgery. Each episode drops the probe at a random entry pose above the cortex; the agent has 120 steps of WASD + look + stim.
A real reward. Distance shaping toward the target, +6 for therapeutic stimulation weighted by how close and which nucleus, −4 for capsular side effects (muscle contraction), −10 and instant death for a vessel haemorrhage, +20 for suppressing the tremor. Tremor starts at 1.00 and only falls when stimulation actually reaches the right tissue.
GLM-5.3-Flash as the surgeon. Served on Modal, it reads the probe-eye render plus numeric telemetry each turn and returns JSON actions with its reasoning. Every prompt it was sent and every raw reply is visible in the UI.
LingBot-World 2 as the visuals. The agent's exact keypresses are mirrored into a live LingBot-World 2 session (via Reactor), which repaints the run as a translucent colour-coded 3D brain scan, seeded from our own renderer.
A director agent. A second GLM instance turns hidden simulator state into visible scene events — a vessel glinting on the left, tissue changing texture — injected into the world model's prompt, so the pixels tell the truth without leaking coordinates.
Free-roam mode. The pilot sees only the generated world-model frames — no maps, no coordinates — while a hidden tracker scores it against the real anatomy. A pure world-model game with a real score.

How we built it

Three layers, deliberately separated:

Physics — a deterministic NumPy voxel environment (DBSEnv) with a gym-style step(action) → (pixels, state, reward, done). It owns anatomy, collisions, tremor dynamics, reward and termination. Always authoritative.
Rendering — our own raymarcher: a probe-eye view, an orbiting whole-brain hologram with synthetic tractography fibres, and axial/coronal/sagittal slice maps with the probe and target marked.
Agents and generative view — FastAPI + a single-screen telemetry UI; GLM-5.3-Flash on Modal for pilot and director; LingBot-World 2 over a live streaming session for the cinematic view, driven by seed image, prompt events and key events.

Challenges we ran into

Reasoning ate the answers. GLM returned empty or truncated JSON until we discovered its reasoning tokens were consuming the entire output budget; reasoning_effort: none fixed it instantly.
A world model has no coordinates. We wanted reward computed inside the generated world — impossible: it returns pixels, not state. The hidden-coordinate tracker was the honest answer, and it's what makes free-roam scoreable.
We tried to self-host LingBot-World 2. The open 14B weights need 8 GPUs and the released path is offline-only (precomputed camera trajectories, no per-step action API). ~$30/hr and no realtime interface, versus the hosted realtime session — so we used the hosted one.
The director lied to the pilot. It told the agent "vessel to the right" while the vessel was on the left; the agent strafed left and haemorrhaged. We gave it the vessel's real bearing.
The bottleneck wasn't the model. The world model streams ~24 fps; our page showed ~1. Every render was also blocking the event loop, so /status took 1.4 s. Frames cached per step, encoding moved off-loop, keypresses coalesced, MJPEG stream instead of polling: /status now 1.5 ms.
Capacity. Credits depleted, then 429s — so the UI degrades to the deterministic view and retries rather than dying.

Accomplishments that we're proud of

The agent genuinely solves it: repeated end-to-end runs at +32 to +41 total reward, tremor 1.00 → 0.01, ~40 of 120 steps, on freshly randomised anatomy it has never seen.
Both failure modes are real and reachable — vessel haemorrhage terminates at −10, premature stimulation costs −4 for a capsular side effect.
Full transparency: the exact prompt, the model's raw reply, and the per-step reward are all on screen. No black box, no cherry-picking.
Diagnosing a navigation pathology from the logs: the agent burned ~60 steps spinning when the target was directly below it, because yaw error is meaningless at steep elevation. Scaling yaw error by bearing elevation fixed it.
The whole thing runs as a watchable game — the model plays, we don't touch the keyboard.

What we learned

"World model" and "video model" are not the same thing, and the difference is exactly where the reward lives. Generated video is a window; the functional model has to be something you can query for consequences.
Vision-language models are far better with a little numeric grounding than with pixels alone: the moment we added aim_error_deg and distance, the feedback loop stopped being vibes.
Reward shaping is where the surgery knowledge actually goes. Every clinical fact — the capsule causes muscle contraction, GPi is a weaker target than STN, a vessel is unrecoverable — is a term in the reward function.
In an agentic loop, honesty in the narration matters more than richness. One wrong spatial claim from the director killed the patient.

What's next for Reinforcement Learning Brain Surgical Simulation

Train a real policy. GLM is a strong zero-shot pilot but a slow one; the environment runs thousands of headless episodes cheaply, so a small policy trained on this reward should beat it — and then the world model renders the trained agent's runs.
Latent planning. With a self-hosted realtime world model we could snapshot its KV state and let the agent imagine K candidate trajectories before committing to one — the real promise of world models.
Ground the anatomy in real scans. Swap the stylised voxel brain for segmented patient MRI/DTI so trajectories mean something clinically.
Sim-to-real. Map the action space onto a surgical arm's joint space, add instrument compliance and tissue deformation, and validate on a phantom skull before anything else.
A learned tremor model. Our tremor response is hand-written; fitting it to published DBS outcome data would make the reward clinically meaningful rather than plausible.