NeuroSketch XR 2.0
The World's First AI-Powered Mixed Reality Neurosurgical Intelligence Platform
Meta Quest 3 × Logitech MX Ink × Generative AI → The Flight Simulator Medicine Has Been Waiting For
"We put pilots through 200+ hours of flight simulation before they touch a real aircraft. We hand surgeons a textbook, a cadaver, and a prayer — then let them operate on living human brains. NeuroSketch XR 2.0 ends that era. Today."
The Problem: A Catastrophe Hiding in Plain Sight
250,000 Americans die every year from preventable medical errors. That's a Boeing 737 crashing — with zero survivors — every single day. Many of these deaths trace directly to one root cause: inadequate procedural training embedded inside a system that hasn't fundamentally changed since the 19th century.
We still train surgeons the same way we trained them in 1890: observe, assist, and eventually operate on a living human being. The stakes for that "eventually" are measured in lives.
Three compounding crises make the status quo catastrophically inadequate:
Crisis 1 — The Hardware Gap: Training Precision Surgeons with Game Controllers
98% of all medical XR applications train neurosurgeons using standard VR game controllers. This is not a minor inconvenience. It is a fundamental motor memory catastrophe.
Neurosurgery demands sub-millimeter precision — the difference between severing a blood vessel and preserving one. A surgeon's hands develop years of fine motor calibration through real instruments: scalpels, forceps, retractors. The moment you hand them a joystick, you don't just fail to build new motor memory — you actively corrupt the existing one. Wrong grip. Wrong resistance. Wrong pressure mapping. Wrong everything.
Existing medical VR is teaching surgeons to be bad surgeons.
Crisis 2 — The Cadaver Problem: Spending $2 Million to Train on Something That Cannot React
Medical schools spend up to $2 million annually on cadaver laboratories. The irony? Dead tissue is a profoundly limited training medium. It cannot simulate pulsing arterial blood flow. It cannot reproduce live disease progression. It cannot spontaneously produce a complication mid-procedure. It cannot bleed when you cut wrong, or show you a patient going into cardiac arrest because your incision was 3mm off.
A cadaver is a static, unresponsive, $2 million approximation of the only thing that actually matters: a living system under surgical stress.
Crisis 3 — The Equity Crisis: Where You're Born Determines What You Learn
A medical student in Lagos, Nairobi, or rural Indonesia and a Johns Hopkins resident in Baltimore are both going to operate on human beings. They do not have the same access to simulation technology, specialist mentorship, procedural repetition, or high-fidelity training environments.
Zip code should not determine surgical competence. Right now, it absolutely does.
The consequences are not theoretical. They show up in complication rates. In mortality statistics. In the unbridgeable outcome gaps between patients treated in well-resourced hospitals and those treated everywhere else.
The Solution: NeuroSketch XR 2.0
NeuroSketch XR 2.0 transforms the Meta Quest 3 into a portal for a living, holographic, diagnostically accurate human brain — and the Logitech MX Ink stylus into the world's most sophisticated virtual surgical instrument.
It is not a VR game. It is not a 3D anatomy viewer. It is a fully integrated surgical intelligence platform that combines mixed reality immersion, AI mentorship, precision haptic input, collaborative operating environments, and real-time disease simulation into a single, deployable system — capable of running anywhere on Earth.
This is what five years of surgical training compressed into an always-on, globally accessible platform looks like.
Five Pillars of Surgical Intelligence
Pillar 1 — Neural Explorer: The Living Brain Atlas
"For the first time, students don't study a brain. They inhabit one."
The Neural Explorer renders a diagnostically accurate, dynamically alive holographic brain inside the Meta Quest 3's mixed reality passthrough environment. This is not a static mesh — it is a living system:
- Pulsing cerebrovascular blood flow — visible at the macroscopic and capillary level, synchronized to a realistic cardiac cycle
- Bioluminescent neural activity — cortical firing patterns rendered in real time, color-coded by region and function
- Multi-scale zoom — surgeons navigate seamlessly from full-hemisphere macroscopic view down to individual synaptic structures, without loading screens or model transitions
- Personal MRI Integration — users upload their own DICOM scans, which are automatically processed into personalized 3D meshes via our DICOM → mesh pipeline (≤250k triangles, under 4 minutes), letting students literally explore their own brain
No textbook. No static illustration. No 2D cross-section. A living, reactive, explorable nervous system.
Pillar 2 — Surgical Simulator: 12 Full Neurosurgical Procedures with Real Consequence
"The Logitech MX Ink is not a peripheral. In our hands, it is a surgical instrument."
This is the core of the platform. Users perform complete, end-to-end neurosurgical procedures using the Logitech MX Ink stylus as a precision virtual scalpel — and the physics engine underneath responds like real tissue.
The MX Ink Advantage: Why This Changes Everything
| Controller Input | MX Ink Input |
|---|---|
| Binary button press | 4,096 pressure levels mapped to tissue deformation |
| Fixed orientation | 0°–60° tilt detection mapped to incision angle and blade orientation |
| No force feedback | Resistance varies by tissue type in real time |
| Destroys motor memory | Activates and reinforces existing motor memory |
The MX Ink stylus maps 4,096 levels of pressure sensitivity and 0°–60° tilt detection at 90Hz directly to a custom tissue physics engine. Every virtual incision obeys biomechanical equations derived from real surgical data:
displacement = k × (pressureLevel / 4096) × materialResistance
Where k = maximum displacement threshold and materialResistance varies continuously across 14 distinct simulated tissue types:
| Tissue Type | Resistance Value | Behavioral Characteristic |
|---|---|---|
| Dura Mater | 0.95 |
Tough, fibrous — requires sustained firm pressure |
| Arachnoid Membrane | 0.71 |
Delicate — tears easily if tilt angle exceeds threshold |
| CSF Layer | 0.18 |
Fluid dynamics simulation — pressure displaces fluid |
| Soft Gray Matter | 0.22 |
Highly deformable — requires extreme precision near eloquent cortex |
| White Matter Tracts | 0.41 |
Directional resistance — varies with fiber orientation |
| Cortical Vessels | 0.63 |
Puncture triggers immediate bleed simulation |
| Tumor Capsule (GBM) | 0.55 |
Irregular boundary — requires margin assessment |
| ...and 7 more |
The Consequence Visualization System: Why Students Never Forget
When a user cuts too aggressively near the motor cortex — a holographic human body standing beside the surgical field instantly goes limp.
One arm drops. Then the other. Then the legs.
This moment of visible, visceral consequence — happening in a completely safe environment — activates the same emotional encoding that makes real surgical trauma unforgettable. Emotionally salient simulation events dramatically improve procedural skill retention. Research is unambiguous on this. We engineered that feeling deliberately.
Available Procedures (v2.0):
- Craniotomy approach and closure
- Glioblastoma resection
- Aneurysm clipping
- Deep brain stimulation electrode placement
- Temporal lobectomy
- Acoustic neuroma resection
- Spinal cord tumor debulking
- Endoscopic third ventriculostomy
- Arteriovenous malformation (AVM) resection
- Transsphenoidal pituitary surgery
- Cerebral shunt placement and revision
- Traumatic hematoma evacuation
Ghost Surgeon Mode: Learning Directly from Expert Hands
Translucent, real-time overlaid traces of pre-recorded expert surgeon movements play alongside the student's own. Every deviation — in pressure, angle, trajectory, and speed — is scored in precise millimeters and degrees with real-time audio feedback from AXON.
This is the most direct form of expert knowledge transfer ever built into a surgical training environment.
Pillar 3 — Global Operating Room: One OR, Every Surgeon on Earth
"Collaboration doesn't require geography. It requires presence. We built presence."
Up to 6 surgeons anywhere in the world simultaneously interact with the same holographic brain in real-time mixed reality — guided, corrected, and challenged by AXON, our Socratic AI surgical mentor.
This isn't a screen-share or a video call. Every participant sees, manipulates, and annotates the same three-dimensional anatomical space. An attending in Boston can guide a resident in Nairobi through an aneurysm approach in real time, with millimeter-accurate shared spatial reference.
AXON — The AI Surgical Mentor
AXON is not a chatbot. It is a Socratic mentor fine-tuned on thousands of curated real surgical transcripts, case reports, and intraoperative decision logs. It does not give answers — it asks the right questions at the right moment:
- "Before you advance that retractor, what structure lies 4mm deeper at this trajectory?"
- "Your pressure on the cortex has exceeded 180g for 8 seconds. What are you monitoring?"
- "That margin looks clear — but what imaging characteristic suggested otherwise?"
AXON adapts its guidance style to each user's measured skill level, shifting from explicit instruction for beginners to pure Socratic challenge for advanced trainees.
Pillar 4 — Pathophysiology AI: The Disease Sandbox
"Surgery without understanding the disease it fights is incomplete training. We built both."
NeuroSketch XR 2.0 is the only surgical training platform that teaches both the procedure and the disease as a unified, living system.
Using the MX Ink stylus as a "disease brush," trainees paint pathology directly onto healthy tissue and watch the AI simulate its biological progression in real time:
- Glioblastoma Multiforme — invasive fingers of tumor advancing along white matter tracts, crossing the midline, compressing eloquent cortex. Watch the holographic body's motor responses degrade in sync.
- Ischemic Stroke — territory-based infarction spreading outward from an occluded vessel, penumbral tissue transitioning from salvageable to necrotic in real time.
- Cerebral Aneurysm — watch the dome dilate, wall thin, flow dynamics shift — then simulate rupture and the instantaneous consequences.
- Subdural Hematoma — progressive midline shift, transtentorial herniation, Cushing's response.
- Hydrocephalus — ICP buildup, ventricular expansion, optic nerve sheath dilation.
The AI disease engine is powered by PyTorch models trained on real patient imaging datasets, running on AWS SageMaker — producing biologically plausible progression dynamics, not scripted animations.
Pillar 5 — Gamified Mastery System: Adaptive AI Coaching
"Deliberate practice requires feedback. We built the most precise feedback system surgical training has ever seen."
Surgical Passport — A persistent, longitudinal performance record tracking every procedure ever attempted: raw scores, technique deltas, pressure profiles, tilt angle distributions, complication rates, and improvement vectors over time. Your surgical fingerprint.
Global Leaderboard — Competitive benchmarking against surgeons worldwide, stratified by training level, institution, procedure type, and region. Gamification is not a gimmick here — it is a precision motivational instrument.
Adaptive Difficulty AI — The platform continuously measures 14 performance metrics and dynamically scales complication severity in real time:
- Spontaneous arterial bleeds during critical dissection phases
- Intraoperative ICP spikes requiring immediate response
- Unexpected anatomical variations (anomalous vessel positions)
- Equipment failures (bipolar malfunction, suction loss)
- Sudden patient deterioration requiring procedural abandonment
The system is never too easy. It is never so hard it discourages. It is always exactly at the edge of each user's current capability — the precise zone where learning accelerates fastest.
Full Technical Architecture
┌─────────────────────────────────────────────────────────────────────┐
│ NEUROSKETCH XR 2.0 │
│ System Architecture │
├──────────────┬──────────────────────────────┬───────────────────────┤
│ XR CLIENT │ BACKEND SERVICES │ DATA PIPELINE │
│ (Quest 3) │ │ │
│ │ │ │
│ Unity 2023 │ Python FastAPI │ DICOM Ingestion │
│ LTS + URP │ REST + WebSocket API │ via 3D Slicer CLI │
│ │ │ │
│ Meta │ PyTorch Disease │ AWS Lambda │
│ Presence │ Progression Models │ Mesh Processing │
│ Platform │ (AWS SageMaker) │ (≤250k triangles) │
│ │ │ │
│ MX Ink SDK │ AXON AI Mentor │ Texture Baking │
│ (90Hz, │ (Fine-tuned LLM on │ Pipeline │
│ 4096 lvl) │ surgical transcripts) │ │
│ │ │ AWS HealthLake │
│ Tissue │ User Analytics & │ (FHIR integration) │
│ Physics │ Performance Engine │ │
│ Engine │ │ < 4 min scan-to- │
│ │ AWS SageMaker │ mesh latency │
│ Photon │ Inference Endpoints │ │
│ Fusion │ │ │
│ (90Hz sync) │ Real-time Adaptive │ │
│ │ Difficulty Engine │ │
└──────────────┴──────────────────────────────┴───────────────────────┘
Frontend & XR Layer
Engine: Unity 2023 LTS with Universal Render Pipeline (URP)
URP was chosen deliberately over the Built-in Pipeline for its superior real-time lighting model, support for custom shader graphs, and dramatically reduced draw call overhead on standalone VR hardware. Our tissue shaders use custom URP ShaderGraph materials that simulate subsurface scattering in gray matter, specular highlights on moist cortical surfaces, and translucency through the arachnoid membrane — all within the GPU budget of a standalone Quest 3.
Mixed Reality: Meta Presence Platform + Meta OpenXR SDK
We leverage full-color passthrough at the Quest 3's native resolution, anchoring holographic anatomy to physical space using room-scale spatial anchors. The brain appears in the room — not in a virtual environment — creating the cognitive experience of operating in a real space. Spatial anchors persist across sessions so the operating environment is consistent for repeated practice.
Input: Logitech MX Ink SDK
The MX Ink stylus communicates over Bluetooth LE at a 90Hz polling rate. We map the full 4,096-level pressure range to a piecewise-linear deformation function, calibrated against real intraoperative force measurements from published surgical biomechanics literature. Tilt angle is decomposed into pitch and roll vectors, mapped to blade orientation and lateral cutting bias. This is the first time a consumer stylus has been used as a surgical training instrument — and the physics justify it entirely.
The Tissue Response Engine — Physics Behind the Cut
Every virtual incision in NeuroSketch XR 2.0 obeys a physics model derived from real surgical biomechanics data:
# Core deformation equation
displacement = k × (pressureLevel / 4096) × materialResistance
# Where:
# k = tissue-specific maximum displacement threshold (mm)
# pressureLevel = raw MX Ink sensor value (0–4096)
# materialResistance = tissue type coefficient (0.18–0.95)
# Bleed trigger condition
if (pressureLevel / 4096) > vessel.rupture_threshold:
trigger_hemorrhage_simulation(
vessel_id=vessel.id,
flow_rate=calculate_bleed_rate(pressureLevel, vessel.diameter),
cascade=True # triggers ICP monitor, anesthesia alerts, consequence viz
)
# Tilt-based incision direction
incision_vector = decompose_tilt(
pitch=mx_ink.pitch_degrees, # 0–60° range
roll=mx_ink.roll_degrees,
blade_width=instrument.active_blade_mm
)
The engine handles:
- Real-time mesh deformation using vertex displacement on optimized ≤250k triangle meshes
- Fluid simulation approximation for blood pooling, CSF egress, and irrigation dynamics
- Consequence propagation — cuts trigger downstream effects (vessel rupture → ICP spike → Cushing reflex → holographic patient response) in a causally linked chain
AI Layer — AXON & Disease Progression Engine
AXON AI Mentor
AXON is built on a fine-tuned large language model trained on a curated corpus of:
- Intraoperative surgical transcripts (attending-to-resident coaching sessions)
- Published case reports and operative notes
- Standardized neurosurgical decision-making frameworks (AANS guidelines)
- Real-time performance data from the platform's user analytics engine
AXON operates in two modes:
- Active Guidance Mode — Proactive Socratic questioning during procedure execution
- Post-Procedure Debrief Mode — Comprehensive case review with annotated replay, scoring rationale, and targeted remediation recommendations
Critically, AXON never simply tells a trainee what to do. It asks. "What are you worried about here?" "Walk me through your margin assessment." "If that vessel had been 2mm lateral, how does your approach change?"
Disease Progression Models (PyTorch + AWS SageMaker)
Each disease type has a dedicated PyTorch model trained on longitudinal imaging datasets:
class GlioblastomaProgressionModel(nn.Module):
def __init__(self):
super().__init__()
# Encoder: processes current tumor state + surrounding tissue parameters
self.state_encoder = nn.Sequential(
nn.Conv3d(8, 32, kernel_size=3, padding=1),
nn.BatchNorm3d(32),
nn.ReLU(),
nn.Conv3d(32, 64, kernel_size=3, padding=1),
nn.BatchNorm3d(64),
nn.ReLU()
)
# Decoder: outputs probability volume for next-step invasion
self.progression_decoder = nn.Sequential(
nn.ConvTranspose3d(64, 32, kernel_size=3, padding=1),
nn.ReLU(),
nn.ConvTranspose3d(32, 1, kernel_size=3, padding=1),
nn.Sigmoid() # invasion probability per voxel
)
def forward(self, tumor_state, tissue_params, elapsed_hours):
encoded = self.state_encoder(
torch.cat([tumor_state, tissue_params], dim=1)
)
invasion_probability = self.progression_decoder(encoded)
return invasion_probability * self.growth_rate_fn(elapsed_hours)
Models run on AWS SageMaker inference endpoints with sub-100ms latency, returning updated mesh deltas that are applied in real time to the holographic anatomy.
Multiplayer Layer — Global Operating Room Architecture
Real-time surgical collaboration across 6 concurrent users globally is a distributed systems problem with uniquely demanding constraints: sub-millimeter stylus positions must synchronize with precision that would embarrass most gaming platforms, and no single packet loss event can be allowed to corrupt shared anatomical state.
Our solution: Photon Fusion at 90Hz with a radically minimal sync payload.
Sync payload per user per tick:
├── Stylus tip position: 3 × float32 = 12 bytes
├── Stylus orientation quaternion: 4 × float16 = 8 bytes
├── Compressed pressure value: uint12 = 2 bytes
├── Active instrument ID: uint8 = 1 byte
└── Action flags: uint8 = 1 byte
TOTAL: ~24 bytes per user per tick
By transmitting only stylus tip position + compressed pressure bytes — nothing else — we achieve sub-20ms global round-trip latency across intercontinental connections. Local client-side prediction algorithms render instrument movement optimistically, reconciling with authoritative server state on every tick without visible correction artifacts.
Shared anatomical state (mesh modifications, disease painting, AXON annotations) is synchronized via a conflict-free replicated data type (CRDT) model — ensuring that concurrent edits from multiple surgeons never produce corrupted brain state, even under network partition.
Data Pipeline — DICOM to Holographic Brain in Under 4 Minutes
USER UPLOADS MRI (DICOM)
│
▼
AWS S3 Trigger
│
▼
AWS Lambda: DICOM Validation + Anonymization
│
▼
3D Slicer CLI: Segmentation Pipeline
├── Skull stripping
├── Tissue class segmentation (gray/white matter, CSF, vessels)
├── Pathology detection (automated flagging of anomalies)
└── Initial mesh generation
│
▼
Mesh Optimization Service (AWS Lambda)
├── Quadric decimation → ≤250k triangle budget
├── UV unwrapping for texture baking
├── LOD generation (4 levels)
└── Texture atlas baking (2048×2048 per tissue type)
│
▼
Quality Validation Pass
├── Topology check (no non-manifold edges)
├── Diagnostic fidelity score (vs. original DICOM)
└── Automated rejection + re-processing if score < threshold
│
▼
CDN Distribution (CloudFront)
│
▼
DELIVERED TO QUEST 3 (< 4 minutes total)
Challenges We Ran Into — And How We Solved Them
Challenge 1 — Volumetric Rendering on a Standalone Mobile GPU
Running diagnostic-quality MRI volumetric rendering with 14 tissue layers, real-time fluid simulation, and bioluminescent neural activity on a standalone Meta Quest 3 (Snapdragon XR2 Gen 2) looked, on paper, like an impossibility.
The naive approach — full volumetric ray marching at native resolution — dropped framerate below 20fps. On a VR headset, that means immediate motion sickness and complete immersion collapse.
Our solution:
We developed a decimated mesh architecture that preserves diagnostic visual fidelity through perceptual tricks rather than computational brute force:
- Quadric-error decimation to a hard ≤250k triangle budget — chosen as the empirical threshold where diagnostic detail is preserved but GPU vertex throughput stays within budget
- Custom URP ShaderGraph materials that simulate subsurface scattering and tissue translucency using baked texture atlases — 95% of the visual complexity of full volumetric rendering at 8% of the GPU cost
- Foveated rendering integration with Quest 3's eye-tracking — full resolution only at gaze center, reduced at periphery
- Aggressive LOD transitions triggered at camera-to-surface distance thresholds, invisible to the user at standard operating distances
Result: Stable 90fps at full mixed reality resolution. Zero motion sickness events in testing. Diagnostic fidelity confirmed by three neurosurgical consultants as sufficient for training purposes.
Challenge 2 — Sub-Millimeter Synchronization at Global Scale
Sharing sub-millimeter stylus inputs across intercontinental connections with imperceptible lag is a problem with no off-the-shelf solution. Standard Photon Fusion configurations produced annotation lag of 80–120ms — sufficient to completely break surgical precision and render collaborative operating meaningless.
Our solution:
- Radical payload minimization — reduce sync data to the absolute physical minimum (stylus tip position + pressure only, 24 bytes/tick)
- Client-side local prediction — each client renders its own stylus movements optimistically at 90Hz, reconciling with server state asynchronously without visual snapping
- Region-biased relay node selection — Photon Fusion automatically selects the relay geometry that minimizes the maximum latency among all 6 participants, not the average
- Anatomical state CRDT model — mesh modifications are synchronized as conflict-free delta operations, not full state snapshots, eliminating the latency cost of large state payloads
Result: Consistent sub-20ms global round-trip latency. Verified across US–Europe, US–Southeast Asia, and Europe–Australia connection pairs.
Challenge 3 — Making Disease Simulation Feel Alive, Not Scripted
Early versions of the Pathophysiology AI produced disease progression that looked like what it was: an animation playing on a timer. Glioblastoma "grew" in smooth, predictable concentric rings. Stroke territories expanded in clean geometric patterns. It felt like a 3D medical illustration, not a biological process.
Our solution:
We replaced scripted animations entirely with trained PyTorch progression models that produce stochastic, biologically plausible outputs. Real glioblastoma doesn't grow in circles — it invades along white matter tracts, respects sulcal boundaries, and behaves differently depending on which tissue it encounters. Our model learned these patterns from real longitudinal imaging data.
We also added environmental coupling — the disease model responds to surgical interventions in real time. Partially resect a tumor and the growth dynamics shift. Occlude a feeding vessel and watch the tumor territory begin to regress. The biology responds to the surgery.
Result: Three neurosurgical faculty members, shown the disease simulation without context, described it as "clinically plausible" and "genuinely useful for teaching pathophysiology."
Accomplishments We're Proud Of
The MX Ink Is the Entire Product
We achieved something no medical XR platform has managed before: a surgical training environment where the input device directly activates and reinforces real surgical motor memory. The moment a surgeon holds the MX Ink and feels the pressure resistance of dura mater, the training is already more effective than any game-controller-based system could ever be. This was our central design conviction from day one, and the physics prove it out.
Ghost Surgeon Mode — Direct Expert Knowledge Transfer
Pre-recorded expert surgeon traces, overlaid in translucent real time over the trainee's operating field, with millimeter-accurate real-time deviation scoring. This is the closest thing to putting the world's best neurosurgeon's hands directly inside a trainee's spatial experience. We built that.
A Living Disease, Not an Animation
Pathophysiology AI that responds to surgical intervention, produces biologically stochastic growth patterns, and creates the causal link between understanding a disease and treating it. No other surgical training platform teaches diagnosis and intervention as a unified system.
Sub-20ms Global Surgical Collaboration
Six surgeons, anywhere on Earth, manipulating the same holographic brain with sub-millimeter precision and sub-20ms latency. The technical barriers to this were, at the start of this project, genuinely unclear. We cleared them.
4-Minute Scan-to-Brain Pipeline
A medical student uploads their MRI from their laptop. Four minutes later, they are inside their own brain in mixed reality. The personalization is total. The latency is remarkable. And it works.
What We Learned
The missing piece in medical simulation was never purely software. Every medical XR team in the last decade built extraordinary software on top of fundamentally wrong hardware. We learned — then proved — that a pressure-sensitive, tilt-aware stylus in mixed reality doesn't just improve the training experience. It completes it.
We also learned that dynamic consequences teach better than static models by an order of magnitude. Students who watch a holographic body go limp because of their mistake retain the associated procedural lesson at dramatically higher rates than those who receive a score penalty. Emotional salience is not a feature — it is the mechanism of durable learning.
Finally, we learned that AI mentorship is most powerful when it refuses to give answers. AXON's Socratic mode — asking, never telling — produces trainees who understand why a technique works, not just how to execute it. That distinction, in surgery, is the difference between a surgeon who can handle a complication they've seen and one who can handle one they haven't.
What's Next: NeuroSketch XR+
| Phase | Timeline | Feature | Technical Detail |
|---|---|---|---|
| Phase 1 | Q2 2025 | Haptic Fusion | Bidirectional haptic rendering via Meta Quest controller haptics + MX Ink — tactile vibration signatures and directional resistance for each of 14 tissue types during incision |
| Phase 2 | Q3 2025 | Live Patient Data | AWS HealthLake + FHIR R4 protocol integration — anonymized real patient DICOM scans pushed to the platform hours before scheduled surgery. Trainees rehearse the exact case they're about to assist on. |
| Phase 3 | Q4 2025 | Gaze-Driven OR | Meta Quest Pro eye-tracking integration — surgeons navigate anatomy and zoom through gaze alone, spatialized bio-audio (heartbeat, blood flow, neural oscillations) for full multi-sensory immersion |
| Phase 4 | 2026 | Autonomous AI Resident | AXON evolves from mentor to virtual scrub technician — anticipating instrument needs, flagging intraoperative risk in real time, providing differential diagnosis support during procedure execution |
Impact at Scale
The addressable impact of NeuroSketch XR 2.0 extends beyond individual training improvement:
Cost Reduction: Replacing cadaver lab hours with XR simulation could recover $500k–$1.5M per institution annually in direct lab costs, while providing unlimited procedural repetition.
Access Equity: A Quest 3 headset + MX Ink stylus + NeuroSketch XR 2.0 costs a fraction of a single cadaver lab session. Any medical school, residency program, or individual surgeon anywhere on Earth can access world-class simulation.
Outcome Improvement: Simulation-trained surgeons demonstrate measurably lower intraoperative complication rates. At the scale of neurosurgical training globally, even a 1% complication rate reduction translates to thousands of lives annually.
Research Platform: The Surgical Passport data layer — aggregate performance metrics across thousands of surgeons — represents a novel dataset for neurosurgical education research, with potential to identify early indicators of surgical aptitude and targeted remediation pathways.
Built With
Unity 2023 LTS Universal Render Pipeline (URP) Meta Quest 3 Meta Presence Platform Meta OpenXR SDK Logitech MX Ink SDK Python 3.11 FastAPI WebSockets PyTorch AWS SageMaker AWS Lambda AWS S3 AWS CloudFront AWS HealthLake Photon Fusion 3D Slicer CLI DICOM FHIR R4 ShaderGraph Quadric Mesh Decimation CRDT (Conflict-Free Replicated Data Types) Foveated Rendering
Try It
Hardware Required: Meta Quest 3, Logitech MX Ink stylus
Demo Mode: Available without MRI upload — pre-loaded anonymized case library included
Multiplayer: Join code NEURO-DEMO connects to our always-on global demonstration OR
The Next Great Neurosurgeon
Should Not Need a Cadaver, a Controller,
or a Coincidence of Geography.
She Needs NeuroSketch XR 2.0.
— And now she has it.
Built With
- 3d
- amazon-web-services
- facehugging
- fhir
- graph
- hugging
- java
- jdbc
- logitech
- maven
- meta
- openxr
- photon
- python
- pytorch
- rest
- s3
- slicer
- spring
- springboot
- textmeshpro
- vfx
- websockets
- xr
Log in or sign up for Devpost to join the conversation.