Simtryx: Advanced Interactions Update (Neurological Exam)

Introducing our new flagship case, Neurological Exam
Real physics on every interaction
Hand tracking you can feel, thanks to our virtual patient rigging
Use AI commands to engage with the patient in a realistic way
Multitude of AI & Hand Tracking Interactions

Inspiration

We created Simtryx to make high quality healthcare training accessible to everyone by combining the latest advances in mixed reality, AI, and hand tracking into a single platform. Our goal has always been to give learners realistic clinical experiences they can practice anytime, without relying on expensive manikins or complex simulation labs.

The neurological exam became the perfect vehicle to push our interaction system further. It is one of the most hands on and communication heavy procedures in medicine, requiring precise tool use, repeated patient contact, and a constant stream of verbal commands. To capture that level of realism, we needed to elevate our physics system, refine our hand tracking logic, and build AI driven patient responses capable of supporting the full flow of the exam. This challenge inspired the Advanced Interactions Update.

What it does

This update introduces over thirty new hand tracking interactions designed specifically for neurological exam skills. Learners can perform the full exam using their hands and voice, observe real time patient reactions, and work inside a mixed reality bedside environment anchored to their physical space. All of these features come together in our new flagship scenario: the Neurological Exam.

How we built it

We rebuilt our interaction framework using a proprietary physics layer optimized for hand tracking on Meta Quest. This included new grab logic, contact rules, and spatial validation for tools like penlights, tuning forks, and reflex hammers. We expanded our AI dialogue system with neurological questioning, symptom clarification, and patient behavior cues. The experience runs in mixed reality, using spatial anchoring to keep the patient stable as the learner moves around the bed.

Challenges we ran into

Building a neurological exam with realistic responses surfaced several challenges across AI, physics, and hand tracking.

AI command fine tuning: During early testing, the AI driven patient didn’t always follow the intended clinical logic. We added conditional triggers and internal state checks to ensure that each verbal command and examination step produced the correct patient response in the right sequence.

Hand tracking precision: Neurological exams require close contact with the patient’s face, hands, and limbs. Small errors in tracking thresholds produced unintended activations or missed interactions. We had to fine tune collision boundaries across dozens of body regions so that each touch behaved predictably and felt physically grounded.

Physics and body rigging: We rebuilt portions of the patient’s rig so the body could react believably to tool contact, hand placement, and reflex tests. This required tuning the physics layer to avoid jitter or overcorrection while still producing clear visual feedback for learners.

These challenges pushed our interaction system forward and helped us create a more realistic and reliable neurological exam experience.

Accomplishments that we're proud of

We created the first mixed reality neurological exam simulation of its kind for one of the most widely used physical assessments in medicine. The exam can now be performed entirely through natural hand tracking and conversational AI, without controllers or scripted sequences.

More than thirty exam interactions have their own physics behaviors, body reactions, and logic states, allowing the full exam flow to feel closer to a real bedside encounter. We also introduced a one minute onboarding tutorial that removes user confusion and prepares learners for the interaction model before entering the scenario. This significantly improves usability and showcases the strength of our updated interaction system.

What we learned

We learned that realism in clinical simulation depends on the balance between hand tracking fidelity, physics behavior, and AI logic. Subtle adjustments to collision boundaries, timing windows, or patient body reactions can dramatically influence how believable the experience feels.

We also learned that AI behavior needs to be tightly aligned with clinical logic. Ensuring that the patient responds consistently, predictably, and in the correct sequence was essential for creating a believable neurological exam. This insight will guide how we expand AI reasoning and patient state management across future Simtryx scenarios.

What's next for Simtryx: Advanced Interactions Update (Neurological Exam)

These new interactions now serve as the core framework for the next generation of Simtryx scenarios. We are extending this system to advanced nursing procedures, focused neurological condition assessments, and emergency care cases.

We are also developing analytics tools for instructors and planning scenario variants with expanded patient states and condition specific behaviors. This includes deeper physiological modeling, variable exam findings, and branching patient presentations that respond dynamically to the learner’s actions.

This update is the beginning of a broader expansion of Simtryx’s capabilities, using hand tracking, physics, and AI to deliver realistic mixed reality training at scale.