PAIR Guide

• 

• 

Inspiration

Surgical planning and intraoperative guidance require accurate spatial correspondence between imaging data and physical anatomy. Image registration is foundational to this process. However, interacting with registered data in Mixed Reality using controllers limits precision and fluidity, especially when fine adjustments or markings are required.

We extended Pair Guide with the Logitech MX Ink stylus to support precise image registration refinement and dynamic drawing directly on anatomical models and CT derived renders. Our objective was to improve accuracy and give clinicians a more natural way to interact with volumetric medical data.

What We Built

We integrated stylus driven image registration adjustment and dynamic spatial drawing into Pair Guide’s workflow.

The MX Ink stylus is now used to:

Refine alignment between CT scan models and physical reference points Adjust registration anchors with fine motor control Trace anatomical landmarks directly on 3D bone renders Draw dynamic annotations on CT derived meshes Mark surgical pathways and reference boundaries in real time

Image registration adjustments can be made incrementally, allowing precise repositioning and rotation of anatomical models. Dynamic drawing enables clinicians to sketch directly onto bone surfaces or in 3D space, with strokes anchored to the registered model.

Pair Guide continues to validate workflow state and ensure that adjustments occur within defined surgical parameters, while the stylus provides the precision required for clinical accuracy.

How We Built It

We mapped MX Ink pose data into the anchored coordinate system used for CT model registration. Stylus input drives controlled transformations of registered meshes, including translation and rotation with constrained axes when required.

Dynamic drawing is implemented as surface attached stroke rendering, allowing annotations to adhere to bone geometry. Stroke data updates in real time and remains spatially stable relative to the registered model.

We optimized performance to support high resolution CT meshes while maintaining smooth stylus responsiveness and low latency interaction.

Challenges We Faced

The primary challenge was maintaining stable registration while allowing fine adjustments. Small errors in transformation handling can produce visible drift between physical reference points and CT geometry.

Rendering dynamic strokes on dense medical meshes without frame drops required careful mesh optimization and incremental stroke generation.

Ensuring that stylus precision translated accurately to registered anatomical space was critical for maintaining clinical relevance.

Accomplishments that we're proud of

We successfully integrated the Logitech MX Ink stylus into PAIR Guide’s surgical workflow without disrupting the core guidance architecture. Precision input is now fully embedded across panel interactions, implant positioning, alignment point manipulation, bone render adjustments, CT model interaction, image registration refinement, and dynamic drawing.

We achieved stable, low latency stylus driven manipulation of dense CT derived anatomical meshes while preserving spatial alignment and workflow validation logic. Fine motor adjustments that were previously difficult with controller input can now be performed with significantly greater accuracy and control.

We also redesigned key interface elements to support stylus first interaction, improving targeting precision and reducing interaction error during critical surgical planning tasks.

What we learned

We learned that precision hardware alone is not enough. To truly benefit from stylus input, validation tolerances, transformation logic, and spatial anchoring systems must be recalibrated for fine motor interaction.

We also learned that image registration and model alignment require careful handling of spatial transforms to prevent cumulative drift. Even small inconsistencies in coordinate mapping can degrade trust in the system.

Dynamic drawing on high resolution anatomical meshes introduced performance challenges that required optimized mesh handling and efficient stroke rendering pipelines.

Most importantly, we learned that clinicians naturally adapt to pen based interaction far more quickly than controller based manipulation when performing alignment and planning tasks.

What's next for PAIR Guide

Next, we will expand stylus driven registration refinement tools with guided constraints and measurement assisted verification to further improve implant placement accuracy.

We plan to introduce layered annotation systems, exportable planning references, and enhanced dynamic drawing modes for marking surgical pathways and anatomical boundaries.

We will also continue optimizing performance for large volumetric datasets and explore predictive alignment assistance to streamline surgical planning workflows.

By continuing to evolve stylus integration within PAIR Guide, we aim to establish precision driven Mixed Reality as a standard for surgical planning and image guided interaction.

Built With

• logitech
• unity