Inspiration

Every year thousands of patients undergo reconstruction surgeries for their knees. Most walk into surgery only with the knowledge of simple diagrams/explanations. On the other hand, residents learn these procedures from 2D illustrations and standardized resources. Although Arthrex has been closing this gap through their various applications, we saw an opportunity to take that further. We wanted to build something that lives in the browser, requires zero setup, and supports both regular patients and residents.

What it does

OrthoVision is an interactive anatomical knee into a browser. You can rotate, zoom, and select the model by toggling various structures independently. An multi-step reconstruction syncs up with camera movement and highlighting so you can follow the procedures easily. Switching between different modes helps reach different audiences. Additionally, an AI assistant helps users answer questions and navigates the 3D scene model. A camera hand-tracking system also lets users rotate, zoom, and reset the model with gestures alone.

How we built it

The 3D scene runs on React Three Fiber, which wraps Three.js objects/scenes as react components. The model we used is an open-source GLB from the AnatomyTOOL project, containing hundreds of individual meshes. We built a layer to sanitize raw mesh names to structured IDs that the sidebar, info cards, procedure panel, and AI assistant all share. Zustand is another library that we used to better mange state across our different features. Camera animations use hooks to move between preset positions for each procedure step. The hand tracker is a Python sidecar using MediaPipe and OpenCV, streaming gesture state over a websocket to communicate back to the 3D scene.

Challenges we ran into

Finding the right model was one of the biggest challenges. This was because other individualized models were too simple. Which is why we took a larger lower limb model and reduced to the knee area. Another challenge was that Three.js also sanitizes node names on load, which strips them causing names in code to be mismatched with the names in the original file. To fix this, we created a pipeline and an alias table to resolve every mesh to the right ID.

Accomplishments that we're proud of

A detailed and interactive 3D model running in a browser tab with no install, no login, and no external hardware is something that we are definitely proud of. The patient/surgeon toggles is also notable because it helps relay the same information depending on whether a surgeon or a patient is referencing it. Another aspect that we are proud of is how we designed the our system. What we did is that we made code generic so that it can easily support additional joints/procedures depending at the task at hand. Our camera gestures and AI assistant are tools that we believe can empower users better understand any questions they may have.

What we learned

Treating a 3D scene as just another layer in react allowed us to quickly iterate ideas and features that we wanted to implement. By having design in mind, we were able to effectively work as a team without slowing each other down. Working with anatomical data also showed how the mapping of the different components was the bigger challenge rather than the 3D scene.

What's next for OrthoVision

As we mentioned previously, because we designed some components and functions to be generic, would like to add additional procedures and models such as hips, ankles, shoulders, etc. Another direction that we are interested is looking into post operations. Surgeons can select different structures of the model and perhaps color code them. This can help them explain and provide insights to patients about their next steps to recovery and what to expect during rehabilitation. At its core, every feature we implemented comes back to the same goal that Arthrex has of helping surgeons treat their patients better.

Company Challenges

We made this project in order to build a submission to the Arthrex Challenge, which seeks to build an application to "Help Surgeons Treat Their Patients Better".