PhantomMirror

This image shows our progress during the development process.

Inspiration

Phantom limb pain can continue long after a limb has been lost. One established approach is mirror therapy, which uses the reflection of an intact limb to create the visual impression that the missing limb is present and moving. However, a physical mirror limits the user’s position, viewing angle, and range of movement.

We saw Apple Vision Pro as a natural way to extend this idea. Instead of relying on a physical reflection, we could create an immersive virtual limb that moves with the user’s intact hand. Our goal was not simply to use the latest technology, but to apply spatial computing to a real rehabilitation challenge.

What it does

PhantomMirror is a Vision Pro rehabilitation prototype for people experiencing phantom limb pain. At the beginning of a session, the user selects which limb is missing and calibrates a realistic virtual hand and arm.

The application then mirrors the motion of the intact hand onto the missing side. A high-fidelity virtual limb is mapped to the tracked hand skeleton, allowing the user to see it open, close, rotate, and move in real time.

The training experience includes several modules:

Guided hand and arm movement exercises
Orb-touching and movement-accuracy games
Bilateral matching and coordination activities
Clapping, slicing, and range-of-motion tasks
A spatial brick-building game that makes repeated movements more engaging
Session tracking for movement, accuracy, completion, and pain ratings
AI-generated clinical notes summarising performance, recognising improvements, and suggesting areas for future training

Together, these modules create a complete journey from calibration and training to session review.

How we built it

We developed PhantomMirror for visionOS using Swift, SwiftUI, RealityKit, and ARKit hand tracking.

ARKit provides a tracked hand skeleton, while our realistic USDZ hand-and-arm model uses its own rig and joint hierarchy. We created a mapping system that connects the tracked hand joints to the corresponding joints in the virtual model. A calibration interface allows the model’s position, rotation, and scale to be adjusted before binding.

To create the mirrored limb, we transform the tracked joint poses across a carefully defined mirror plane while preserving depth and orientation. After calibration, the same binding is maintained across the debugging, training, gaming, and brick-building modules.

We also created lightweight procedural bricks directly in RealityKit instead of relying on high-polygon imported models. Their placement system uses grid-based overlap and stud alignment so that bricks stack in a predictable way.

During each session, PhantomMirror records training metrics and produces a structured summary. We integrated Apple’s on-device language model on visionOS to turn these measurements into a short clinical-style report containing a session overview, positive observations, and recommendations for improvement.

Challenges we ran into

Our biggest challenge was making the virtual limb feel believable. A model can look realistic while stationary but immediately feel unnatural when its joints, proportions, or coordinate systems do not match the tracked hand.

The ARKit skeleton and the model rig used different joint counts, hierarchies, local axes, and reference spaces. We had to carefully map the joints and convert their transforms into a consistent coordinate system. Small errors in depth or mirroring could make the two hands appear asymmetric, while directly forcing every joint onto the tracked skeleton could distort the surface of the virtual hand.

We also encountered challenges involving left- and right-hand selection, model occlusion, forearm rotation, lifecycle resets, slow asset loading, and maintaining the same binding when users moved between activities or started a new session.

Because spatial interaction is embodied, many issues could only be understood by repeatedly testing on a real Vision Pro. A difference of only a few centimetres could significantly change how convincing the experience felt.

Accomplishments that we're proud of

We are proud that PhantomMirror became an end-to-end experience rather than a single technical demonstration. Users can configure and calibrate the virtual limb, complete multiple training activities, play a spatial brick-building game, and receive a report at the end of the session.

We are particularly proud of:

Creating a realistic virtual limb with detailed geometry and skin materials
Achieving responsive skeletal mapping between ARKit tracking and a custom rigged model
Maintaining one calibrated binding across the complete rehabilitation flow
Combining rehabilitation exercises with interactive spatial games
Generating meaningful session reports from recorded training data
Building a working prototype within a three-day development period

PhantomMirror received the Best Problem Solution award at Spatial Hack AI 2026. Our team consisted of three NUS undergraduate students and two NTU graduate students, and every member contributed across research, design, development, 3D modelling, testing, and presentation.

What we learned

First, solving a meaningful problem is more important than simply using the newest technology. With only three days, choosing a focused direction and using Vision Pro’s capabilities where they created genuine value was essential.

Second, we learned to trust our teammates. By combining different strengths and allowing each person to take ownership, we achieved much more than any of us could have built independently.

Third, every hackathon is an adventure. We began without knowing exactly which technical problems we would face, what we would learn, or how far we could take the idea. That uncertainty made the experience both challenging and rewarding.

What's next for PhantomMirror

Our next step is to refine PhantomMirror with input from rehabilitation professionals and people with lived experience of limb loss. We want to evaluate whether the interactions are comfortable, understandable, and appropriate for repeated use.

We also plan to improve calibration, support more personalised limb models, expand the range of rehabilitation activities, and make hand and forearm tracking more robust. Future versions could provide longitudinal progress tracking and more detailed reports that help users and clinicians compare multiple sessions.

Ultimately, we hope to explore how spatial computing can make mirror-therapy-inspired rehabilitation more immersive, measurable, accessible, and engaging.