GhostHands

Inspiration

A flat tutorial can show what to do without making the movement clear. We wanted to bring a patient, repeatable demonstration into the learner’s own workspace.

What it does

GhostHands is a native Apple Vision Pro learning workspace. A paper-boat lesson combines translucent animated hands, staged paper geometry, written instructions, and a YouTube tutorial that seeks to the selected step. Users can record real hand joints and replay a demonstration relative to their workspace, slow it down, or repeat it. Voice commands include next, slow down, and show that again. Narration coordinates with video audio.

The same workspace includes interactive educational models of car engines, jet turbines, gearboxes, brakes, pumps, cams, belts, and crank mechanisms, plus a cabin-air-filter repair rehearsal. Users can author lessons with video timestamps, spatial arrows, and hand recordings.

Community impact

Our goal is to make practical skills easier to learn and share through repeatable spatial demonstrations. Community workshops, classrooms, and repair education could use this approach to help people understand mechanisms, build confidence, and maintain belongings instead of replacing them. The current prototype requires Vision Pro; wider hardware access and expert-authored lessons are future work.

How we built it

Swift, SwiftUI, RealityKit, and ARKit provide the native spatial interface, hand tracking, joint replay, and procedural 3D models. WebKit and the YouTube IFrame Player API provide video playback and step seeking. Speech and AVFoundation support voice commands and narration. Recordings and custom lessons are stored locally. We used AI coding assistance during development.

Challenges and what we learned

We iterated on stable table placement, consistent panel layouts, smooth animation, hand readability, and preventing narration from talking over video. Synchronizing real-world placement, recorded joint motion, and video timing required careful state management.

Accomplishments

A working headset prototype with spatial lessons, hand recording/replay, voice interaction, custom lesson authoring, and multiple animated mechanical labs. The source includes automated core-logic and video checks, while physical-device testing remains important.

Current limits and next steps

Built-in hand motions and mechanical assemblies are authored teaching illustrations, not expert recordings or engineering simulations. YouTube does not automatically generate 3D motion. Apple Watch haptics is experimental: companion pairing work is in progress and automatic button feedback is currently disabled. Next we want expert-recorded repair lessons, stronger alignment, and broader access.