Inspiration
Learning guitar can be a nightmare: flipping through pages, working out your fingering, and trying to understand scales that make absolutely no sense, all at the same time. FretFire XR finally gives you a totally hands-free, AI-assisted teacher: the guitar teacher you always wanted. It's there for you at 2 in the morning or 2 in the afternoon. It keeps pace with you, works at your own pace, and still challenges you. All hands-free, voice-assisted and AI-guided. That's FretFire XR.
What it does
- Put on Quest 3 or 3S, sit with your guitar and look at it. FretFire XR finds the neck in passthrough, locks onto the strings, frets and headstock, and keeps following the guitar as you move.
- Three calibration notes: a ghost hand shows each note on your neck, you play it, and the app hears it through the microphone and confirms the fit.
- Press Play and a song lesson starts: the real recording plays, the tablature scrolls in a window beside you, and a ghost hand plays each note on your actual fretboard so you can see exactly where your fingers go.
- Slow it to 25, 50 or 75 percent, restart from the top, switch between the recording and plain tones, and use the built-in tuner.
- Light it up: your own hands catch fire as you play.
- Everything is controlled with hands: pinch and poke the windows, no controller needed.
How we built it
Native Android on the Meta Spatial SDK (Kotlin), with our own panel UI kit for the windows. Fretboard tracking combines the Passthrough Camera API (Camera2 frames matched to head poses), OpenCV line and fret detection, the Depth API environment raycaster and hand-tracking fingertips. The song lesson comes from a Guitar Pro tab, converted into a fingering and hand-posture plan that drives the ghost hand. Pitch detection runs on device for calibration and the tuner. We built a Mac lab that replays recorded headset sessions so tracking changes can be scored without putting the headset on every time.
Challenges we ran into
Registering a moving, shiny guitar neck in real time from headset cameras; matching camera frames to the right head pose; keeping the outline steady while the player's own hand covers the strings; and lining the ghost hand up with a real recording.
Accomplishments that we're proud of
The neck locks in about six seconds, the ghost hand plays the song on the real fretboard, and the first time our hands burst into flames on a real guitar in passthrough, we knew we had something.
What we learned
Real-world tracking lives or dies on small details: camera timing, depth, and what the player's hands are doing. Testing every change against recorded sessions made progress measurable instead of guesswork.
What's next for FretFire XR
More songs and lessons, practice scoring from what the microphone hears, a voice teacher you can talk to with both hands on the guitar, and a version for Meta's upcoming glasses.