Inspiration
Learning piano means connecting the notes on the page, the keys under your hands, and the movements needed to play them. Traditional tutorials spread that information across a score, a screen, and the instrument.
We wanted to bring instruction directly to the piano.
Our AR piano teacher uses XREAL Aura’s optical see-through glasses to place guidance over a real keyboard. Students see their actual hands and keys alongside spatial highlights, floating sheet music, and a conversational coach—all without turning away from the instrument.
What it does
Learn with guidance built around your performance
A Yamaha P71 connects directly to the Aura puck over USB MIDI, so every key press reaches the app immediately and drives the feedback on the keys.
Guided lessons teach measure by measure:
- Practice each hand: learn the right hand, left hand, then both together.
- Build speed progressively: advance through 50%, 75%, and full tempo.
- See and hear the passage: follow demonstrations, illuminated keys, floating notation, and a synchronized count-in.
- Repair difficult sections: revisit weak measures and hands before returning to the larger passage.
- Use the right finger: fingering badges sit on the keys, and the fingertip that should play the note glows as it gets close.
The lesson tracks accuracy, timing, missed notes, and hesitation. It groups difficult measures into focused practice blocks and includes lead-in measures when a transition needs attention.
Explore in free play
Free play gives students room to experiment. The application identifies chords, estimates musical key, evaluates rhythmic steadiness, and recognizes supported song openings. Floating sheet music follows the player’s progress.
Saying “hey teacher” lets students ask a question, request sheet music, or use supported voice controls.
Keep the coach within reach
An animated coach orb communicates listening, thinking, and speaking, with text appearing alongside spoken feedback. A menu on the back of the left hand follows the hand and holds the session controls: lessons, free play, recalibration, fine-tuning, and the fingering cues.
How we built it
Spatial computing on a real instrument
We built the application in Unity, using OpenXR, Android XR, XR Hands, and the XR Interaction Toolkit. The deployed application runs on the Aura puck, with network APIs supporting conversational coaching and speech.
A custom Android Java MIDI bridge connects the piano to Unity. A Windows backend supports development through the same event interface. These events power keyboard visualization, performance evaluation, lesson progression, and free-play analysis.
Responsive musical interaction
We separated immediate musical interaction from AI feedback.
Local code matches notes, evaluates performance, and determines the next practice task. The coach receives structured information about the score, the student’s performance, and the upcoming lesson step, then provides concise guidance between runs. Routine transitions use short cues and musical chimes.
The coaching system integrates Claude and Groq, speech recognition, speech synthesis, and native Android text-to-speech fallback.
Calibration at piano-key scale
Two printed ArUco markers behind the lowest and highest keys give the app the keyboard's position and direction. Getting there took several rounds of testing on the glasses. Our first approach had players pinch the ends of the keyboard by hand, which worked but was slow and easy to get slightly wrong. The marker version needed stable-pose sampling, a check against the real keyboard width, and offsets measured for our piano before it held steady. Now the overlay locks on at true scale in a few seconds.
Four-point pinch calibration is still there as an automatic fallback when markers are not available, and fine-tuning moves the keyboard in millimeter steps and saves the result on the device.
A reusable music pipeline
Our library contains 37 song entries, spanning beginner melodies and classical repertoire excerpts.
Custom tools:
- Parse and validate structured musical notation.
- Import MIDI with timing, hand, key-signature, and pickup information.
- Export MusicXML and generate sheet images through Verovio.
- Preserve measure and note coordinates for visual highlighting.
- Record source and licensing information for imported material.
Challenges we ran into
Aligning virtual keys with physical keys.
Small tracking errors become obvious when a highlight has to sit on one piano key. Hand pinches got the overlay close, but not close enough to trust, so we moved to printed corner markers and refined them on the device with pose sampling, geometric checks, saved offsets, and fine-tuning.
Designing for optical see-through glasses.
The display adds light, so contrast behaves differently from a conventional screen. We used bright outlines and carefully placed controls. Depth-only hand occlusion keeps virtual highlights from drawing over the player’s real fingers.
Maintaining musical timing.
Frame-based playback introduced uneven metronome clicks and demonstrations as rendering performance changed. Scheduling audio against Unity’s audio clock gave the lesson consistent timing.
Making the coach helpful without becoming distracting.
We developed explicit policies for when the coach should speak, which performance facts it should reference, and how much it should say. Silent attempts, missed notes, repeated mistakes, and successful passages receive distinct handling.
Accomplishments that we're proud of
We built a complete interaction loop:
Play a real key → receive spatial feedback → evaluate the passage → adapt the practice → hear targeted coaching.
The experience brings together native instrument integration, AR calibration, hand interaction, music engraving, structured lesson progression, and conversational AI.
We’re especially proud that these systems share musical context: the physical key, the displayed measure, the evaluated performance, and the coach’s feedback all relate to the same practice task.
Our team
Arindam (Ari) — Spatial Computing & AR Systems Lead
Arindam built the spatial foundation of the experience: an 88-key overlay aligned to a physical piano, and the calibration that puts it there. He took calibration from hand pinches to printed corner markers through repeated testing on the glasses, adding saved offsets, millimeter fine-tuning, and an automatic fallback.
He also built the fingering guide, rebuilt the radial menu into controls that track the left hand, and handled the optical see-through rendering, hand occlusion, and Android XR setup that made the app usable on the Aura. He connected calibration, sheet placement, permissions, and completed-run coaching into the full experience.
Jordan Muniz — Instrument Integration & Learning Systems Lead
Jordan engineered the connection between the instrument and the learning system. He built native Android and Windows MIDI integration, real-time performance evaluation, the sheet display, and the initial application flow.
He expanded the curriculum into right-hand, left-hand, and both-hands practice, with tempo progression, full-passage assessment, and targeted repair of weak measures. His audio scheduling, count-in system, and demonstration playback gave the lesson its musical timing and structure.
Canady — AI Coaching & Musical Intelligence Lead
Canady created the application’s conversational teaching layer, integrating language models, speech recognition, speech synthesis, and fallback behavior. He established the initial curriculum architecture and developed “hey teacher” interactions for contextual questions and voice controls.
He also built free-play analysis for chords, musical key, rhythmic steadiness, and song recognition, alongside score following and mode switching. His coaching policies shaped the assistant into a concise, context-aware practice companion.
Nehemiah Oginni — Music Infrastructure & Interactive Experience Lead
Nehemiah built the infrastructure connecting musical content, visual presentation, and player interaction. He developed the 37-entry song library, notation validation, MIDI import tools, and automated sheet-generation pipeline with measure and note metadata.
He created the coach’s visual identity through an animated orb, speech-responsive effects, and synchronized text. His integration work connected database music to the lesson and coach, while his hand-accessible radial menu brought essential session controls into the AR experience.
What we learned
An AR music teacher depends on alignment, timing, readability, and feedback working together for a rich experience.
We learned a lot regarding creating an experience from a learners perspective and understanding user needs, while trying to make a clean, unobstructive, but extremely informative user experience.
We learned to ground musical decisions in explicit data, use AI where contextual language adds value, and test spatial interfaces on the hardware throughout development.
What's next
- More guided repertoire: connect the broader song library to lesson selection.
- Richer musical instruction: expand fingering and notation coverage.
- Broader instrument support: refine calibration for additional keyboards.
- Greater student independence: introduce adjustable guidance levels that gradually reduce visual prompts.
