PianoVision AI-based Magic Keyboard Calibration

Inspiration

At PianoVision, everything is about removing friction to playing music you love. Learning an instrument requires reaching a flow state quickly, but until Passthrough Camera access arrived in the Meta Spatial SDK, users had to pick their keyboard type/count/width and manually calibrate it in 3D--often imperfectly. We wanted to remove that process entirely, with no markers and essentially zero setup.

What it does

When a user is in a new environment, Magic Calibration scans the room via the Passthrough cameras, detects a piano/keyboard, estimates its 3D position/rotation/scale, and locks the PianoVision system to it.

Once calibrated, we save that setup using persistent Spatial Anchors, so the user only has to do this once per room.

How we built it

We combined AI/ML with traditional computer vision to estimate:

The number of keys on the keyboard
Key spacing
The position of the keyboard in 3D relative to the headset
The rotation of the keyboard

All with a moving camera and accounting for partial views. We then integrated this directly into the PianoVision Quest application as a full replacement for manual calibration.

Challenges we ran into

Partial views. Users rarely frame the whole keyboard, so we fuse partial observations across frames instead of requiring every key in view.

2D -> 3D pipeline. After detecting landmarks in 2D, we need their global 3D positions. Designing a robust 2D -> 3D pipeline was one of the hardest problems.

Keyboard key count. Keyboards range from 25 to 88 keys. To be useful, our system has to infer the exact key count, not just “a piano here.”

Lighting. Off-the-shelf detectors struggled in dim or reflective rooms, so we tuned our pipeline for low-light robustness. We wrote a custom CV algorithm that outperforms a piano YOLO detector in both speed and lighting tests.

Speed. If Magic Calibration was slower than manual calibration, it wouldn’t be useful. We optimized every part of the system for low latency, bringing each partial estimate down to under 50 ms.

Accomplishments that we're proud of

We built a novel, markerless calibration system for a problem that, to our knowledge, hasn’t been solved before. We’re proud that we could go from exploratory R&D to an efficient engineering solution, and ship it inside a mature, widely-used product.

This is our initial beta release of Magic Calibration, and we’ve implemented solutions to every challenge listed above. The system is heavily optimized to work in low light, at low latency, and with any common keyboard configuration.