Floor Piano

Play with feet
SnapCode
Floor Piano
Step front part of natural (non-sharp) keys to play

Inspiration

The augmented reality Floor Piano was inspired by the 1988 movie which made floor pianos famous.

What it does

Places a piano keyboard on the detected ground.

Keyboard can be repositioned by tap and other touch gestures.

Has 23 keys from F3 to E5.

Keys are played by feet, i.e., when a foot is detected touching a key.

A natural key (e.g., A, B, C) can be played when a foot is on the front part of that key. By design a natural key cannot be played when a foot is placed on that key between its neighboring two sharp keys.

How we built it

Uses ML Foot Tracker component to detect feet.

Attaches physics body to each foot and each key. Uses physics collision events to detect when a foot collides with a key and as a response plays the corresponding note.

Uses camera world tracking and world mesh to detect ground to place the keyboard.

Uses glass-like material for keys to give the illusion of feet not being occluded by keys.

Challenges we ran into

Making sharp keys to be played independent of neighboring keys: I borrowed the keyboard from the physics template, where this was not possible, since colliding with a sharp key also collided with the two natural/white keys neighboring that sharp key. I solved this problem by adding physical body to the tip of each natural key.

Making feet not to be occluded by keys: 1- Since this is the first time I am working with Lens Studio and Snap AR, I didn't have time to explore whether Lens Studio has stable support for monitoring depth to solve this problem. 2- Alternatively I implemented a version with 3d shoe model and occluder to make 3d model shoes appear ON the keys, but I preferred the look of the solution I submitted.

Therefore, I made keys out of glass like material so that the user can see her feet as part of the key material.

Making a key to be played once when it is collided with a foot and making it replayed when a foot is moved up and down again. For next versions of the Floor Piano, I will be working for a more consistent behavior in these cases.

Making the movement of a key when it is played similar to a physical keyboard. For next versions of the Floor Piano, I will be working for more realistic movements of keys.

Accomplishments that we're proud of

Since I have never worked with Lens Studio until this project, I am happy to have a functional floor piano.

What we learned

This is my first Lens Studio project, so I needed to figure out everything and learn Lens Studio and all its features used in this project including foot tracking, physics collisions, etc.

What's next for Floor Piano

When a foot collides with a key and while the foot is still on the key, a better control for when that key can be replayed again and when it shouldn’t be.

Nicer sound recordings of notes F3 to E5.

Making foot really appear over the keys, i.e., not occluded by keys, using Lens Studio depth occlusion support.

Making key movements more realistic.