Street Piano

Inspiration

Foot Tracking inspired me to create a street piano to turn people’s footsteps into music and generate soundscapes of streets.

What it does

It is a giant piano to be played by feet:

• A key is played when a foot is detected on the key.
  
• To add rhythm, user can tap percussion or drums button.
  
• User can tap to reposition the piano, and to rotate it with double fingers.

How we built it

I used Foot Tracking with full body person segmentation since I wanted people appear on the piano not partially (not their feet only) but fully.

I used physics bodies and colliders for piano and feet, so that keys to be played by feet.

To control movements of the keys and isolate key collisions, I used mesh type colliders and physics filters.

I used notes generated by a keyboard/synthesizer to create a vibraphone like timber so that it sounds pleasant when vibration of notes rolls onto each other.

Challenges we ran into

Getting the collider shapes so that feet collide with the intended key without colliding with neighboring keys.

Right now, segmentation with foot tracking is not as clean as I wish. For example, when a person walks on the piano with a luggage or dogs, they interfere with the look.

Street Piano needs a large ground surface to be placed and surface detection sometimes do not work well. I tried both native AR option checked and unchecked for surface detection and decided on not enabling it for stability, especially when user taps for repositioning.

Accomplishments that we're proud of

A functioning giant floor piano which both sounds and looks esthetically pleasant. The people can be fully seen on the keys of the piano (not just their feet).

What we learned

Foot tracking and physics.

What's next for Street Piano

I have different use cases in mind for further development:

1- Use it for generating soundscapes of streets.

2- For fashion brands to use it as a runway for their models to showcase clothing and shoes.

3- For children or adults to use it as a floor piano: Even on the famous real world so called giant piano, it is difficult for players to target an intended key, and to isolate a key without stepping on the neighboring key. So can the key isolation on my piano be improved to come up with a learnable playing style, which I intend to work on.

Built With

• blender
• javascript
• lensstudio