faceband

Play the band with your friends using facial instruments

Comment

Inspiration

Recently more people are using Video chat apps such as Snapchat and Snow. However most functions such as filter and games were used more personally and lacked the concept of communicating.

The fun of Video chat is that you can show your emotion at the moment and music is a powerful content to build a win-win connection between artists' creativity and general users' everyday life as if 'Instagram Music Sticker' has done since May 2018. (https://techcrunch.com/2018/06/28/instagram-stories-music/)

Facebook already has dominant active users in social media market. Especially, WhatsApp and Messenger are the first and second globally popular mobile messenger apps, so a multi-play experience can intrigue more engagement from users and communities. Our team thought if people could play with their faces and make something in harmony as a band, it would be much more fun.

What it does

faceband is basically a rhythm action game using facial movement effects from facebook or instagram. You can use your face as an interaction to play the notes like opening your mouth or winking. Each player can choose an instrument and build a sort of band and they can play a single song making harmony.

How we built it

Using Spark AR Studio, we tracked facial movements through the facetracker, and connected them into interaction.

For example, a simple wink is a continuous action of, one eye closing and opening/ while the other eye is opened.

Then we added a music script, so that when these facial actions are performed in the precise spot, the score would go up. We designed the Graphic elements and layout with sketch, and modeled the instrumental assets with cinema4d. In order to import 3D objects, we used AR Library in Spark AR Studio after uploading them to sketchfab.

Challenges we ran into

There were many obstacles in building the faceband but one of the interesting challenge was deciding which actions we could use, since the actions were limited to the head. When we tested 'shaking your head at the beat' we all got dizzy in a few shakes and thought it would be more like torturing, so we changed the action to slightly tilting the head.

We wanted to make a multi-play AR experience, however currently Spark AR Studio doesn't have any feature and resource for an (independent camera user's) multi-play effect. If coming updates can provide creators with such assets and resources, faceband team will definitely use them for the next milestone.

Accomplishments that we're proud of

By extending the effects of the camera app, we proposed the possibility of a game implementation, and furthermore, adds communication elements that can be enjoyed together, such as competing by sharing each other's images or combining them into a single content.

What we learned

We've used Spark AR Studio, MuseScore, MusicXML, and many other tools for the first time and are now familiar with them.

What's next for faceband

We do not yet use a facemesh, but it might be useful and effective for users who do not want to expose their genuine face or for faceband game to give penalties for missing beats by distorting players' face. Thus, we can update it to the next version.

Additionally, if a group of players were able to select music in a library being updated by the music industry, faceband can be another entertainment platform service with messengers integrated with augmented reality.

Since we didn't have much time, we only made the process till announcing the winner. But if we could develop more, we think it would be fun if the winner could apply funny filters to the losers while they face chat and make some laughter.

Built With

+ 31 more
Share this project:

Updates