It’s sad that we couldn’t get out to arcades to play our favorite Taiko Master since coronavirus social-distancing and city lockdowns kicked in. We came across Beat Saber mixed reality on YouTube, but figured that we couldn’t afford such expensive VR headsets…That’s how we decided to make our own beats/rhythm game. And thanks to the Audio Analyzer released in v89, creating a music-themed effect has been a really cool journey.
What it does
An innovative form of Dance Dance Revolution, Emoji DDR drops stickers according to the beats of a song. If the user successfully make the corresponding facial expressions and/or head rotations as the emojis hit the score bar, they get points.
How we built it
We used Spark AR Studio v89 to build Emoji DDR Game. Most of the game logics and animations were created in the patch editor. We used almost all of the built-in patches, including the newly released Audio Analyzer.
Emojis: We imported beat parameters (in time scale) into patch to control the fallings of emojis of different songs. Here we used Loop Animation to generate random numbers and trigger animation of a random pattern of emoji falling effects. Tracks: we used Audio Analyzer to change the texture of the background tracks in response to the power of some frequency bands of the music. User: we placed the user in the game by using camera texture and person segmentation.
We used the Face Tracker to track user faces and detect interactions like leaning head, opening mouth and blinking. Once an interaction is detected, it sends out a trigger to compare the emoji positions and the hit bar position. If they match (with some tolerance), we will add one point to the total score. Meanwhile, the bar flashes and a scoring sound is played.
To provide a complete game flow, we designed a two-page onboarding process, the replay button during the game, and the restart option on the game-over page. We applied Offset to Runtime to manage the timeline of the game. Visibilities of various game elements were in made possible with the Counter patch.
Challenges we ran into
Our two-person team was based on opposite sides of the world – Aileen was in Beijing and Xiaozhi was in St. Louis with a 13-hour time difference. Sometimes, we were awake at 3AM to make it work for the other person’s schedule. It’s all worth it now that our game is complete!
One technical challenge was we only realized that each effect can’t exceed 4 MB in size after we finished making the game. Thus, we had to cut the number of songs to fall within what is allowed by size.
Accomplishments that we're proud of
We are most proud of the fact that we could implement our idea into a real, working effect, given we had 0 knowledge of Spark AR before entering the hackathon. And ~200 views on our demo video within 1 day wasn’t bad, either!
What we learned
Kudos to Facebook for making Spark AR such a lovely low-code developer tool that we enjoyed our developing process 100%. In addition to working out on the technicalities of Spark AR, we learned how to decide on and refine our project goal as well as details according to what’s accessible in the developer tool so that bringing our ideation to reality wasn’t built on a whim.
It was also amazing to talk with people in the game and music production profession to pick their brains on beats design.
On a side note, we also learned how to use Git and Github (first time ever for Aileen!) thanks to this project.
What's next for Emoji DDR — Hit ’em all
- Improve game playability and fun
- beats extraction and design
- combo effects
- song selection
- Figure out the song copyright issue and then publish the effect on Facebook and Instagram
- Work on compression methods to pack as many songs as possible into our effect
- Promote the lovely, low-code Spark AR developer tool
We’re also thinking of making a series of Emoji effects/games that wouldn’t incur the size problem. A few ideas include:
- On-screen Emojis for video chats: Detect user’s facial expression and generate emojis accordingly. We envision this effect to be integrated into video chats.
- Mimic Emoji Game: The user tries to mimic as many Emojis as they float across screen.