Virtual Club

Game Screens
Game Rules Display Screen

Inspiration

This lens is Inspired by the digital avatar and virtual meetings era. Emergence of platforms like ChatVR, Cosmos.Video and Hardware like Sony Mocopi which will allow more human interaction in applications will make users want for more experiences that have new and interesting ways of interaction which are fun.

What it does

This lens uses snap sync framework and connected lens capabilities, featuring virtual avatar system by having the user and their friend present on screen together with their head motion and facial expressions synchronised. Just like a video call but with virtual avatar!

There are 3 different games to play -

Croc Game - A Crocodile Head emerges, each player gets a chance to press it's tooth, first player to press the faulty tooth which closes the crocodile mouth loses.
Tap Sumo - Each player is a sumo, the player needs to tap on screen which pushes the other sumo out, first player to push other sumo wins.
Quick Gold - 20 Tiles are placed on screen, each player gets a chance to flip the tile by tapping on it, every tile may or may not have gold, by the end the player with maximum gold wins.

All of this with you and your friend chatting and interacting with your virtual avatars which adds a more human touch and personalisation to the experience!

How we built it

The first part of the lens is sharing facial expression and head data. I managed this by having a Sync Entity which constantly shares facial and head data over events which is fired and received by both the users. This data is not stored but constantly streamed via events to simulate Avatar Movements.

The second part is managing activities flow. This is also managed by having a shared and syncrhonized data. The user is presented with options of activities and either of the user can select the activity. This launches the respective activity where each activity has it's own scripts which manages touch interactions and activity state flow. The activities manager has the options to reset the activities and present the menu again.

The third part of the lens is each game controlling it's activities and managing shared state. Each game has script which manages touch inputs and reports to the main game controller to take actions. The animations and actions are done with events to have proper sync with respect to events.

Challenges we ran into

The first challenge was to pass reliable mouth opening and eye opening data for proper avatar simulations. My first approach was to calculate distances in screen space using face landmarks and return events like mouth open/closed, eyes open/closed but this wasn't enough. Then I found calculating Eye Aspect Ratio (EAR) and Mouth Aspect Ratio (MAR) can be very useful which will help me with values depicting mouth and eye openness with no issues of variation with depth of face.

My second challenge was to manage turn system which simulates player interaction turn. Initially I had problems by having states which are owned but managing states for both was getting complicated, I overcame this by simply having separate script which queries both players connectionIds and name and has a single sync state which alternates and also the animation of name box pulsating to indicate their turn.

My Overall challenge was to have multiple shared states and to manage them. This was helped by sync framework and proper code splitting.

Accomplishments that we're proud of

The experience of having your friend as a virtual avatar in a lens that is so easily accessible and combined with snap's feature to chat in connected lens made it such an interesting activity and with all the fun and short engaging games to keep the conversation going. So proud of how it turned out!

What we learned

I learned Snap's Sync framework to do connected lenses which I so much wanted to do. Complex State management with proper code splitting with multiple script components interacting with each other. Also use of Blender and Inkscape for my asset development.

What's next for Virtual Club

The virtual club right now supports 2 users at a time. I will add more user support and multiple avatar options for the next version. I also have a prototype of using face meshes as the face, simulating more accurate face expressions. More activities like painting, karaoke, criss-cross on it's way.