Try it out:


Imagine becoming your favorite movie character by taking their place as the main star. Embodying Darth Vader and “altering the deal”, or stepping into Jack Sparrow’s boots and drunkenly slurring “where’s the rum?!” Innovative entertainment means taking mediums we know and love and taking a new perspective.

Another main inspiration was Ernest Cline’s “Ready Player One”. In the story, when an OASIS avatar begins a FlickSync game, they become a specific character from a movie and have to recite lines of dialogue. When the avatar speaks, they will speak in the voice of the specific character, move around as the character, see, hear, and feel anything the character would see, hear, and feel in the movie.

What it does

Users can take on the roles of their favorite story characters, acting out scenes and speaking the lines of their assigned character. They enter into the Tea Party scene from Alice’s Adventures in Wonderland and become Alice, the Mad Hatter, or the March Hare. They can also play the scenes with their friends, with each person being assigned different characters using their own Oculus Rift, HTC Vive, and Windows MR headsets. During the scenes, when a character needs to speak, a cue card shows the user the lines assigned to them. They have to speak the lines correctly within an assigned period of time. Players are scored by how well they speak their lines of dialogue. The title of winner of the game goes to whoever gets the highest percentage!

How we built it

We used Unity with collaboration through Github. We used SDKs for voice recognition and split the integration of those SDKs between the devs. The artist created 3d models for the scene and the director made a script and recorded our audio. We then built out the game logic, and tested with several mentors for a few rounds of feedback for the polish.

Challenges we ran into

Public Domain Story: While there are a ton of amazing stories we would have loved to pick scenes from, including Star Wars, The Princess Bride, Rocky Horror Picture Show, Monty Python & The Holy Grail, Pulp Fiction, The Room, etc. the scope of this Hackathon required us to pick a story that was in the public domain and that was instantly recognizable. We decided on Alice’s Adventures in Wonderland as this fit the bill for the Creating Reality Hackathon.

Watson Speech to Text API: The Watson Speech to Text API was a little bit of a challenge to work with, but was extremely crucial to the success of this project as it hinges on speech recognition. The main struggle was in determining whether the user had said the correct lines and this was done through C# scripts analyzing the words said (given by Watson) and the current line they were assigned (via JSON). Another issue we faced is sometimes Watson Speech to Text API struggles with certain words and phrases making it a little difficult to properly emulate lines.

Photon & Watson conflicting about microphone ownership We used Photon for the multiplayer aspect of this project and encountered a few difficulties concerning the microphone ownership. Both Photon & Watson would fight for ownership of the microphone and so ultimately our solution was to share it, but initially this was a big issue.

Multiplayer Virtual Reality: Our multiplayer functionality ultimately wasn’t working 100% up to our standards but getting both players synced up and hearing each other did end up working. We struggled greatly with placement of players as each player is assigned a unique role.

Animation: We knew we wouldn’t have time to animate lip synced facial animation, so we used hand drawn facial animations hovering right in front of the character’s faces. This allowed us for a faster workflow and ended up being a quick fix that worked very well.

Original Assets: While it might look good to use pre-made free assets, we decided it would look even better if we utilized the talents of our very own team to create unique stylistic beautiful art assets. We created 3d characters, sounds fx, and music.

Character acting: There was a lot of dialogue, and 4 characters that already needed to be designed, modeled, painted, and rigged, which didn’t leave a lot of time to animate each character speaking their unique lines. We designed a single talking animation for each character due to the time constraint.

Accomplishments that we're proud of

We are proud of putting together a gorgeous and fun scene from Alice in Wonderland that allows the user to act out the part of Alice. We are proud to have gotten the multiplayer experience working, and for the Watson and Photon SDKs to be working together well. We are especially proud of the great teamwork throughout the few days. We made decisions quickly and each team member made unique contributions.

What we learned

We learned how to get multiplayer working (save a few bugs), how to use Watson, and how to get Unity working well with a team, working on it through Github and Source Tree. We tracked our tasks by using Trello having four columns - To Do, Work in Progress, Ready for QA, Done.

What's next for FlickSync

The potential of the tools and game we created is huge. We want to implement the following features: Track blocking and character action depending on the scene (interacting with props, moving around, throwing things) Camera To Watch Performance (if you want to re-watch or record from 3rd person) Multiplayer without bugs. Characters synced up in multiplayer A hard mode for disabling the cue cards Movable and grabable cue card like a tablet More comprehensive report card Smarter Watson Speech-to-Text Checking HoloLens and mobile VR headset support Voice Augmentation depending on character

FlickSync could be a used as a tool for actors looking to practice lines, or an artist could produce a full length play all in VR!

Google sheet with list of Assets:

Below is that sheet copy and pasted should the above link not work (sorry for the formatting)

Assets and SDK Asset Name Asset Type Creator Url 1stteapot.skp 3d object andeciuala n tea+set+1.skp 3d object archidol Castle_Table_270X100.skp 3d object asee pocket watch.skp 3d object AW. Caquetoire.skp 3d object cealpup rustic+english+farm+table.skp 3d object david heim Old+chair.skp 3d object dragon_of_metal Alice_wonderland.blend 3d character model flicksync
hare_wonderland.blend 3d character model flicksync
hatter_wonderland.blend 3d character model flicksync
mouse_wonderland.blend 3d character model flicksync
hatter_wonderland.blend11 3d character model flicksync
mouse_wonderland.blend1 3d character model flicksync
alice.png character sketch flicksync
hare.png character sketch flicksync
hatter.png character sketch flicksync
ahh.png face animation flicksync
ch.png face animation flicksync
eee.png face animation flicksync
gah.png face animation flicksync
hmm.png face animation flicksync
mm.png face animation flicksync
oo.png face animation flicksync
wha.png face animation flicksync
eee.png face animation flicksync
k.png face animation flicksync
m.png face animation flicksync
oh.png face animation flicksync
oo.png face animation flicksync
sleep.png face animation flicksync
circle.png ui graphic flicksync
dialogue text box.png ui graphic flicksync
Mic_1 red.png ui graphic flicksync
endmusic1.mp3 music flicksync
intro narration.wav dialogue flicksync
Alice_(1-15).wav dialogue flicksync
dormouse_(1-3) dialogue flicksync
hatter_(1-11) dialogue flicksync
march hare_(1-9) dialogue flicksync
Adirondack+chair.skp 3d object gene Watson SDK sdk ibm Chair2.skp 3d object ikea dining chair forest_loop.wav sound inspectorJ 292+hill+house+1.skp 3d object kkc Alice's Adventure in Wonderland script lewis carroll Butter-2.skp 3d object lucas B tea+cup+06.skp 3d object manolis s. vegetation+set.skp 3d object martin c weeds1.skp 3d object martin c Oculus SDK sdk oculus Photon voice sdk photon Ponton Unity Networking sdk photon sucess1.wav sound fx renatalmar success2.wav sound fx robinhood76 Chair3.skp 3d object seeadler teacup1.skp 3d object sparky Chair.skp 3d object stk notori0us waiting_clock.wav sound fx straget VRTK sdk thestonefox Unity3D unity Unity Standard Assets unity
SteamVR sdk valve Old+chair(1).skp 3d object venste failsound.mp3 sound endmusic2.mp3 music flicksync
medievalintromusicB.mp3 music

Built With

Share this project: