Try it out: http://store.steampowered.com/app/827170/FlickSync__Mad_Hatter_VR/ https://play.google.com/store/apps/details?id=com.rockettigerstudios.flicksync

Inspiration

Imagine becoming your favorite movie character by taking their place as the main star. Embodying Darth Vader and “altering the deal”, or stepping into Jack Sparrow’s boots and drunkenly slurring “where’s the rum?!” Innovative entertainment means taking mediums we know and love and taking a new perspective.

Another main inspiration was Ernest Cline’s “Ready Player One”. In the story, when an OASIS avatar begins a FlickSync game, they become a specific character from a movie and have to recite lines of dialogue. When the avatar speaks, they will speak in the voice of the specific character, move around as the character, see, hear, and feel anything the character would see, hear, and feel in the movie.

What it does

Users can take on the roles of their favorite story characters, acting out scenes and speaking the lines of their assigned character. They enter into the Tea Party scene from Alice’s Adventures in Wonderland and become Alice, the Mad Hatter, or the March Hare. They can also play the scenes with their friends, with each person being assigned different characters using their own Oculus Rift, HTC Vive, and Windows MR headsets. During the scenes, when a character needs to speak, a cue card shows the user the lines assigned to them. They have to speak the lines correctly within an assigned period of time. Players are scored by how well they speak their lines of dialogue. The title of winner of the game goes to whoever gets the highest percentage!

How we built it

We used Unity with collaboration through Github. We used SDKs for voice recognition and split the integration of those SDKs between the devs. The artist created 3d models for the scene and the director made a script and recorded our audio. We then built out the game logic, and tested with several mentors for a few rounds of feedback for the polish.

Challenges we ran into

Public Domain Story: While there are a ton of amazing stories we would have loved to pick scenes from, including Star Wars, The Princess Bride, Rocky Horror Picture Show, Monty Python & The Holy Grail, Pulp Fiction, The Room, etc. the scope of this Hackathon required us to pick a story that was in the public domain and that was instantly recognizable. We decided on Alice’s Adventures in Wonderland as this fit the bill for the Creating Reality Hackathon.

Watson Speech to Text API: The Watson Speech to Text API was a little bit of a challenge to work with, but was extremely crucial to the success of this project as it hinges on speech recognition. The main struggle was in determining whether the user had said the correct lines and this was done through C# scripts analyzing the words said (given by Watson) and the current line they were assigned (via JSON). Another issue we faced is sometimes Watson Speech to Text API struggles with certain words and phrases making it a little difficult to properly emulate lines.

Photon & Watson conflicting about microphone ownership We used Photon for the multiplayer aspect of this project and encountered a few difficulties concerning the microphone ownership. Both Photon & Watson would fight for ownership of the microphone and so ultimately our solution was to share it, but initially this was a big issue.

Multiplayer Virtual Reality: Our multiplayer functionality ultimately wasn’t working 100% up to our standards but getting both players synced up and hearing each other did end up working. We struggled greatly with placement of players as each player is assigned a unique role.

Animation: We knew we wouldn’t have time to animate lip synced facial animation, so we used hand drawn facial animations hovering right in front of the character’s faces. This allowed us for a faster workflow and ended up being a quick fix that worked very well.

Original Assets: While it might look good to use pre-made free assets, we decided it would look even better if we utilized the talents of our very own team to create unique stylistic beautiful art assets. We created 3d characters, sounds fx, and music.

Character acting: There was a lot of dialogue, and 4 characters that already needed to be designed, modeled, painted, and rigged, which didn’t leave a lot of time to animate each character speaking their unique lines. We designed a single talking animation for each character due to the time constraint.

Accomplishments that we're proud of

We are proud of putting together a gorgeous and fun scene from Alice in Wonderland that allows the user to act out the part of Alice. We are proud to have gotten the multiplayer experience working, and for the Watson and Photon SDKs to be working together well. We are especially proud of the great teamwork throughout the few days. We made decisions quickly and each team member made unique contributions.

What we learned

We learned how to get multiplayer working (save a few bugs), how to use Watson, and how to get Unity working well with a team, working on it through Github and Source Tree. We tracked our tasks by using Trello having four columns - To Do, Work in Progress, Ready for QA, Done.

What's next for FlickSync

The potential of the tools and game we created is huge. We want to implement the following features: Track blocking and character action depending on the scene (interacting with props, moving around, throwing things) Camera To Watch Performance (if you want to re-watch or record from 3rd person) Multiplayer without bugs. Characters synced up in multiplayer A hard mode for disabling the cue cards Movable and grabable cue card like a tablet More comprehensive report card Smarter Watson Speech-to-Text Checking HoloLens and mobile VR headset support Voice Augmentation depending on character

FlickSync could be a used as a tool for actors looking to practice lines, or an artist could produce a full length play all in VR!

Google sheet with list of Assets: https://docs.google.com/spreadsheets/d/1bZYu_tRkHBk17OogwLab47cY7-YJfVPcdrMEpXy67uo/edit#gid=0

Below is that sheet copy and pasted should the above link not work (sorry for the formatting)

Assets and SDK Asset Name Asset Type Creator Url 1stteapot.skp 3d object andeciuala n https://3dwarehouse.sketchup.com/model/13961f08fe5b4c549ef60e9225cbea3a/Tea-Pot tea+set+1.skp 3d object archidol https://3dwarehouse.sketchup.com/model/50771bde64eff3ac27a11dcb35000f7a/Tea-Set Castle_Table_270X100.skp 3d object asee https://3dwarehouse.sketchup.com/model/u2de0e495-f57c-4b26-90d8-688163ce817d/Castle-wooden-table pocket watch.skp 3d object AW. https://3dwarehouse.sketchup.com/model/ae570b0ec2b265cd14621506c22882a0/Elgin-Father-Time-pocket-watch Caquetoire.skp 3d object cealpup https://3dwarehouse.sketchup.com/model/uaa6ae479-ea7c-4dbe-9434-b5d3b0cfbc46/Caquetoire-French-Renaissance-chairs rustic+english+farm+table.skp 3d object david heim https://3dwarehouse.sketchup.com/model/ueaba7abc-6145-40b0-b823-5c0ac640c644/Rustic-English-farm-table Old+chair.skp 3d object dragon_of_metal https://3dwarehouse.sketchup.com/model/967aa48c33622cb19fafdf8df0fa5da2/Old-Chair Alice_wonderland.blend 3d character model flicksync
hare_wonderland.blend 3d character model flicksync
hatter_wonderland.blend 3d character model flicksync
mouse_wonderland.blend 3d character model flicksync
hatter_wonderland.blend11 3d character model flicksync
mouse_wonderland.blend1 3d character model flicksync
alice.png character sketch flicksync
hare.png character sketch flicksync
hatter.png character sketch flicksync
ahh.png face animation flicksync
ch.png face animation flicksync
eee.png face animation flicksync
gah.png face animation flicksync
hmm.png face animation flicksync
mm.png face animation flicksync
oo.png face animation flicksync
wha.png face animation flicksync
eee.png face animation flicksync
k.png face animation flicksync
m.png face animation flicksync
oh.png face animation flicksync
oo.png face animation flicksync
sleep.png face animation flicksync
circle.png ui graphic flicksync
dialogue text box.png ui graphic flicksync
Mic_1 red.png ui graphic flicksync
endmusic1.mp3 music flicksync
intro narration.wav dialogue flicksync
Alice_(1-15).wav dialogue flicksync
dormouse_(1-3) dialogue flicksync
hatter_(1-11) dialogue flicksync
march hare_(1-9) dialogue flicksync
Adirondack+chair.skp 3d object gene https://3dwarehouse.sketchup.com/model/u5aa32b04-d7a2-4300-b3cd-b7908ace4e16/Adirondack-chair Watson SDK sdk ibm https://github.com/watson-developer-cloud/unity-sdk Chair2.skp 3d object ikea dining chair https://3dwarehouse.sketchup.com/model/ue3d2a7d0-f6fb-4ba1-be8b-8a718dc82a9c/Ikea-Dining-Chair forest_loop.wav sound inspectorJ https://freesound.org/people/InspectorJ/sounds/405561/ 292+hill+house+1.skp 3d object kkc https://3dwarehouse.sketchup.com/model/u2c83022c-e713-4875-b514-574c89b60162/292-HILL-HOUSE-1 Alice's Adventure in Wonderland script lewis carroll https://www.gutenberg.org/files/11/11-h/11-h.htm Butter-2.skp 3d object lucas B https://3dwarehouse.sketchup.com/model/udb516d3e-e9d1-4710-8450-1621f50ed837/Butterholder-with-butter-and-butter-knife tea+cup+06.skp 3d object manolis s. https://3dwarehouse.sketchup.com/model/31fcf965836ab3484212ff51b27f0221/tea-cup vegetation+set.skp 3d object martin c https://3dwarehouse.sketchup.com/model/ue91bb468-af3e-47f0-a13c-135aee99c604/Vegetation-set weeds1.skp 3d object martin c https://3dwarehouse.sketchup.com/model/ua02d5943-a899-40d5-915b-e09f6e41b555/Weed-set Oculus SDK sdk oculus https://developer.oculus.com/downloads/ Photon voice sdk photon https://assetstore.unity.com/packages/tools/audio/photon-voice-45848 Ponton Unity Networking sdk photon https://assetstore.unity.com/packages/tools/network/photon-unity-networking-free-1786 sucess1.wav sound fx renatalmar https://freesound.org/people/renatalmar/sounds/264981/ success2.wav sound fx robinhood76 https://freesound.org/people/Robinhood76/sounds/178318/ Chair3.skp 3d object seeadler https://3dwarehouse.sketchup.com/model/678ae2d4a386f4e4d0250aa1ac19fcf9/Chair teacup1.skp 3d object sparky https://3dwarehouse.sketchup.com/model/e97a6c7176c918c77f6af6e903744c5a/cup-of-tea Chair.skp 3d object stk notori0us https://3dwarehouse.sketchup.com/model/b24b70c9aaec3932cf577b918c6f725b/char waiting_clock.wav sound fx straget https://freesound.org/people/straget/sounds/405423/ VRTK sdk thestonefox https://github.com/thestonefox/VRTK Unity3D unity https://unity3d.com/ Unity Standard Assets unity
SteamVR sdk valve https://github.com/ValveSoftware/openvr Old+chair(1).skp 3d object venste https://3dwarehouse.sketchup.com/model/u73e27858-9f40-4dd7-ac55-35622800ad7f/Louis-XVI-style-chair failsound.mp3 sound http://soundbible.com/2185-Old-School-Bell.html endmusic2.mp3 music flicksync
medievalintromusicB.mp3 music http://imslp.org/wiki/Songs_for_the_Lute%2C_Viol_and_Voice_(Danyel%2C_John)

Built With

Share this project:

Updates