People often imagine what it would be like to be their favorite characters in their stories. We decided to flip this and make it so that we could be our favorite characters in our daily lives.
What it does
Waifyou takes a webcam feed of you and produces you as a anime character. Not only do you become a character on video, you also can say things using the characters voice.
How we built it
We achieved this by using OpenPose for body recognition and then used OpenGL to render the models over you making you into an anime character. For the voice, we created a library of words from the voice actor of the character and then we used PocketSphinx on a Dragonboard to analyze what the user says, converts it into a string which is then used with ffmpeg to string together the words into audio files. This was done by making a word library of the voice actor by taking clips from their respective shows, cutting out each word and then saving them as audio files. Then, after we received a string from PocketSphinx, we put parsed the string and then created a merged audio file of what was said using the word bank. Then we played out the audio file of the sentence you said except with the voice actor.
Challenges we ran into
The Openpose library returns a list of 2d points, and inferring the 3d data from the image proved to be a formidable task. We tried using skeletal animation for the model, but instead settled for breaking the model into individual independent pieces and controlling them individually. For the voice recognition and playback, we had trouble making it playback at a meaningful speed where it wasn't incredibly delayed. We did this by simplifying our System()calls as well as making our directories more favorable towards faster system calls.
Accomplishments that we're proud of
We are really proud of the voice recognition system. Not only does it recognize voice reasonably accurately, unlike Google voice recognition ours is entirely offline and the coding was done entirely in C. We chose c because its easier to talk to the hardware which allowed for faster more efficient programming which lead to lower delay on the playback of the characters voice. Plus you can see the arms move.
What we learned
String manipulation in C sucks. In the process we learned a lot more about working with the terminal because this was the most effective way to interact with the Dragonboard. We learned a lot about the various libraries that were used in this project as well as the dangers of C++ and C.
What's next for Waifyou
Our current version only works with Korra as we were limited on time to develop a solid library for other characters as well as their 3-D models. Eventually we wish we would be able to implement this on something like google glasses so we can take augmented reality to the next level by making AR into AR (Anime Reality).