High res link: https://youtu.be/cQZrp-UZsKg

Inspiration

Anime. You already know. It’s a larger-than-life, exhilarating roller coaster of emotions carried by intense and beautiful artwork.

What if we could make anything into anime?

What it does

Babigo turns video clips into animes by transforming the video and audio.

How it’s made©

A general adversarial network (CartoonGAN) is used to style transfer anime onto the video. Then, the audio is fed through AWS Transcribe, Translate, and Polly to translate, caption, and dub video in Japanese. Audio clips are separated by pauses, then adjusted to prevent overlaps. Finally, they’re stitched together using moviepy, producing your very own anime.

Challenges we ran into

• Getting dubbed audio to match original speech cadence.
• Balancing between detailed rendering and execution time.
• Installing dependencies with the appropriate versions.
• Imagemagick is hard to install the way you want to

Accomplishments we’re proud of

• The videos are funny, and probably contain atrocious, embarrassing Japanese.
• Automated dubbing matches lip flaps

What we learned

• How to style transfer, translate in the cloud, different approaches to syncing captions with speech - Josh
• I learned not to export the demo video 4 minutes before the deadline - Tim
• I learned why no one uses python to make videos - Quinn
• Babigo is the Japanese equivalent to Pig Latin - Andrew

What’s next for Babigo

We would have liked to merge the video and audio processing into a continuous pipeline so that we could put up a website to upload and convert videos. The video and audio processes took a significant amount of time to finetune, so the pipeline is one area for improvement in the next iteration. There are also other unique anime styles that could be added but require more complex methods. Ex. scene transitions, sound effects, etc.

Built With

• amazon-web-services
• general-adversarial-network
• moviepy