When you were a child, you probably love superheroes like Spider-Man or Wonder Woman. Sometimes, it is pretty expensive to order a costume and it will only fit you for a while before you grow up. Swapperheroes provides a way for you to body swap yourself with another superhero in order for you to roleplay and create your own story!

What it does

Swapperheroes swaps your body with a superhero like Spider-Man, including your face, so you get to be able to live in the world of the superhero of your choice. Frames are sampled every 2 seconds to not overwhelm the backend, to produce the output where you see yourself as a superhero like Spider-Man. For each of the frames, we gather the Marvel API comic descriptions and other comic sources to attempt to create a logical story in the Story Generator. There are also eBay listings where you get to purchase the particular costume if you really love this superhero! If you want to talk to the superhero, you can also use the chat box, where a chatbot will be more than happy to initiate a conversation with you!

How we built it

We use React, JavaScript, HTML, and CSS to create the frontend web application. We also used the eBay API to obtain the top three listings of the particular costume of the superhero. To obtain the stories and backgrounds of the superhero, we used the Marvel API, in particular, the Marvel Comics API.

On the backend, we used Python to train the Tensorflow machine learning model Cycle-GAN (Zhu et al., 2017). To obtain training data, we used the Spider-Man video game trailer as well as various Spider-Man movie trailers, obtain the particular frames where Spider-Man is mostly present, and extract those video frames into images using VLC. For some videos that need rotation for the image outputs, we used Magick and Shell Script to batch rotate the images 90 degrees to produce image outputs that are in the desired orientation.

For the chatbot, we used Microsoft LUIS (Language Understanding Intelligent Service) and the Azure Bot framework to understand and respond to queries from the user.

Challenges we ran into

The Cycle-GAN machine learning model was the most challenging part since we had to use a powerful GPU in order to train our model of superhero poses and a human equivalent. It also took a lot of effort to obtain training data for the machine learning model.

What's next for SwapperHeroes

Provide functionality to swap to other superheroes like Hulk, Batman, and Superman, which requires separate models for the Cycle-GAN. We would also like to add a more robust way to create a story. Improve chatbot functionality by providing text message mechanisms and database storage of what users like in terms of the superheroes.

Built With

Share this project: