Inspiration

We wanted to be able to predict events in the upcoming Avengers Endgame movie, and what better way to do it, than to use data? We also wanted to deep dive into the topics of LSTMs and GANs when applied to text generation and image-from-text synthesis tasks.

What it does

Ghostwriter allows you to interactively co-create a script with the help of an AI agent that creates new lines based on small inputs by the user. It then creates an entire interactive report (complete with relevant GIFs) and an audiobook for the complete script.

How we built it

We created an RNN LSTM using Python (Jupyter, Keras, TensorFlow GPU) and trained it on a dataset of 23 Marvel movie scripts using Google Cloud Platform. This data was collected from https://www.springfieldspringfield.co.uk. Custom Python scripts were used to generate tags for the paragraphs, apply frequency analysis to predict an appropriate topic based on the sequence prediction. Based on these tags, an appropriate GIF was fetched using REST APIs. Furthermore, we used Google’s Text to Speech engine to create engaging audiobooks.

Challenges we ran into

The large quantity of data made it impossible for us to use local resources and setting up a machine learning environment from scratch was a difficult task. The LSTM model we created would give a very small coherent output and tuning its parameters (sequence length, number of nodes etc) would take copious amounts of time.

Accomplishments that we're proud of

We were able to create a working prototype of what we intended. Each component that we built developed our skills in a particular domain; be it in large scale machine learning, developing LSTM networks, researching, or interacting with APIs.

What we learned

We learnt how to model a text generation problem and effectively apply an LSTM neural network to generate subsequent text instances. We also developed practical skills on training large scale neural networks on cloud platforms within time bounds. The most challenging part of this project was attempting to create new characters and scenes given a script using GANs. It was perhaps too big and new a project to tackle within the given time frame. Although, we were unsuccessful in employing that technique, we learnt a ton of new stuff and figured out an alternative way to achieve our initial goal of creating movie scripts with images (GIFs), plus audiobooks.

What's next for GhostWriter

Tone and scene understanding of the script would be a huge step forward because it would enable the model to dynamically alter the script to create remarkable experiences. For eg: if the scene has been dramatic for a long time, the LSTM will generate scenes to make it romantic or thrilling instead. Using GANs to create new Marvel characters by training it on features (characteristics, images) of existing heroes. Using GANs to improve the relevancy and flow of the texts generated, thereby improving the quality of the text generations

Built With

Share this project:

Updates