Check-in #2:
Introduction
We will be creating a Poetry Generator. This project will be heavily centered around Natural Language Processing and developing a Seq2Seq model. We ultimately decided on this project because we were more interested in natural language processing compared to structured image prediction, as well as an interest in poetry and writing in general.
The poems generated should be both cohesive in subject matter AND rhymes. There has been previous work to address each of these issues individually, but there has not been a successful model that can solve both and generate meaningful, rhyming poetry.
The problem of pursuing generating poetry using language models is as old as Deep Learning and the problem is becoming deeper year after year. After learning about the progress on OpenAI’s generative pre-trained transformer, as avid poets we wanted to explore how the model would perform given large corpuses of poems. By generating realistic sequences of rhythmic poetry, we can train the model to improve both the content and form of poems.
Challenges
What has been the hardest part of the project you’ve encountered so far?
Preprocessing two different datasets of varying styles proved to be more challenging than originally expected because content from the Poetry Foundation dataset needed additional cleaning to match the consistency of the content from the Gutenberg poetry dataset. Preprocessing and matching the style of the datasets for a unified one was frustrating to the point we were considering only using a single dataset. Ultimately, we decided to leverage both datasets for a more effective model!
We also took some time to pin down what we saw as the core of the project, the most difficult and interesting part of our idea. We decided that this was the process of converting some sample of text into a structured, rhyming poem with minimal loss of content. If we can do that, then we can be confident that the entire process of rhyming poem generation can be automated from scratch. This is because we know that GPT-3 can generate meaningful text samples from scratch. We would then be able to feed these sequences to our model.
Insights
Are there any concrete results you can show at this point? How is your model performing compared with expectations?
While there are no concrete results shown at this point, we are optimistic that our model will start to generate rhythmic poetry once we have completed preprocessing our poetry data and loaded in the phonemes dictionary. We have started to implement a basic LSTM model, however, we will not be able to test it or optimize it until later.
Plan
Are you on track with your project?
We may be slightly behind in our initial goals for this project, but we are looking forward to making a good amount of progress this week and more so once preprocessing is complete. It is difficult to build a good model when we cannot train it and test it to measure its performance, however, we hope to have a basic model implemented by the end of this week. We are excited to start to gather preliminary results, which we can then use to expand and improve our model.
What do you need to dedicate more time to?
We should probably spend more time meeting and discussing the specifics of our model implementation. We have been stuck at high-level discussions for too long, probably.
What are you thinking of changing, if anything?
As mentioned in the challenges section, we are dropping the GPT component of our project. However, we may use GPT-2 in our presentation in order to demo a complete poem generation from scratch.
Log in or sign up for Devpost to join the conversation.