Imagine

Home Page
Gallery
Upload Page

Inspiration

As a team, we wanted to challenge ourselves to work on a project in a new field that we are not very familiar with. We had heard about the rise in digital art in the past few months and we thought it would be interesting to take a deeper look at how AI can be used to create digital art. We also thought about the unique twist of including how we could have AI enhance human art, but after some setbacks and a major pivot we refocused on how AI can interpret and build up on many of our classic works of literature.

What it does

Imagine is a web platform that lets users upload their ebooks and view digital art created by AI from the summaries of those ebooks. Users can understand how digital art created by AI is similar and different to art created by people. It gives users a unique perspective on how AI such as DALL-E interprets the meaning of books both literally and contextually and represents that meaning through digital art.

How we built it

Imagine is powered by Python, and it leverages the FastAPI framework. Our LLM of choice was Cohere, as it doesn’t limit us nearly as severely as GPT’s API does. With Cohere we can leverage the instant summarization model to extract all the visual descriptions from chunks of text. For generating the image, DALL-E 2’s API cannot be beat, for now. DALL-E generates an image, but we spent a significant amount of time tuning the LLM aspects of the project to fit within DALL-E’s 1,000 character prompt limit. Lastly, to keep everything in order, we utilized MongoDB deployed with MongoDB Atlas to keep track of the books we have processed and information about books that we have processed.

Challenges we ran into

The biggest challenge we faced was the response times of DALL-E and LLMs. Our original idea was to insert images to your reading multiple times per paragraph to encourage reading to a greater audience, but we ran into a major challenge with speed. The first plan to get around this was to parallelize the processing, but that actually was the direct cause of a different issue. We were still using GPT which was utilizing more than ten thousand tokens per minute and subsequently was cut off. Switching away from the GPT API changed how we decided where to add an image because Cohere was not nearly as well suited for the task, and we eventually were forced to pivot the direction of our project. We decided rather than improving our interpretation of human creativity with generative art, what if we explored generative AI’s ability to interpret classic works of literature. As such, we were able to limit the scope significantly more, and use Cohere and DALL-E to make a functional product, which also did not cost too much to run, compared to the early GPT API version we were building. With this major shift, we needed to rethink a large chunk of our project, with less than one third of the time remaining. We struggled finding a reasonable feature set to accomplish the idea in the limited time available. However, we decided that since books from Project Gutenberg could be automatically downloaded and processed, it would be considered a success. We essentially had to scrap all the UI code and the prototypes we made in Figma, but we pulled together pages that met all of our functional requirements. As for smaller issues, we ran into some troubles fine tuning our prompts. As alluded to earlier, DALL-E’s 1,000 character limit was hard to work around. Getting Cohere to be effective required significantly more thought than our initial GPT 4 implementation because while it could extract information, getting it to extract visual descriptions turned out to be very challenging.

Accomplishments that we're proud of

We are proud of the many learning experiences we had throughout this hackathon. We are proud of our ability to learn and build a seamless front-end that is easy to use. We are also proud of our use of MongoDB to store data that we produced from DALL-E’s digital art feature. Configuring MongoDB for this task was challenging but benefits such as keeping multiple types of data organized in one place helped save a lot of time in the long run. Most of all, we are proud of our flexibility and ability to pivot! Our original plan did not work out and with little time left we were able to take advantage of some of the code we architected for a new purpose.

What we learned

By working on this project, we learned that AI and DALL-E are not perfect and there are many times when they make mistakes that can affect the entirety of the program. We also learned that the problems arising from working with AI are not always easily solvable and can lead to a major challenge. Overcoming these challenges and finding ways to work around these problems when we were stuck was something we learned together.

What's next for Imagine

In the future, we will add a more thorough user interface that gives the user further information about a certain book or piece of art produced to represent that book. We hope to go back in the future and use DALL-E to generate more digital art for both previous and new books to observe the differences in generated art based on different times. We would also like to expand the collection of books and their corresponding visual representations by making it more accessible to users of all ages.