Poet.AI

Inspiration

• My grandmother loves to write poetry.
• When I was a kid she would write poems,
• We would record her reading them,
• I would google images for scenes, edit it all together, and upload to YouTube.
• I never forget the glee on her face when we released a new video.
• She would go tell all of her friends about her new video!
• I created Poet.AI to automate my role as googler and editor,
• To use AssemblyAI, GPT-3 and Stable Diffusion to do what I did but much better!

What it does

• You upload an audio recording of your poetry,
• Poet.AI generates a transcript of what you said,
• Uses GPT-3 to partition that into scenes and generate prompts for those scenes,
• Uses Stable Diffusion to generate images for those scenes,
• Puts it all together in Python and gives it back to you.

How we built it

• The website is built in Django and it uses the AssemblyAI, OpenAI GPT-3 and Replicate Stable Diffusion APIs.

Challenges we ran into

• It was a pleasure using all of the APIs,
• However for a long time the videos for the poetry just didn't look good!
• Then I found a blog post of using GPT-3 to make better prompts, implemented that and it made it so much better!

Accomplishments that we're proud of

• It actually works and generates beautiful videos!

What we learned

• How to use a bunch of different machine learning APIs (Assembly AI, Replicate Stable Diffusion)
• How to use GPT-3 for weird things that it probably shouldn't be used for lol (extracting important phrases from text, etc.).
  • I would loved to have used Cohere for this but just didn't find the time to learn another new API/toolkit.
• Creating a polished UI

What's next for Poet.AI

• Making it much faster, it takes longer than 10minutes to generate a single video.

Built With

• assemblyai
• django
• python
• stable-diffusion