We all know a picture is worth a thousand words, but how about 17 syllables? This project was born from the synthesis of computer science, photography, and poetry. We really wanted to explore the possibilities of applying machine learning to artistic creations.
What it does
Piccoku uses computer vision, synonyms, grammar, and syntax rules to write you a haiku based on user-generated photos, attempting to distill the artistic essence of smartphone snapshots into poetry. The haiku is a style of poetry characterized by 3 lines in specific syllable patterns.
The haiku is generated by our algorithm, then overlaid onto your image, creating your "piccoku" (picture + haiku), which can easily be shared through social media channels or messaging applications.
To see a gallery of all piccokus, check piccoku.com.
How we built it
Algorithm & backend: We used Microsoft Azure's Computer Vision API to extract keywords from the user-submitted images. Then, the keywords were added to phrases from datamuse API (synonyms, syntax, grammar, syllable count). We had a web app built through Rails, primarily for data persistence. Communication between the iOS app and web app is facilitated by exposed endpoints that acted as RESTful APIs.
iOS app: Swift and XCode
Web app: The web app is running on Heroku due to the ease of configuration and usage. Formatting done in HTML/CSS.
Challenges we ran into
We spent a significant amount of time figuring out how to encode and decode information that was being sent between our iOS app and our web app. This was because spaces are usually encoded as "+" signs. Tackling this was quite a challenge. Setting up Rails on Heroku was painful considering a new database had to be setup in PostgreSQL, replacing the default mySQL. This is because Heroku does not support mySQL databases for Rails apps.
Accomplishments that I'm proud of
Our group was extremely proud of our diverse skill set, and the way we delegated and trusted each other with parts of the project. We all took ownership immediately over different areas (ex. web app, iOS app, UI/UX, backend), and we couldn't have done this project without any of the group members.
What we learned
All of us did analysis on haiku structure, language, and poetry, in order to inform the algorithm that Piccoku uses.
What's next for piccoku
We would love to build out the functionality of Piccoku to write more complicated poetry, such as sonnets or limericks. We also think Piccoku could be a useful tool for educational purposes, as it brings some interesting insights about the meaning of art, intention, and artificial intelligence.