Inspired by the ml5 machine learning library and Google's emoji scavenger hunt, we decided to create a similar game that incorporated train models to create an entertaining experience for the user. This is the first time we dip our toes in the vast ocean of possibility that comes with machine learning.

What it does

MadLads is a mobile web app reinvention of the famous phrasal template word game called Mad Libs, in which a player is prompted to fill in blanks to create a custom story. We took the same concept and incorporated machine learning and image recognition into the process of choosing words. In order to pick a word for the fill-in-the-blank, the user takes a photo of an object, which is then recognized and inputted into a list. After choosing up to five words, the user is presented with the crafted story, often humorous. The user then has a choice to try again for a different story with different words.

How we built it

We used ml5.js, a pre-trained, web-friendly machine learning library based on Tensorflow that allowed us to create web applications that use machine learning processes on the web. We integrated the image recognition portion of the library with a live feed and used p5 and vanilla Javascript to create an interactive, entertaining experience in the form of an AI-assisted Mad Libs game.

Challenges we ran into

There was a lot of trouble when it came to incorporating an environment-facing camera for mobile. However, in the end, we solved the issue by linking Google Chrome Dev Tools to our phones and learned that we had a simple syntax error in the createCapture object. In addition, the text-to-speech synthesis had a few glitches involving repeating strings of sentences, but it was fixed. There were also some tweaks required to adjust the sensitivity of the image recognition.

Accomplishments that we're proud of

We are proud of the smoothly functioning image recognition and phrasal template system, and our user experience design. We paid close attention to giving the user helpful feedback and options, such as allowing them to retry to capture the word and providing them with a little counter to keep in mind how many images they have left to take. Another example of user experience that we are proud of is our consistent style guides and interactive onboarding in the first slide, where the user can start trying out the image recognition system from the get-go.

What we learned

We learned how to incorporate pre-trained image recognition models into web projects.

What's next for MadLads

We would like to rewrite a lot of the code base, because much of it was done in a hurry without proper programming conventions. This should better our site’s performance. Another idea is to create a desktop version of the site that either redirects one to the mobile version or is a game in its own right.

Built With

Share this project: