Our brainstorming brought up the idea of being able to identify the text and template for memes. Popularly used on the internet, it seemed like a fun idea to try attempt this.
What it does
Our goal was to make a project that could do 2 key things: extract text from the image, and recognize the meme based off the template. The challenges presented an alternative method halfway through to attempt to identify the template from the text instead of using the image. There is no cohesive project, there's no proper code to run, we simply submitted all the incomplete misc code we worked on.
How we built it
We divided the task into several sections, making a simple front-end, the image OCR to extract the characters, image pre-processing, and the ML that would be used to identify the template. We found a dataset of over 50,000 images, already sorted into categories based on the meme, with the text on the side. https://labs.loc.gov/experiments/webarchive-datasets/ links to our used dataset.
Challenges we ran into
Halfway through the second day, after consultation with experts and some thinking on our part, we realized that it was unfeasible to read and analyse 50,000 images then use a simple linear SVM. Rather than devote the resources required to get that working, we opted to try a simpler approach; to identify the images by text rather than by image. Image analysis using a tool such as SimpleCV proved to be a fairly complex task in terms of setup, whereas the text is a lot lighter, likely to have a linear relation and thus easier to model with ML. For the OCR extraction, most of it ran smoothly. However due to the image nature of memes, no single image processing strategy was universal, and we had difficulty extracting the text accurately. We were able to get around 70% of the memes to extract text accurately. Since this was the step now required to complete before running the image through the model, 70% correctness was low. After this, we ran into organizational difficulties; we decided to try working on both models for the ML (text and image based) since each approach favoured either accuracy or simplicity. Of course, this divided resources and our work and additions no longer supplemented each other. The front end, a simple website, was also developed on the side. This makes use of any files we had working, currently just the basic OCR taking an image as input and outputting its text: http://nateladd.com/my_files/MMF.html
Accomplishments that we're proud of
We did a lot of experimentation and learning. Ultimately our project didn't work; we ran into a lot of random bugs with getting the tools to work as well as the organizational difficulties with members working on the same section and difficulties with some people adding their code to the github. All of these were learning experiences, which is what we all wanted, especially being all of our first attempts at a hack-a-thon.
What we learned
Many of us went into this project with essentially no ML experience. After this we learnt a lot about the tools used in the process, different implementations, different models, various algorithms, etc. We learnt more about python, and better work practices, which although not well implemented this time, we'll be able to use smoothly in future projects.
What's next for Meme Machine Learning
This tool itself is incomplete. One day we might come back to this and link all the disjoint elements together as a side project. Beyond this, the original long term plan would be for meme analysis which could ultimately be used to identify meme trends, and potentially from there be used to automatically generate memes.