The Why

Starting Friday night, we knew that we wanted to do something fun and interesting, but being first-time hackers we also wanted to make sure that we would be able to finish our hack. We discussed doing machine-learning based solutions for art analysis, writing analysis, the stock market, baseball, and many others. We talked about making a video game and trying to do VR. But from the minute this idea was brought up, there was no other option. We knew that with our various skills we would be able to create a workable project that was reasonably doable within the 36 hours, but that it would also be a lot of fun and require some thinking to complete.

The What

Rather than the ever-popular good photo identifying algorithms, our project aims to do the exact opposite: Identify the absolute worst and funniest photos of people from a video stream or a set of images. We had to do some creative thinking to actually determine which were the most terrible and hilarious photos, which are detailed below.

The How (Also Dependencies)

For this project, we used a lot of different Python libraries that allowed us to complete our tasks:

• OpenCV
  • This was used for almost every part of the project, as it involved numerous manipulations of images and other libraries that built off of it.
• dlib
  • dlib is a machine learning library built in C++ with bindings for Python. This allowed us to recognize faces in the images as well as estimate locations of facial features, which we used to determine which photos were the absolute worst in a video stream.
• Google Cloud with Vision API
  • We used Google Cloud's Vision API to classify emotions in our terrible images. We then used these emotion probabilities to determine which text database to pull from when drawing text.
• Pillow
  • Although OpenCV did about 95% of what we needed in terms of image processing, the quickest and easiest way to draw formatted text on images was through Pillow's ImageDraw object.

The Challenges

• Installing software libraries
  • dlib, we're looking at you.
  • Many of the software libraries we needed to use were harder to install than they were to use (and they were plenty hard to use)
• Creating the best algorithm to determine the worst images
  • We spent the majority of our time developing methods for detecting when people looked goofy and capture those images to be further analyzed and saved forever.
• Being funny
  • Being funny is hard when you've been developing for the past 18 hours straight.

The Learning Experiences

• Image processing
  • Both deterministic and machine-learning based image processing including writing text, extracting colors, manipulating points, face detection, pose estimation, etc.
• Google Vision API and web API conventions
  • Many of us had never used a major web API, so there was a learning curve.
• Python

Built With

• dlib
• google-cloud
• google-vision
• opencv
• pillow
• python