We've found that groups that use GroupMe on a regular basis (such as our a cappella group, Acapellago!) often have fun taking pictures of each other and with each other and sharing with the group. We wanted a way to add a kick to this fun.
What it does
Our GroupMe bot analyzes photos sent to a GroupMe and uses machine learning to determine a one-line caption for what is in the photo, and if the photo contains faces, make an assessment of the emotions of the people in the photo. It cleverly and intuitively delivers a message back to the group with this information, varying the response based on how confident it is in it's guesses.
How We built it
We used a server written in Python using Flask. It extracts images from GroupMe messages as they are posted, feeds it to Microsoft Computer Vision to determine the caption of the photo, analyzes the number of faces present (and their locations) and then feeds it to Microsoft Emotion to detect emotions of any and all faces in the photo (if present).
Challenges We ran into
Our hack uses several different APIs, and as a result handling interactions between them all was frustrating. The differences in text encoding between Python 2 and Python 3 also gave us some challenges. We also occasionally ran into problems with Microsoft's API where high-resolution images would hit the 4MB file size limit.
Accomplishments that We're proud of
We were really proud when we were able to work through the errors in connecting to the server and the Python version issues, and our messages were sending what we needed them to send. After that, we were excited to have a chance to be creative with the bot's phrasing and words!
What We learned
We learned about Python and Flask, and especially how to integrate Microsoft Cognitive Service APIs.
What's next for Imager? I hardly know 'er!
We're excited to use Imager in our a cappella group me for fun and giggles!