Inspiration

This pandemic has forced us all to be involved in some unconventional social situations. We are often asked to go on professional Zoom calls unexpectedly, video chat with a romantic interest at a moments notice, or have impromptu hangouts with other lonely friends. With our appearance still playing a pivotal role in all of this, how can we feel like our appearance is up to par if we have to be isolated? To solve this issue, we turned to existing artificial intelligence. That way if we are looking to FaceTime with a romantic interest, we'll have a bot capable of being honest about how we look.

What it does

The compliment generator requests a picture of the user that can be taken directly on our web page using the current device's camera or uploaded from the device's storage. With each submission, the user is given the option to indicate what kind of social setting they will be attending (casual, professional, or date) and whether they would like to hear gender-related comments (the AI is capable of detecting gender). Artificial Intelligence is then used to assess the image, weigh all of the prominent features observed in the user's picture, and then return a comment about the user's appearance as text on their screen. Despite the name of the project, an overwhelming majority of the comments the bot is configured are jocular in nature and intend to give the user a laugh during these trying times.

How I built it

The front end of this project was constructed using codepen.io and contains a combination of Javascript, HTML, and CSS. This portion of the web application is responsible for having the user configure the social setting they are going to, as well as giving them the opportunity to take a selfie directly in their web browser if they do not have a photo to upload. Residing on a Linux-based Apache web server, PHP is used to store a temporary URL in a text server.

A series of Python scripts that run a combination of Betaface facial recognition software and Amazon Web Services libraries are then used to assess about two dozen facial features and make distinctions about the user's appearance. The scripts then generates confidence scores, indicating which features are the most prominent. Once the most prominent feature is identified, it locates a matching "compliment" that is compatible with a user's social setting/gender by searching through an Excel sheet using Pandas, and displays it on the screen.

Challenges I ran into

The most prominent challenge was being able to hand over the photo from the front end of the web application to the Python code on the back end. With team members having minimal PHP and Database experience, transferring the uploaded photo to our Python code created a huge bottleneck. Despite coming up with a rather simple text file solution, the challenge was working with a language (PHP) and a concept (exchanging data between the front and back end) we were completely unfamiliar with.

Accomplishments that I'm proud of

I'm extremely proud of our group's ability to work with two relatively new and complex technologies that we would never have exposure to inside of the classroom. The first of which was acquiring images directly through a user's device camera via their web browser. This granted us the opportunity to work with HTML canvas objects and see the potential they have in web development in the future. Second was working with the AWS and Betaface API. The underlying machine learning properties of these facial feature extraction libraries were extremely hard to grasp, documentation was rather limited, and the code bases themselves were massive. These were all hurdles that we needed to jump over in order to have a functioning project. Dealing with these complex technologies despite having no background makes having a finished project an accomplishment in itself.

What I learned

If I were to dive into every new technology or concept I was exposed to this section would be a few pages in itself. To keep it brief, for Keval and myself, we gained some valuable insight in to how powerful Javascript is. Both of us have only done coursework with back end technologies and sort of took web pages for granted. Being able to help write Javascript code and seeing the powerful/complex features it could implement really opened our eyes to a part of software development that we had neglected. Our other teammate, Keegan, had an opposite educational experience in the regard that he had worked exclusively with front end technologies. He had made it clear to us how significant it was to see the ups and downs of navigating through large libraries of code and attempting to tie together their inputs and outputs. Seeing the finished project gave him a new appreciation to how much time and effort is required to even get the simplest of features implemented.

What's next for Compliment Generator

We would like to find a way to integrate text to speech software on the front end of our project. That way when the user provides their photo, the AI will actually be able to verbalize its comments. In addition, we would like to upload our project to a commercial web server to handle increased traffic. Since it currently resides on a home server, it is in no way ready to handle a heavy/steady stream of users.

Share this project:

Updates