This pandemic has forced us all to be involved in some unconventional social situations. We are often asked to go on professional Zoom calls unexpectedly, video chat with a romantic interest at a moments notice, or have impromptu hangouts with other lonely friends. With our appearance still playing a pivotal role in all of this, how can we feel like our appearance is up to par if we have to be isolated? To solve this issue, we turned to existing artificial intelligence. That way if we are looking to FaceTime with a romantic interest, we'll have a bot capable of being honest about how we look.
What it does
The compliment generator requests a picture of the user that can be taken directly on our web page using the current device's camera or uploaded from the device's storage. With each submission, the user is given the option to indicate what kind of social setting they will be attending (casual, professional, or date) and whether they would like to hear gender-related comments (the AI is capable of detecting gender). Artificial Intelligence is then used to assess the image, weigh all of the prominent features observed in the user's picture, and then return a comment about the user's appearance as text on their screen. Despite the name of the project, an overwhelming majority of the comments the bot is configured are jocular in nature and intend to give the user a laugh during these trying times.
How I built it
A series of Python scripts that run a combination of Betaface facial recognition software and Amazon Web Services libraries are then used to assess about two dozen facial features and make distinctions about the user's appearance. The scripts then generates confidence scores, indicating which features are the most prominent. Once the most prominent feature is identified, it locates a matching "compliment" that is compatible with a user's social setting/gender by searching through an Excel sheet using Pandas, and displays it on the screen.
Challenges I ran into
The most prominent challenge was being able to hand over the photo from the front end of the web application to the Python code on the back end. With team members having minimal PHP and Database experience, transferring the uploaded photo to our Python code created a huge bottleneck. Despite coming up with a rather simple text file solution, the challenge was working with a language (PHP) and a concept (exchanging data between the front and back end) we were completely unfamiliar with.
Accomplishments that I'm proud of
I'm extremely proud of our group's ability to work with two relatively new and complex technologies that we would never have exposure to inside of the classroom. The first of which was acquiring images directly through a user's device camera via their web browser. This granted us the opportunity to work with HTML canvas objects and see the potential they have in web development in the future. Second was working with the AWS and Betaface API. The underlying machine learning properties of these facial feature extraction libraries were extremely hard to grasp, documentation was rather limited, and the code bases themselves were massive. These were all hurdles that we needed to jump over in order to have a functioning project. Dealing with these complex technologies despite having no background makes having a finished project an accomplishment in itself.
What I learned
What's next for Compliment Generator
We would like to find a way to integrate text to speech software on the front end of our project. That way when the user provides their photo, the AI will actually be able to verbalize its comments. In addition, we would like to upload our project to a commercial web server to handle increased traffic. Since it currently resides on a home server, it is in no way ready to handle a heavy/steady stream of users.