Background story

When we first heard about the AI Hackathon by Amazon, we were excited and decided to take part in it. As fellow Machine Learning practitioners, we were keen on the available APIs provided by Amazon. The first round of brainstorming was to decide on which API services to be used in our project. Since we have mainly dealt with image processing and videos in the past, we selected the Amazon Rekognition to analyse the metadata of an image and Amazon Comprehend to identify the insights and relationships of the metadata. Subsequent round of brainstorming lead us to the idea of having an application that match songs based on the image. "You provide image, we match songs for you!" Think of it as a reverse image search engine for songs. Instead of using keywords to search for songs, you use image instead.


When the user upload an image, there are two processes involved. In the first part, it will be sent to our server for visual sentiment analysis. For the second part, it will be uploaded to S3 and a lambda function is called to perform object identification and facial analysis using Amazon Rekognition. Both of the processes will generate synonyms in json file format. Prior to this, we already built an index library used to search and match songs using Amazon Elastic Search. A total of 3000 raw lyrics of songs, tags and emotion were fed to Amazon Comprehend to extract keywords and emotions via a batch job in AWS Lambda. The synonyms that are generated from Rekognition and visual sentiment analysis will be used to do a elastic search to match the most appropriate lyrics. Lyrics will be returned to our server via AWS Javascript. Then, the server will look for open-licensed songs in Youtube using their API and embed the videos to our webpages to be listened by the user. We extended it further by integrating another machine learning algorithm to generate a wordcloud from the lyrics. With this, the user can easily find songs that match their feelings with just an image because there are times that your feelings can't be easily expressed in words.


All in all, we are pretty satisfied with our project even though we faced some little hiccups during the integration process. The current application can be easily extended further by integrating other machine learning algorithms to provide a whole new level of services and experience to the users. We have learned a lot from this project and looking forward to more APIs from Amazon in the future.

Built With

Share this project: