MoodMarketAI desktop app (1/4) - detecting attention & neutral face
MoodMarketAI desktop app (2/4) - detecting attention & happy face
MoodMarketAI desktop app (3/4) - detecting attention & fearful face
MoodMarketAI desktop app (4/4) - detecting no attention
MoodMarketAI online merchant dashboard (1/4) - on login/register page
MoodMarketAI online merchant dashboard (2/4) - on dashboard page, with notification panel open
MoodMarketAI online merchant dashboard (3/4) - on dashboard page, scrolled down
MoodMarketAI online merchant dashboard (4/4) - on profile page, with user options prompt open
MoodMarketAI chrome extension (1/1) - showing amount the user was rewarded for shopping online using MoodMarketAI this month
MoodMarketAI is the AI-powered Ecommerce Analytics Platform of the future, which provides businesses with new emotional insight into their customers, and monetarily incentivizes customers to spend more time shopping online.
Our team was inspired to create this project when we realized that there were many problems with all ecommerce analytics platforms. Currently, if you own a business, you would typically use solely Google or Facebook Analytics for your ecommerce data analysis needs. However, these existing platforms have one major flaw — they lack emotional insight into the customer. According to an enormous amount of research papers, your emotions (especially how happy or sad you are) play an important role in your purchasing decisions.
In fact, Harvard Business School Professor Emeritus Gerald Zaltman believes that:
“95% of purchasing decisions are subconscious and emotionally-driven”
Since existing analytics solutions lack customer emotion monitoring, ecommerce sites using only them are missing out on a lot of untapped potential revenue and growth.
With MoodMarketAI, we solved this problem, filling a gap in the market.
[see next section “What it does” for more details on what MoodMarketAI is]
What it does
MoodMarketAI is an artificial-intelligence-powered ecommerce analytics platform that benefits both the business and the customer, allowing ecommerce businesses to safely monitor and make business decisions based on anonymous customer facial emotions and microexpressions, while rewarding customers for spending more time shopping online.
All businesses have to do is create a business account on https://moodmarketai.com. Once they have linked and verified that they own their ecommerce site, they can add money to their MoodMarketAI account to pay for the platform’s customer mood tracking services.
On the other hand, all customers have to do is download the MoodMarketAI app (which is available for Windows, MacOS, and Linux) and Chrome extension and create a MoodMarketAI customer account.
Technical, Detailed “How It Works”
The MoodMarketAI desktop app uses computer vision and Haar Cascade classifiers to detect, locate, and isolate your face from the rest of the webcam feed. It then takes the FER-2013 dataset of 35,887 grayscale 48x48 images of faces and feeds them through a 4-layer convolutional neural network. This creates an artificial intelligence model that is trained against sample images in the dataset for 50 epochs to improve accuracy. The resultant machine learning model is used to quickly predict your emotion on your face in the webcam feed that the previously mentioned Haar Cascade classifier already isolated. All video processing is done locally on your computer, so privacy is ensured, since only anonymous text data is sent to MoodMarketAI cloud storage.
The MoodMarketAI desktop app then anonymously and securely stores the customer mood data, attention data, and other data in Firebase Cloud Storage. The user’s Chrome extension also stores ecommerce site data on Firebase. This data is then aggregated per ecommerce site for all users, and is is viewable by the original business through the online MoodMarketAI merchant dashboard at https://moodmarketai.com which can then be used by the merchant make business decisions based on the anonymous customer facial emotions and microexpressions.
Finally, at the end of each month, the business is charged 20 cents for each hour a customer actively spends browsing their ecommerce site. Additionally, to incentivize customers to spend more time actively shopping online, customers will earn 10 cents per active hour of online shopping on a website that has MoodMarketAI enabled.
How we built it
Finally, for the MoodMarketAI desktop app, we used OpenCV with the open-source frontal face Haar Cascade classifier to detect, locate, and isolate the face and determine whether the user was looking at the screen or not. We used Tensorflow with Keras to create a 4-layer convolutional neural network in Python which was used to create a machine learning model and predict customer moods.
Challenges we ran into
We had plenty of issues getting Firebase Storage Database/Login and the Chrome extension to work together as intended. I had only used Firebase as part of a web app or website before, so this was unexplored territory for me, and it turns out there were many essential details specific to Firebase in Chrome extensions, and I ended up having one or two extension signing key and login issues which I wasn’t able to fix.
We also had plenty of issues with OpenCV and the convolutional neural network that was used to create the AI model. I would frequently get vague and generic errors with no further explanation, so I spent a lot of time debugging obscure issues whose resolutions couldn’t even be found on StackOverflow. In the end though, the OpenCV code and the artificial intelligence model did work perfectly as intended in isolating the face, checking for the user’s attention, and predicting their moods. However, I ran out of time to connect the completed frontend UI/UX of the MoodMarketAI desktop app with the Firebase login and cloud storage.
Additionally, the MoodMarketAI online merchant dashboard website took quite a long time to make because it had so many elements, only adding onto the pressure of coding so much in such a short time frame. As a result, I never got time to connect the completed frontend UI/UX of the dashboard website with the Firebase login and cloud storage.
On the less technical side, I also encountered the issue of sleep, or rather the lack thereof, since I spent so much time working on this hackathon project. Over the 3 days and 2 nights of this hackathon, I only got around 10 hours of sleep. You can bet that I’ll be going to sleep as soon as possible after the hackathon closing ceremony.
Accomplishments that we're proud of
I am proud that I got the OpenCV computer vision Haar Cascade classifier working as I had very limited experience with these types of classifiers. This is the first time I’ve personally used artificial intelligence and machine learning in my hackathon project, so I’m very proud that it ended up working perfectly as intended in predicting the user’s mood. What we learned
What's next for Mood Market AI
Our team would like to further improve the accuracy of the computer vision face detection, and further develop and further train the AI model for predicting emotions to a more complex and granular convolutional neural network. Finally, we are interested in polishing the project and hopefully launching an alpha version of the service online initially as a real-world test run. MoodMarketAI has business potential in a large market since it’s safe and secure customer emotion monitoring provides ecommerce businesses with new data analytics that can help them make better business decisions and can help increase revenues in addition to monetarily incentivizing customers to spend more time shopping online.
- Computer vision, AI/ML model for face detection and emotion detection
- User interface for web merchant dashboard
- User interface for consumer-facing chrome extension
What still needs to be done:
- Add potentially more AI/ML classifiers such as age
- Link firebase storage database and login functionality to web merchant dashboard interface and the consumer-facing chrome extension and desktop app