-
watson screen of accuracy
Inspiration
Digital reviews are vital to sustainable e-commerce ecosystems. Weather you’re checking out a new TV on Amazon, looking for a place to stay on AirBnB, or just trying to find a recipe for quiche, a helpful review can be the make-or-break for your decision. However, many online reviews are plagued with misinformation and can lead readers astray. We find it is common to post a review if you are exceptionally impressed or uninpressed, often making these reviews driven by strong emotion. A big problem we face today is that there is no foolproof way to tell if reviews are constructive and useful automatically. If we could curate valuable reviews (outside of the crowdsourced space), we could better inform customers and encourage review quality.
What it does
Our Quality Scorer uses a machine learning algorithm that was trained on thousands of yelp reviews. We can input a new review and output a usefulness score with 95% accuracy.
How we built it
We used IBM Watson’s Natural Language Understanding, python notebooks, and a set of 21,000 yelp reviews to train a decision tree classifier that analyzes text to determine its usefulness as a review. First the review is parsed and given explicit features with IBM’s NLU, then we run it though our decision tree living in IMB’s Watson suit.
Challenges we encountered
As with all machine learning problems, finding a large enough data set that is always the most important and difficult challenge. Luckily, after much searching, we found a dataset we could easily manipulate and label. However, this didn’t spell the end-all for our issues. We ran into memory shortages and painful slowdowns since we were working with huge files — originally we had a set of 6 million labeled examples.
Accomplishments that we're proud of
We were able to achieve a 94.7% test accuracy for our algorithm after training it with 16,000 examples compaired against a validation and test set of 2,500 examples each.
What we learned
Machine learning algorithms take a long time to build when you don’t already have a prelabled dataset. We also learned how simple training and model selection can be with the power of IBM.
What's next for Review Quality Analyzer
We would like to create a user interface and perhaps even test the data with a RNN to evaluate at the possible importance of word positionings. In addition to this, since we had so much success with our model accuracy, we would like to look towards real world implementations of this to provide more incentive for people to leave quality reviews.
Log in or sign up for Devpost to join the conversation.