Project Story

Inspiration

The inspiration for this project arose from the growing need for trustworthy and relevant user-generated content on platforms like Google Reviews. As online reviews become an essential decision-making tool for consumers, the prevalence of fake reviews, spam, and irrelevant content has undermined their credibility. The goal of this project is to develop a machine learning system that can automatically classify and filter Google reviews based on truthfulness, relevance, and authenticity, helping businesses maintain a transparent and reliable online reputation.

What it does

This machine learning system classifies Google reviews into distinct categories: 'truthful', 'ads_spam', 'irrelevant', and 'rant'. By analyzing the textual content of reviews, combined with metadata (such as the reviewer’s history and sentiment), the system detects and filters out reviews that are potentially deceptive, irrelevant, or promotional in nature. It helps users make informed decisions by surfacing only the most trustworthy reviews. Additionally, the model flags low-confidence predictions for further human validation, ensuring that no relevant content is missed.

How we built it

The machine learning system was built using a multi-step pipeline that combines fine-tuned transformer models and heuristics for classifying Google reviews. The pipeline involves the following stages:

DistilBERT Model Trained on Hotel Data:
We first fine-tuned a pre-trained DistilBERT model using a labeled dataset of hotel reviews. The dataset contained binary labels: 'truthful' and 'deceptive'. The model learned to classify reviews based on the credibility of the content, focusing on identifying deceptive reviews, which often contain misleading or exaggerated claims.
DistilBERT Model Trained on Fake Product Reviews:
The second phase involved training another DistilBERT model on product review data, with binary labels indicating whether the review was 'computer-generated' or 'original'. This model aimed to detect automated or synthetic reviews, which are commonly used for spamming or misleading consumers. The model was trained to distinguish between human-authored and machine-generated text based on linguistic patterns.
Heuristic Rules for California Reviews:
To further improve the model's ability to classify reviews accurately, we integrated domain-specific heuristics, particularly for reviews originating from California. These heuristics focus on detecting spammy or promotional language (e.g., excessive use of URLs or specific keywords like "sale", "discount", or "offer"). Reviews that pass through the heuristic filtering system are then passed into the AutoModelForSequenceClassification class, which loads the appropriate fine-tuned DistilBERT model (depending on whether the review is from a hotel or a product) for final classification. The model then categorizes the review into predefined categories: 'truthful', 'spam', 'irrelevant', and 'rant'.
Integration into a Unified Pipeline:
The three components—two fine-tuned DistilBERT models and the heuristic filtering—are integrated into a unified pipeline. The pipeline starts with heuristic filtering to catch easy-to-detect spam or irrelevant content. If the review passes the filtering, it is classified using the appropriate fine-tuned DistilBERT model, depending on the source data (hotel, product, or general review). Finally, reviews that do not meet the criteria for classification are flagged for further review, allowing for manual intervention if needed.

This multi-model approach, combining fine-tuned transformers with rule-based filtering, ensures that the system can effectively classify a wide range of reviews, including those with subtle nuances or disguised deceptive content.

Challenges we ran into

Class Imbalance: Some classes, especially 'spam' and 'rant', were not represented, leading to performance degradation. We tackled this issue by using heuristic methods to identify between ads/spam, irrelevant and non-visit rants.
Subtle Context and Nuances: Many reviews contained complex, nuanced language (e.g., sarcasm, indirect hints), making it difficult for the model to classify them accurately. For example, a review might seem negative but be based on a valid experience, which required us to refine the model's understanding of sentiment via sentiment analysis.
Scalability: As the number of reviews grew, processing them at scale became a challenge. We tackled this by optimizing our pipeline for batch processing and parallelization, improving the speed and efficiency of review classification without sacrificing accuracy.

Accomplishments that we're proud of

High Classification Accuracy: After several iterations and optimizations, we were able to achieve a classification accuracy of 91% in distinguishing truthful reviews from deceptive ones, surpassing our initial targets.
Real-Time Filtering: We built an efficient, real-time review classification system that can handle a high volume of incoming reviews, flagging spam and irrelevant content while maintaining high accuracy.
Bias Reduction: Through careful attention to data diversity and class balancing, we successfully reduced model biases related to specific reviewer demographics or business sectors, ensuring that the system works fairly across different types of businesses and users.

What we learned

Importance of Data Quality: The quality of labeled data is paramount in training effective models. We learned that even small errors in labeling can significantly impact model performance, especially in tasks like detecting deceptive content. We refined our labeling process to ensure higher consistency.
Data Augmentation: We found that augmenting training data using techniques like pseudo-labelling (labeling unlabeled data based on the model’s predictions) helped us expand the dataset and improve model generalization.
Iterative Improvement: Building a machine learning system is an iterative process. Regular evaluation and fine-tuning allowed us to adapt the model to new data and edge cases, improving its robustness over time.

What's next for PonyTune

Dataset Expansion: We plan to expand the dataset with additional labeled reviews from diverse business sectors (e.g., food, healthcare, retail), enabling the model to generalize better across different domains.
Model Refinement: We will continue improving the model by integrating more advanced techniques such as attention mechanisms, transfer learning, and active learning to better understand complex review patterns and handle ambiguity.
User Feedback Integration: We aim to integrate a user feedback mechanism where users can flag misclassified reviews, which will then be used to retrain the model and refine its accuracy.
Business Integration: We’re working on integrating this system with businesses’ review management platforms, providing real-time insights into the quality and authenticity of customer feedback.