toktok loklok

Inspiration

Online location reviews are often noisy, containing spam, irrelevant chatter, or promotional content that misleads users and businesses. We wanted to build a system that filters the noise and highlights trustworthy reviews, empowering people to make informed decisions about where to eat, visit, or seek services. Our inspiration came from noticing how platforms like Google Maps often suffer from fake or irrelevant reviews, which can undermine user trust and local businesses’ reputations.

What We Learned

Through this project, we learned how to combine rule-based heuristics with machine learning and deep learning to design a layered, interpretable pipeline. We also gained experience in:

Review scraping and cleaning at scale (handling multilingual text, emojis, duplicates, spam).
Feature engineering for readability, sentiment, grammar, and image quality.
Semantic embeddings with transformers for text-image and category relevance.
Contrastive learning for building robust multi-task models that predict both review quality and relevance.
Managing API-based LLM calls and designing fault-tolerant batch pipelines.

How We Built It

Our system integrates several components:

Data Collection

Scraped Google Maps reviews across 50+ locations and multiple categories.
Augmented with Kaggle datasets and synthetic irrelevant reviews generated via LLMs.

Feature Engineering

Text features: sentiment, readability, grammar, review length.
Image features: sharpness, text-image relevance, advertisement detection (QR codes, coupons).
Category features: LLM-generated keywords + semantic similarity.

Modeling Pipeline

Quality prediction:
- Rule-based spam/advertising detection.
- Random Forest classifier on engineered features.
- Dual-head contrastive neural network predicting both quality (low/medium/high) and relevance.
- Final decision via rule override + probability ensembling.
Relevance prediction:
- Logistic Regression on category and image relevance.
- Dual-head neural network relevance head.
- Final decision via ensemble averaging.

The final quality label is given by:

$$ \text{FinalQuality} = \begin{cases} 0 & \text{if Rule-based = 0 (spam/low quality)} \ \arg\max \big( \tfrac{1}{2}(P_{\text{RF}} + P_{\text{Dual}}) \big) & \text{otherwise} \end{cases} $$

The final relevance label is given by:

$$ \text{FinalRelevance} = \mathbb{1}\left(\tfrac{1}{2}(P_{\text{LogReg}} + P_{\text{Dual}}) > 0.5\right) $$

where \$\mathbb{1}\$ is the indicator function.

Evaluation

Benchmarked against baselines: TF-IDF + Logistic Regression and Frozen Embeddings + Logistic Regression.
Our dual-head ensemble models consistently outperformed the baselines in both quality and relevance classification.

Challenges We Faced

Data imbalance: Genuine reviews vastly outnumber spam/irrelevant ones, requiring careful sampling and weighted losses.
Image-text alignment: Ensuring CLIP captured meaningful relevance between noisy review images and text.
Integration complexity: Combining rule-based heuristics, tree-based models, and neural networks into a seamless, interpretable pipeline.
Time constraints: Training deep models and running inference at scale was resource-intensive; batching and partial pipelines saved us.

Despite these challenges, we built a robust and interpretable ML system that filters out noise and highlights trustworthy reviews.