LocalPulse - Business Insights

LocalPulse helps small shops by scanning Google/Yelp reviews, revealing what customers love or hate so they can fix critical flaws and thrive in this data driven world.

Comment

Inspiration

We built LocalPulse AI because we kept thinking about the same person: the owner of a small restaurant who wakes up at 5 AM, preps the kitchen, serves hundreds of customers, and finally sits down at night — only to find out that a review mentioning "cockroaches near the food station" has been sitting online for three days.

Large chains have dedicated reputation teams and enterprise analytics tools. Small business owners have nothing but a phone notification and not enough hours in the day to read every review. We wanted to close that gap.

The Yelp Academic Dataset — nearly 7 million reviews — made us realize the scale of the problem. With that much signal buried in unstructured text, there had to be a way to surface the critical moments automatically. That conviction became LocalPulse AI.

What it does

LocalPulse AI is an end-to-end Big Data NLP pipeline that ingests raw Yelp reviews, classifies their sentiment, and instantly alerts business owners when something critical — a hygiene complaint, a food safety issue, a pest sighting — slips through in a negative review.

The core flow:

  1. Ingests 6.99M raw Yelp reviews from JSON at scale using Apache Spark
  2. Cleans and processes the text through a full NLP pipeline (tokenization, stopword removal, TF-IDF feature extraction)
  3. Classifies each review as Positive, Neutral, or Negative using a Logistic Regression model trained on PySpark MLlib
  4. Triggers smart alerts through a dual-layer system — a review only raises a 🔴 Critical Alert when both the ML model flags it as negative and a keyword scan detects high-risk language (e.g., rat, food poisoning, mold, raw chicken)
  5. Displays everything on a Streamlit dashboard where any business owner can search by their Business ID and instantly see their sentiment breakdown and active alerts

The result: business owners stop reacting to crises and start preventing them.

How we built it

We split the project across two clearly owned tracks and used production-grade Git workflows — feature branches, pull requests, and peer code reviews — to stay in sync throughout.

Data Engineering Track (Mano Rakshitha)

The pipeline started with raw Yelp JSON at a scale that ruled out pandas immediately. We leaned on PySpark 4.0 running on Google Colab with Java 17 to handle the full 6.99M review dataset in a distributed fashion.

The NLP preprocessing chain was built sequentially:

$$\text{Raw Text} \xrightarrow{\text{Tokenizer}} \text{Tokens} \xrightarrow{\text{StopWordsRemover}} \text{Filtered Tokens} \xrightarrow{\text{TF-IDF}} \text{Feature Vectors}$$

TF-IDF was chosen deliberately over more complex embeddings — at this scale, interpretability and training speed mattered more than marginal accuracy gains from dense representations.

ML Engineering Track (Cashlin)

With feature vectors ready, we trained a Logistic Regression classifier using PySpark MLlib on an 80/20 train-test split:

$$\text{Training Set: } 5{,}591{,}291 \quad \text{Test Set: } 1{,}398{,}117$$

The model outputs a probability score across three classes (Positive, Neutral, Negative). Rather than simply thresholding on the negative class, we layered a keyword detection system on top:

$$\text{Critical Alert} = \mathbb{1}[\hat{y} = \text{Negative}] \;\cap\; \mathbb{1}[\text{keywords} \in \text{review}]$$

This dual-gate approach keeps the false positive rate low — owners only get paged for genuine emergencies, not every mildly unhappy customer.

The Streamlit dashboard was built last, pulling from the processed output to give owners a real-time personalized view.

Challenges we ran into

Memory and environment constraints Running Spark on Google Colab meant constantly fighting memory limits. Processing 6.99M reviews without hitting OOM errors required careful partition tuning and incremental pipeline testing before running on the full dataset.

Class imbalance The dataset was heavily skewed, positive reviews (approx 67%) vastly outnumbered negative ones (approx 23%). Without addressing this, the model would have learned to simply predict "positive" for everything and still look accurate. We had to be deliberate about how we evaluated model performance beyond raw accuracy.

Calibrating the dual-layer alert system The hardest design decision was setting the threshold for what constitutes a "Critical Alert." Too sensitive and owners get alert fatigue; too conservative and real problems slip through. We iterated on the keyword list and the ML confidence threshold together until the numbers felt genuinely actionable:

$$\text{Precision over recall — a missed alert is better than a false alarm.}$$

Collaboration across tracks Keeping the data engineering output format perfectly aligned with what the ML pipeline expected — column names, schema, data types — required tight communication. This is where our PR review process earned its keep.

Accomplishments that we're proud of

  • 86.20% model accuracy on a held-out test set of 1.4M reviews — meaningful performance at a scale most classroom projects never approach
  • Processing the entire 6.99M review dataset end-to-end in a distributed Spark pipeline without downsampling
  • Flagging 11,134 Critical Alerts and 12,487 Warning Alerts — a real signal that the dual-layer system is doing useful work, not just generating noise
  • Shipping a working, interactive Streamlit dashboard that any business owner could actually use
  • Maintaining clean, reviewable Git history throughout — something we're genuinely proud of as first-time collaborators on a project this size

What we learned

Scale changes everything. Techniques that work fine on 10,000 rows fall apart at 7 million. We learned to think in partitions, lazy evaluation, and distributed memory management in ways that tutorials don't prepare you for.

Simple models, well-tuned, go far. Logistic Regression on TF-IDF features achieving 86.2% accuracy on a 3-class sentiment problem is a reminder that you don't always need transformer-based models. Understanding why a simple model works well is more valuable than reaching for complexity by default.

Alert design is a product problem, not just a technical one. The math behind the dual-layer system is straightforward. Deciding what to alert on, and how often, required thinking from the business owner's perspective — a kind of empathy that no amount of model tuning can replace.

Good Git hygiene is a superpower. Working across two tracks with shared data contracts made us appreciate how much a clean branching strategy and consistent PR reviews reduce integration bugs.

What's next for LocalPulse

  • Real-time streaming ingestion via Apache Kafka — moving from batch processing to live review classification as reviews come in
  • Fine-grained topic extraction using LDA or BERTopic to tell owners not just that something is wrong, but what specific aspect (food quality, service, cleanliness) is driving negative sentiment
  • Competitor benchmarking — letting owners see how their sentiment trends compare to nearby businesses in the same category
  • SMS / email alert delivery so critical alerts reach owners directly, not just on a dashboard they have to remember to open
  • Expanding beyond Yelp to Google Reviews and Tripadvisor for broader coverage

Built With

  • alert-systems
  • apache-spark
  • big-data
  • dashboard-development
  • data-engineering
  • data-pipelines
  • data-preprocessing
  • data-visualization
  • distributed-computing
  • feature-extraction
  • git
  • google-colab
  • java
  • json-data-processing
  • keyword-detection
  • logistic-regression
  • machine-learning
  • mllib
  • model-evaluation
  • natural-language-processing
  • plotly
  • pyspark
  • python
  • sentiment-analysis
  • stopwords-removal
  • streamlit
  • text-classification
  • tf-idf
  • tokenization
Share this project:

Updates