brandpluse

BrandPulse ingests real-time social trends into Delta Lake, finds competitor content gaps with Spark ML, writes Instagram captions with Mosaic AI Llama 3, and generates a live 6-second video

Comment

Inspiration

What it does

Meet Priya.

Priya is a senior brand manager at Nykaa. Every Monday morning she opens 14 browser tabs — Instagram, X, Google Trends, three competitor feeds, two agency Slack channels — and spends three hours manually figuring out which hashtag is trending, whether Mamaearth just posted something Nykaa should respond to, and what the creative brief for this week's reel should say.

By Wednesday, the brief is ready. By Friday, it's reviewed. The video shoots the following week. By the time Nykaa's content goes live, the trend has peaked, the competitor has already captured the conversation, and Priya is back at her 14 tabs starting over.

BrandPulse gives Priya her Monday mornings back. The moment a trend spikes, the pipeline fires, a competitor gap is detected, three campaign options are drafted by Llama 3, and — on Priya's single click of Approve — a real Instagram-ready video reel is generated and logged. Total time from trend to video: under 10 minutes.

Architecture

Social APIs 
        │
        ▼
[Auto Loader — Spark Structured Streaming]
        │  JSON batches land in BRONZE_PATH
        ▼
Bronze Delta Table  ←  @dlt.expect quality gates
(raw_social_posts)
        │
        ▼
[DLT Pipeline — Silver transform]
  • explode hashtags       • engagement_rate
  • sentiment UDF          • hour_of_day
        │
        ▼
Silver Delta Table
(cleaned_trend_metrics)
        │
        ▼
[DLT Pipeline — Gold aggregation]
  • competitor_dominance   • brand_fit_score
  • opportunity_gap        • trend_velocity
        │
        ▼
Gold Delta Table
(brand_insights)
        │
        ├──────────────────────────────────────────┐
        ▼                                          ▼
[Spark ML Job]                          [Mosaic AI — Llama 3]
 TF-IDF + KMeans                         Reads Gold + Gaps
 Competitor gap clusters                 Generates 3 captions
 MLflow tracked                          + image prompts
        │                                          │
        ▼                                          ▼
Gaps Delta Table              Suggestions Delta Table
(competitor_gaps)             (content_suggestions)
                                         │
                                         ▼
                              [Databricks App — Dash]
                               Human-in-the-loop UI
                               Approve / Tweak / Reject
                                         │
                                   (on Approve)
                                         ▼
                              [fal.ai Veo3 API]
                               6-second video reel
                               generated async
                                         │
                                         ▼
                              Video URL → Delta Table
                                         │
                                         ▼
                              App shows live video player

Databricks Features — Why Each One Is Essential

Databricks Feature Why it's essential (not just what it does)
Delta Lake Every table — Bronze, Silver, Gold, Gaps, Suggestions — uses Delta format. ACID transactions mean the App and the ML job read consistent snapshots simultaneously without locking. Time travel lets judges replay any historical state.
Delta Live Tables (DLT) Declarative Bronze → Silver → Gold pipeline with @dlt.expect data quality enforcement. DLT handles incremental processing, retries, and lineage automatically — no custom orchestration code needed.
Spark Structured Streaming Auto Loader (cloudFiles) ingests JSON batches from object storage with exactly-once semantics. read_stream propagates micro-batch processing end-to-end from Bronze through Silver so every new social post flows through enrichment in real time.
Spark ML KMeans + TF-IDF pipeline runs natively on Delta DataFrames — no data movement, no serialisation overhead. Scales to millions of posts without code changes. The Pipeline API chains all transforms into a single reusable, MLflow-loggable artifact.
MLflow Every ML run, every Mosaic AI call, every human approval and rejection is tracked. Judges can browse the Experiment UI and see the full decision trail — making BrandPulse auditable and reproducible, not a black box.
Mosaic AI Model Serving Llama 3 runs inside the Databricks security perimeter — brand data never leaves the workspace. The endpoint auto-scales to zero between demo runs, costs nothing idle, and is governed by workspace IAM.
Databricks Apps (Dash) The UI runs inside Databricks — it calls spark.sql() directly on Delta tables without an external backend, API layer, or egress cost. Workspace IAM governs who can approve content.
Unity Catalog All Delta tables are registered in the brandpulse database, making them discoverable, governable, and shareable across the workspace without path management.

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for brandpluse

Built With

Share this project:

Updates

posted an update —

Maintain a log of how BrandPluse evolves over time. Share updates about new features, app store releases, screenshots, or even code snippets.

Your followers can stay up to date by seeing these updates directly in their feeds, and they can engage with your posts by leaving comments and feedback.

This creates a transparent timeline of development while building a community around the product.

Log in or sign up for Devpost to join the conversation.