Marketing Strategy A/B Testing + Conversion Prediction

This project analyzes marketing effectiveness through A/B Testing (PSAs & Paid Ad strategy); PLUS Logistic Regression model to Predict customer conversion probability based on user behavior.

Comment

Live App

📈 Marketing Strategy A/B Testing + Conversion Prediction Using Machine Learning

Analyzes marketing effectiveness through A/B Testing between (PSAs) and new Paid Ad strategy; PLUS Logistic Regression model to Predict customer conversion probability

Python Streamlit

📌 Inspiration

  • Marketing budgets are often wasted on ineffective campaigns due to a lack of data-driven decision-making.
  • This project was inspired by the need to combine statistical A/B Testing with machine learning to not just measure campaign impact, but also predict future customer conversions.
  • The goal was to give businesses a scalable, reliable, and actionable tool to improve ad spending efficiency and maximize ROI.

🛠️ What it does

  • Performs A/B Testing to measure the impact of Public Service Announcements (PSAs) vs Paid Ad strategy.
  • Uses statistical significance testing to validate results.
  • Builds a Logistic Regression model to predict the probability of customer conversion based on behavior data.
  • Provides interactive Streamlit dashboards to explore results in real time.
  • Generates business impact insights such as revenue estimation, recall/precision trade-offs, and campaign profitability.

🧱 How I built it

1.Data Preparation & Exploration:

  • Cleaned and analyzed marketing campaign dataset (Marketing_AB_Testing.csv) using Pandas and Seaborn.
  • Identified conversion rate patterns by hour and day.

2.A/B Testing:

  • Conducted hypothesis testing (p-value & statistical significance checks) to compare PSA vs Paid Ad performance.

3.Machine Learning Model:

  • Built and trained a Logistic Regression model using Scikit-learn to predict customer conversion.
  • Saved the model (model.pkl) and input schema (input_columns.pkl) for deployment.

4.Visualization & Evaluation:

  • Created confusion matrix, ROC curve, and classification report to evaluate model performance.
  • Calculated key metrics: Accuracy (86%), AUC (0.85), Recall (69%), Precision (12%).

5.Streamlit Deployment:

  • Developed an interactive dashboard (streamlit_app.py) for real-time predictions and insights.
  • Deployed to Streamlit Cloud for public access.

🧗‍♀️ Challenges I ran into

  • Balancing recall vs precision for business relevance — high recall was prioritized to ensure valuable converters were not missed.
  • Handling class imbalance in conversion data.
  • Translating statistical A/B results into business-friendly insights that decision-makers can act on.
  • Ensuring seamless integration between the A/B Testing component and the ML prediction pipeline in one interactive app.

🏆 Accomplishments that I'm proud of

  • Built an end-to-end analytics + ML solution that merges statistical testing with predictive modeling.
  • Achieved 86% accuracy and 69% recall for converters, capturing over $150K in revenue in the test sample.
  • Designed a professional-grade Streamlit dashboard for non-technical business users.
  • Demonstrated that Logistic Regression can outperform more complex models like Random Forest in profitability and stability.

📚 What I learned

  • How to integrate A/B Testing and ML models into a unified decision-support tool.
  • The importance of high recall in marketing scenarios to maximize revenue capture.
  • Deploying data science projects to Streamlit Cloud for interactive business use.
  • Communicating technical findings in a clear, ROI-focused way for stakeholders.

🚀 What's next for Marketing Strategy A/B Testing + Conversion Prediction

  • Add more ML models (e.g., Gradient Boosting, XGBoost) for comparison and ensemble predictions.
  • Introduce real-time data streaming for live campaign monitoring.
  • Expand dashboard to include cost-per-acquisition (CPA) and lifetime value (LTV) metrics.
  • Deploy as an internal marketing intelligence tool for enterprises to optimize campaigns on the fly.

👩‍💼 About the Author

Sweety Seelam | Business Analyst | Aspiring Data Scientist | Passionate about building end-to-end ML solutions for real-world problems
Email: [email protected]
LinkedIn
GitHub
Medium
My Portfolio

🔐 Proprietary & All Rights Reserved

© 2025 Sweety Seelam. All rights reserved.
This project, including its source code, trained models, datasets (where applicable), visuals, and dashboard assets, is protected under copyright and made available for educational and demonstrative purposes only.
Unauthorized commercial use, redistribution, or duplication of any part of this project is strictly prohibited.

Built With

Share this project:

Updates