Introduction

Our project started by getting a feel of the provided dataset --in this case, policy purchase and relevant features.

Preprocessing

We deepened our data preprocessing skills, particularly in dealing with missing data and erroneous data types. For the missing data, one way to deal with them was just to drop them directly, but that would lead to the loss of potential features for the model to learn from. Eventually, we imputed the missing data instead with median and mode values depending on the data type. As for the data types, we also had "fun" trying to cast them to the appropriate ones for the model to process.

Next, we also applied feature engineering to make our input suitable for machine learning models. Notably, we used one-hot encoding to convert categorical variables into numerical representations. This is crucial for the model to be able to learn from both categorical and numerical features without bias from either. We also scaled the numerical data to prevent certain features from impeding the model performance, such as certain date elements.

Imbalanced dataset

Initially, we considered using SMOTE as it helps to create synthetic data around the minority class to balance the dataset for training. However, we realized that SMOTE alone was causing our model to underperform due to overfitting. That's when we learned about combining both oversampling and undersampling techniques to deal with imbalanced datasets. Here, we used ENN as it filters out noisy and irrelevant samples from the dataset. Combining these two methods led to an improved balancing of the data.

Model Selection and Evaluation

Thereafter, we worked on building our model to predict customer's likelihood of policy purchase. Before we arrived at our final model, we also trained a Random Forest Classifier and a Logistic Regression with PCA pre-processing for dimensionality reduction, as well as two Neural Networks, one with weighted inputs, and the other using SMOTE-ENN-balanced inputs.

We evaluated the effectiveness of our models using both the AUC-ROC score and the accuracy score, as accuracy would help us define the success of the model on the eventual problem, while the AUC-ROC score also helps us to keep check the false positive rates as well, and a more robust metrics against an imbalanced dataset.

These metrics, especially accuracy, will help us to measure correct predictions of the customers who would and would not buy the policies, thus helping Singlife to dive deep into the reasons why customers are less likely to buy their products and better refine their marketing strategies.

Results

The results for our models are as follows:

  1. [Chosen model] Neural Network (SMOTE-ENN): Accuracy: 0.9338, AUC-ROC Score: 0.7497

  2. PCA -> RandomForestClassifier: Accuracy = 0.9513, AUC-ROC Score = 0.6832

  3. PCA -> Logistic Regression: Accuracy: 0.6850, AUC-ROC Score: 0.6505

  4. Neural Network (Weighted): Accuracy: 0.8849, AUC-ROC Score: 0.6917

Based on both metrics, the Neural Network using SMOTE-ENN performed the best and would be the most effective model for this dataset.

Built With

Share this project:

Updates

Submission history