Download the data

Download the dataset and, within the handson-unsupervised-learning directory, place the csv file in a folder called /datasets/credit_card_data/. If you downloaded the GitHub repository earlier, you will already have this file in this folder in the repository.

Import the necessary libraries

Import Python libraries that we will need to build our fraud detection solution.

'''Main'''
import numpy as np
import pandas as pd
import os

'''Data Viz'''
import matplotlib.pyplot as plt
import seaborn as sns
color = sns.color_palette()
import matplotlib as mpl

%matplotlib inline

'''Data Prep'''
from sklearn import preprocessing as pp
from scipy.stats import pearsonr
from sklearn.model_selection import train_test_split
from sklearn.model_selection import StratifiedKFold
from sklearn.metrics import log_loss
from sklearn.metrics import precision_recall_curve, average_precision_score
from sklearn.metrics import roc_curve, auc, roc_auc_score
from sklearn.metrics import confusion_matrix, classification_report

'''Algos'''
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
import xgboost as xgb
import lightgbm as lgb

Read the Data

current_path = os.getcwd()
file = '\\datasets\\credit_card_data\\credit_card.csv'
data = pd.read_csv(current_path + file)

Preview the Data

Figure 2-1 displays the first five rows of the dataset. As you can see, the data has been properly loaded.

data.head()

Figure 2-1. Preview of the data

Generate Summary Statistics.

Figure 2-2 describes the data, column by column. Figure 2-3 lists all the column names for easy reference. Figure 2-4 displays the total count of fraudulent transactions.

data.describe()

Figure 2-2. Simple summary statistics

data.columns

Figure 2-3. Columns in the dataset

data['Class'].sum()

Figure 2-4. The total number of positive labels

There are 284,807 instances and 31 columns as expected - 28 numerical features (V1 through V28), Time, Amount, and Class.

The timestamps range from 0 to 172,792, the amounts range from 0 to 25,691.16, and there are 492 fraudulent transactions. These fraudulent transactions are also referred to as positive cases or positive labels (labeled as one); the normal transactions are negative cases or negative labels (labeled as zero).

The 28 numerical features are not standardized yet, but we will standardize the data soon. Standardization rescales the data to have a mean of zero and standard deviation of one.

Identify non-numerical values by feature

Some machine learning algorithms cannot handle non-numerical values or missing values. Therefore, it is best practice to identify non-numerical values (also known as not a number or NaNs).

In the case of missing values, we can impute the value - for example, by replacing the missing points with the mean, median, or mode of the feature - or substitute with some user-defined value.

In the case of categorical values, we can encode the data such that all the categorical values are represented with a sparse matrix. This sparse matrix is then combined with the numerical features. The machine learning algorithm trains on this combined feature set.

Figure 2-5 shows that none of the observations have NaNs, so we will not need to impute or encode any of the values.

nanCounter = np.isnan(data).sum()

Figure 2-5. The number of NaNs by feature

Identify distinct values by feature

To develop a better understanding of the credit card transactions dataset, let’s count the number of distinct values by feature.

Figure 2-6 shoes that we have 124,592 distinct timestamps. But, from earlier, we know that we have 284,807 observations in total. That means that there are multiple transactions at some timestamps.

And, as expected, there are just two classes - one for fraud, zero for not fraud.

distinctCounter = data.apply(lambda x: len(x.unique()))

Figure 2-6. The number of distinct values by feature

Create the feature matrix X and the labels array y

dataX = data.copy().drop([‘Class’],axis=1)
dataY = data[‘Class’].copy()

Standardize the feature matrix X

Let’s rescale the feature matrix so that each feature, except for time, has a mean of zero and standard deviation of one.

featuresToScale = dataX.drop(['Time'],axis=1).columns
sX = pp.StandardScaler(copy=True)
dataX.loc[:,featuresToScale] = sX.fit_transform(dataX[featuresToScale])

As shown in Figure 2-7, the standardized features now have a mean of zero and a standard deviation of one.

Figure 2-7. Summary of scaled features

Check correlation of features

correlationMatrix = pd.DataFrame(data=[],index=dataX.columns,
columns=dataX.columns)
for i in dataX.columns:
    for j in dataX.columns:
        correlationMatrix.loc[i,j] = np.round(pearsonr(dataX.loc[:,i],
         dataX.loc[:,j])[0],2)

Figure 2-8. Correlation matrix

Set hyperparameters

penalty = 'l2'
C = 1.0
class_weight = 'balanced'
random_state = 2018
solver = 'liblinear'

logReg = LogisticRegression(penalty=penalty, C=C,
            class_weight=class_weight, random_state=random_state,
                            solver=solver, n_jobs=n_jobs)

We will set the penalty to the default value L2 instead of L1. Compared to L1, L2 is less sensitive to outliers but will assign non-zero weights to nearly all the features, resulting in a stable solution. L1 will assign high weights to the most important features and near-zero weights to the rest, essentially performing feature selection as the algorithm trains. However, because the weights vary so much feature-to-feature, the L1 solution is not as stable to changes in data points compared to the L2 solution.⁹

C is the regularization strength. As you may recall from chapter one, regularization helps address overfitting by penalizing complexity. In other words, the stronger the regularization, the greater the penalty the machine learning algorithm applies to complexity. Regularization nudges the machine learning algorithm to prefer simpler models to more complex ones, all else equal.

This regularization constant, C, must be a positive floating number. The smaller the value, the stronger the regularization. We will keep the default 1.0.

Our credit card transactions dataset is very imbalanced - out of all the 284,807 cases, only 492 are fraud. As the machine learning algorithm trains, we want the algorithm to focus more attention on learning from the positive labeled transactions - in other words, the fraudulent transactions - because there are so few of them in the dataset.

For this logistic regression model, we will set the class_weight to balanced. This signals to the logistic regression algorithm that we have an imbalanced class problem; the algorithm will need to weigh the positive labels more heavily as it trains. In this case, the weights will be inversely proportional to the class frequencies; the algorithm will assign higher weights to the rare positive labels (i.e., fraud) and lower weights to the more frequent negative labels (i.e., not fraud).

The random state is fixed to 2018 to help others - such as you, the reader - reproduce results. We will keep the default solver liblinear.

Train the model

Now that the hyperparameters are set, we will train the logistic regression model on each of the five k-fold cross validation splits, training on four-fifths of the training set and evaulating the performance on the fifth slice that is held aside.

As we train and evaluate like this five times, we will calculate the cost function - log loss for our credit card transactions problem - for the training (i.e., the four-fifths slice of the original training set) and for the validation (i.e., the one-fifth slice of the original training set). We will also store the predictions for each of the five cross-validation sets; by the end of the fifth run, we will have predictions for the entire training set.

trainingScores = []
cvScores = []
predictionsBasedOnKFolds = pd.DataFrame(data=[],
                                        index=y_train.index,columns=[0,1])

model = logReg

for train_index, cv_index in k_fold.split(np.zeros(len(X_train))
                                          ,y_train.ravel()):
    X_train_fold, X_cv_fold = X_train.iloc[train_index,:], \
        X_train.iloc[cv_index,:]
    y_train_fold, y_cv_fold = y_train.iloc[train_index], \
        y_train.iloc[cv_index]

    model.fit(X_train_fold, y_train_fold)
    loglossTraining = log_loss(y_train_fold,
                               model.predict_proba(X_train_fold)[:,1])
    trainingScores.append(loglossTraining)

    predictionsBasedOnKFolds.loc[X_cv_fold.index,:] = \
        model.predict_proba(X_cv_fold)
    loglossCV = log_loss(y_cv_fold,
                         predictionsBasedOnKFolds.loc[X_cv_fold.index,1])
    cvScores.append(loglossCV)

    print('Training Log Loss: ', loglossTraining)
    print('CV Log Loss: ', loglossCV)

loglossLogisticRegression = log_loss(y_train,
                                     predictionsBasedOnKFolds.loc[:,1])
print('Logistic Regression Log Loss: ', loglossLogisticRegression)

Evaluate the results

For each of the five runs, we have the training log loss and the cross-validation log loss (see Figure 2-11). Generally - but not always - the training log loss will be lower than the cross-validation log loss. Because the machine learning algorithm has learned directly from the training data, its performance (i.e., its log loss) should be better on the training set than on the cross-validation set. Remember that the cross-validation set has the transactions that were explicitly held out from the training exercise.

Figure 2-11. Cross-validation results of logistic regression

For each of the five runs, the training and cross-validation log losses are similar to each other. The logistic regression model does not exhibit severe overfitting; if it did, we would have a low training log loss and comparably high cross-validation log loss.

Since we stored the predictions for each of the five cross validation sets, we can combine the predictions into a single set. This single set is the same as the original training set, and we can now calculate the overall log loss for this entire training set. This is the best estimate for what the logistic regression model’s log loss will be on the test set (see Figure 2-12).

Figure 2-12. Overall log loss of logistic regression

Evaluating the logistic regression model

We have just finished introducing several evaluation metrics, including the confusion matrix, the precision-recall curve, and the receiver operating characteristic.

Let’s use these evaluation metrics to better understand the logistic regression model’s results.

First, let’s plot the precision-recall curve and calculate the average precision.

preds = pd.concat([y_train,predictionsBasedOnKFolds.loc[:,1]], axis=1)
preds.columns = ['trueLabel','prediction']
predictionsBasedOnKFoldsLogisticRegression = preds.copy()

precision, recall, thresholds = precision_recall_curve(preds['trueLabel'],
                                                       preds['prediction'])

average_precision = average_precision_score(preds['trueLabel'],
                                            preds['prediction'])

plt.step(recall, precision, color='k', alpha=0.7, where='post')
plt.fill_between(recall, precision, step='post', alpha=0.3, color='k')

plt.xlabel('Recall')
plt.ylabel('Precision')
plt.ylim([0.0, 1.05])
plt.xlim([0.0, 1.0])

plt.title('Precision-Recall curve: Average Precision = {0:0.2f}'.format(
          average_precision))

Figure 2-14. Precision-recall curve of logistic regression

Figure 2-14 is the plot of the precision-recall curve. Putting together what we discussed earlier, you can see that we can achieve approximately 80% recall (i.e., catch 80% of the fraudulent transactions) with approximately 70% precision (i.e., of the transactions the model flags as fraud, 70% are truly fraud while the remaining 30% were incorrectly flagged as fraud).

We can distill this precision-recall curve into a single number by calculating the average precision, which is 0.73 for this logistic regression model. We cannot yet tell whether this is a good or bad average precision yet since we have no other models to compare our logistic regression against.

Now, let’s measure the area under the receiver operating characteristic.

fpr, tpr, thresholds = roc_curve(preds['trueLabel'],preds['prediction'])

areaUnderROC = auc(fpr, tpr)

plt.figure()
plt.plot(fpr, tpr, color='r', lw=2, label='ROC curve')
plt.plot([0, 1], [0, 1], color='k', lw=2, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic:
          Area under the curve = {0:0.2f}'.format(areaUnderROC))
plt.legend(loc="lower right")
plt.show()

Figure 2-15. Area under the ROC curve of logistic regression

As shown in Figure 2-15, the area under the ROC curve is 0.97. This metric is just another way to evaluate the goodness of the logistic regression model, allowing you to determine how much of the fraud you can catch while keeping the false positive rate as low as possible. As with the average precision, we do not know whether this area under the ROC curve of 0.97 is good or not, but we will know more once we compare the logistic regression model’s area under the ROC curve with those of other models.

Set the hyperparameters

n_estimators = 10
max_features = 'auto'
max_depth = None
min_samples_split = 2
min_samples_leaf = 1
min_weight_fraction_leaf = 0.0
max_leaf_nodes = None
bootstrap = True
oob_score = False
n_jobs = -1
random_state = 2018
class_weight = 'balanced'

RFC = RandomForestClassifier(n_estimators=n_estimators,
        max_features=max_features, max_depth=max_depth,
        min_samples_split=min_samples_split, min_samples_leaf=min_samples_leaf,
        min_weight_fraction_leaf=min_weight_fraction_leaf,
        max_leaf_nodes=max_leaf_nodes, bootstrap=bootstrap,
        oob_score=oob_score, n_jobs=n_jobs, random_state=random_state,
        class_weight=class_weight)

Let’s start with mostly default hyperparameters. The number of estimators is set at 10; in other words, we will build 10 trees and average the results across these 10 trees. For each tree, the model will consider the square root of the total number of features (in this case, the square root of 30 total features, which is 5 features, rounded down).

By setting the max_depth to none, the tree will grow as deep as possible, splitting as much as possible given the subset of features. Similar to what we did for logistic regression, we set the random state to 2018 for reproducibility of results and class weight to balanced given our imbalanced dataset.

Train the model

We will run k-fold cross validation five times, training on four-fifths of the training data and predicting on the fifth slice. We will store the predictions as we go.

trainingScores = []
cvScores = []
predictionsBasedOnKFolds = pd.DataFrame(data=[],
                                        index=y_train.index,columns=[0,1])

model = RFC

for train_index, cv_index in k_fold.split(np.zeros(len(X_train)),
                                          y_train.ravel()):
    X_train_fold, X_cv_fold = X_train.iloc[train_index,:], \
        X_train.iloc[cv_index,:]
    y_train_fold, y_cv_fold = y_train.iloc[train_index], \
        y_train.iloc[cv_index]

    model.fit(X_train_fold, y_train_fold)
    loglossTraining = log_loss(y_train_fold, \
                                model.predict_proba(X_train_fold)[:,1])
    trainingScores.append(loglossTraining)

    predictionsBasedOnKFolds.loc[X_cv_fold.index,:] = \
        model.predict_proba(X_cv_fold)
    loglossCV = log_loss(y_cv_fold, \
        predictionsBasedOnKFolds.loc[X_cv_fold.index,1])
    cvScores.append(loglossCV)

    print('Training Log Loss: ', loglossTraining)
    print('CV Log Loss: ', loglossCV)

loglossRandomForestsClassifier = log_loss(y_train,
                                          predictionsBasedOnKFolds.loc[:,1])
print('Random Forests Log Loss: ', loglossRandomForestsClassifier)

Evaluate the results

Figure 2-16 shows the training and cross-validation log loss results.

Figure 2-16. Cross-validation results of random forests

Notice that the training log losses are considerably lower than the cross-validation log losses, suggesting that the random forests classifier - with the mostly default hyperparameters - is somewhat overfitting the data during the training.

Figure 2-17. Overall log loss of random forests

Figure 2-17 shows the log loss over the entire training set (using the cross-validation predictions). Even though it is somewhat overfitting the training data, the random forests has a validation log loss that is about one-tenth that of the logistic regression - we have a significant improvement over the previous machine learning solution. The random forests model is better at correctly flagging the fraud among credit card transactions.

Figure 2-18. Precision-recall curve of random forests

Figure 2-18 displays the precision-recall curve. As you can see from the curve, the random forests model can catch approximately 80% of all the fraud with approximately 80% precision. This is more impressive than the approximately 80% of all the fraud the logistic regression model caught with 70% precision.

The average precision of 0.79 of the random forests model is a clear improvement over the 0.73 average precision of the logistic regression model.

Figure 2-19. Area under the ROC curve of random forests

However, the area under the curve, shown in Figure 2-19, is somewhat worse - 0.93 for random forests versus 0.97 for logistic regression.

Set the hyperparameters

We will set this up as a binary classification problem and use log loss as the cost function. We will set the max depth of each tree to the default six and use a default learning rate of 0.3. For each tree, we will use all the observations and all the features; these are the default settings. We will set a random state of 2018 to ensure reproducibility of results.

params_xGB = {
    'nthread':16, #number of cores
    'learning rate': 0.3, #range 0 to 1, default 0.3
    'gamma': 0, #range 0 to infinity, default 0
        # increase to reduce complexity (increase bias, reduce variance)
    'max_depth': 6, #range 1 to infinity, default 6
    'min_child_weight': 1, #range 0 to infinity, default 1
    'max_delta_step': 0, #range 0 to infinity, default 0
    'subsample': 1.0, #range 0 to 1, default 1
        # subsample ratio of the training examples
    'colsample_bytree': 1.0, #range 0 to 1, default 1
        # subsample ratio of features
    'objective':'binary:logistic',
    'num_class':1,
    'eval_metric':'logloss',
    'seed':2018,
    'silent':1
}

Train the model

As before, we will use k-fold cross-validation, training on a different four-fifths of the training data and predicting on the fifth slice for a total of five runs.

For each of the five runs, the gradient boosting model will train for as many as 2,000 rounds, evaluating whether the cross-validation log loss is decreasing as it goes. If the cross-validation log loss stops improving (over the previous 200 rounds), the training process will stop to avoid overfitting. The results of training process are very verbose, so we will not print them here but they can be found via the code on GitHub.

trainingScores = []
cvScores = []
predictionsBasedOnKFolds = pd.DataFrame(data=[],
                                    index=y_train.index,columns=['prediction'])

for train_index, cv_index in k_fold.split(np.zeros(len(X_train)),
                                          y_train.ravel()):
    X_train_fold, X_cv_fold = X_train.iloc[train_index,:], \
        X_train.iloc[cv_index,:]
    y_train_fold, y_cv_fold = y_train.iloc[train_index], \
        y_train.iloc[cv_index]

    dtrain = xgb.DMatrix(data=X_train_fold, label=y_train_fold)
    dCV = xgb.DMatrix(data=X_cv_fold)

    bst = xgb.cv(params_xGB, dtrain, num_boost_round=2000,
                 nfold=5, early_stopping_rounds=200, verbose_eval=50)

    best_rounds = np.argmin(bst['test-logloss-mean'])
    bst = xgb.train(params_xGB, dtrain, best_rounds)

    loglossTraining = log_loss(y_train_fold, bst.predict(dtrain))
    trainingScores.append(loglossTraining)

    predictionsBasedOnKFolds.loc[X_cv_fold.index,'prediction'] = \
        bst.predict(dCV)
    loglossCV = log_loss(y_cv_fold, \
        predictionsBasedOnKFolds.loc[X_cv_fold.index,'prediction'])
    cvScores.append(loglossCV)

    print('Training Log Loss: ', loglossTraining)
    print('CV Log Loss: ', loglossCV)

loglossXGBoostGradientBoosting = \
    log_loss(y_train, predictionsBasedOnKFolds.loc[:,'prediction'])
print('XGBoost Gradient Boosting Log Loss: ', loglossXGBoostGradientBoosting)

Evaluate the results

Figure 2-20. Overall log loss of XGBoost gradient boosting

As shown in Figure 2-20, the log loss over the entire training set (using the cross-validation predictions) is one-fifth that of the random forests and one-fiftieth that of logistic regression, representing a substantial improvement over the previous two models.

Figure 2-21. Precision-recall curve of XGBoost gradient boosting

As shown in Figure 2-21, the average precision is 0.82, just shy of that of random forests (0.79) and considerably better than that of logistic regression (0.73).

Figure 2-22. Area under the ROC curve of XGBoost gradient boosting

As shown in Figure 2-22, the area under the receiving operating characteristic curve is 0.97, same as that of logistic regression (0.97) and an improvement over random forests (0.93).

So far, gradient boosting is the best of the three models based on the log loss, the precision-recall curve, and the area under the receiving operating characteristic.

Set the hyperparameters

We will set this up as a binary classification problem and use log loss as the cost function. We will set the max depth of each tree to 4 and use a learning rate of 0.1. For each tree, we will use all the samples and all the features; these are the default settings. We will use the default number of leaves for one tree (31) and set a random state to ensure reproducibility of results.

params_lightGB = {
    'task': 'train',
    'application':'binary',
    'num_class':1,
    'boosting': 'gbdt',
    'objective': 'binary',
    'metric': 'binary_logloss',
    'metric_freq':50,
    'is_training_metric':False,
    'max_depth':4,
    'num_leaves': 31,
    'learning_rate': 0.01,
    'feature_fraction': 1.0,
    'bagging_fraction': 1.0,
    'bagging_freq': 0,
    'bagging_seed': 2018,
    'verbose': 0,
    'num_threads':16
}

Train the model

As before, we will use k-fold cross validation and cycle through this five times, storing the predictions on the validation sets as we go.

trainingScores = []
cvScores = []
predictionsBasedOnKFolds = pd.DataFrame(data=[],
                                index=y_train.index,columns=['prediction'])

for train_index, cv_index in k_fold.split(np.zeros(len(X_train)),
                                          y_train.ravel()):
    X_train_fold, X_cv_fold = X_train.iloc[train_index,:], \
        X_train.iloc[cv_index,:]
    y_train_fold, y_cv_fold = y_train.iloc[train_index], \
        y_train.iloc[cv_index]

    lgb_train = lgb.Dataset(X_train_fold, y_train_fold)
    lgb_eval = lgb.Dataset(X_cv_fold, y_cv_fold, reference=lgb_train)
    gbm = lgb.train(params_lightGB, lgb_train, num_boost_round=2000,
                   valid_sets=lgb_eval, early_stopping_rounds=200)

    loglossTraining = log_loss(y_train_fold, \
                gbm.predict(X_train_fold, num_iteration=gbm.best_iteration))
    trainingScores.append(loglossTraining)

    predictionsBasedOnKFolds.loc[X_cv_fold.index,'prediction'] = \
        gbm.predict(X_cv_fold, num_iteration=gbm.best_iteration)
    loglossCV = log_loss(y_cv_fold, \
        predictionsBasedOnKFolds.loc[X_cv_fold.index,'prediction'])
    cvScores.append(loglossCV)

    print('Training Log Loss: ', loglossTraining)
    print('CV Log Loss: ', loglossCV)

loglossLightGBMGradientBoosting = \
    log_loss(y_train, predictionsBasedOnKFolds.loc[:,'prediction'])
print('LightGBM Gradient Boosting Log Loss: ', loglossLightGBMGradientBoosting)

For each of the five runs, the gradient boosting model will train for as many as 2,000 rounds, evaluating whether the cross-validation log loss is decreasing as it goes. If the cross-validation log loss stops improving (over the previous 200 rounds), the training process will stop to avoid overfitting. The results of training process are very verbose, so we will not print them here but can be found via the code on GitHub.

Evaluate the results

Figure 2-23. Overall log loss of LightGBM gradient boosting

As shown in Figure 2-23, the log loss over the entire training set (using the cross-validation predictions) is similar to that of XGBoost, one-fifth that of the random forests, and one-fiftieth that of logistic regression. But, compared to XGBoost, LightGBM is considerably faster.

Figure 2-24. Precision-recall curve of LightGBM gradient boosting

As shown in Figure 2-24, the average precision is 0.82, the same as that of XGboost (0.82), better than that of random forests (0.79), and considerably better than that of logistic regression (0.73).

Figure 2-25. Area under the ROC curve of LightGBM gradient boosting

As shown in Figure 2-25, the area under the receiving operating characteristic curve is 0.98, an improvement over that of XGBoost (0.97), logistic regression (0.97), and random forests (0.93).

Combine layer one predictions with the original training dataset

First, let’s combine the predictions from each of the four machine learning models that we have built with the original training dataset.

predictionsBasedOnKFoldsFourModels = pd.DataFrame(data=[],index=y_train.index)
predictionsBasedOnKFoldsFourModels = predictionsBasedOnKFoldsFourModels.join(
    predictionsBasedOnKFoldsLogisticRegression['prediction'].astype(float), \
    how='left').join(predictionsBasedOnKFoldsRandomForests['prediction'] \
    .astype(float),how='left',rsuffix="2").join( \
    predictionsBasedOnKFoldsXGBoostGradientBoosting['prediction'].astype(float), \
    how='left',rsuffix="3").join( \
    predictionsBasedOnKFoldsLightGBMGradientBoosting['prediction'].astype(float), \
    how='left',rsuffix="4")
predictionsBasedOnKFoldsFourModels.columns = \
    ['predsLR','predsRF','predsXGB','predsLightGBM']

X_trainWithPredictions = \
    X_train.merge(predictionsBasedOnKFoldsFourModels,
                  left_index=True,right_index=True)

Set the hyperparameters

Now, we will use LightGBM Gradient Boosting - the best machine learning algorithm from the earlier exercise - to train on this original features plus layer one predictions dataset. The hyperparameters will remain the same as before.

params_lightGB = {
    'task': 'train',
    'application':'binary',
    'num_class':1,
    'boosting': 'gbdt',
    'objective': 'binary',
    'metric': 'binary_logloss',
    'metric_freq':50,
    'is_training_metric':False,
    'max_depth':4,
    'num_leaves': 31,
    'learning_rate': 0.01,
    'feature_fraction': 1.0,
    'bagging_fraction': 1.0,
    'bagging_freq': 0,
    'bagging_seed': 2018,
    'verbose': 0,
    'num_threads':16
}

Train the model

As before, we will use k-fold cross validation and generate fraud probabilities for the five different cross validation sets.

trainingScores = []
cvScores = []
predictionsBasedOnKFoldsEnsemble = \
    pd.DataFrame(data=[],index=y_train.index,columns=['prediction'])

for train_index, cv_index in k_fold.split(np.zeros(len(X_train)), \
                                          y_train.ravel()):
    X_train_fold, X_cv_fold = \
        X_trainWithPredictions.iloc[train_index,:], \
        X_trainWithPredictions.iloc[cv_index,:]
    y_train_fold, y_cv_fold = y_train.iloc[train_index], y_train.iloc[cv_index]

    lgb_train = lgb.Dataset(X_train_fold, y_train_fold)
    lgb_eval = lgb.Dataset(X_cv_fold, y_cv_fold, reference=lgb_train)
    gbm = lgb.train(params_lightGB, lgb_train, num_boost_round=2000,
                   valid_sets=lgb_eval, early_stopping_rounds=200)

    loglossTraining = log_loss(y_train_fold, \
        gbm.predict(X_train_fold, num_iteration=gbm.best_iteration))
    trainingScores.append(loglossTraining)

    predictionsBasedOnKFoldsEnsemble.loc[X_cv_fold.index,'prediction'] = \
        gbm.predict(X_cv_fold, num_iteration=gbm.best_iteration)
    loglossCV = log_loss(y_cv_fold, \
        predictionsBasedOnKFoldsEnsemble.loc[X_cv_fold.index,'prediction'])
    cvScores.append(loglossCV)

    print('Training Log Loss: ', loglossTraining)
    print('CV Log Loss: ', loglossCV)

loglossEnsemble = log_loss(y_train, \
        predictionsBasedOnKFoldsEnsemble.loc[:,'prediction'])
print('Ensemble Log Loss: ', loglossEnsemble)

Evaluate the results

As shown in Figure 2-35, the ensemble log loss is very similar to the standalone gradient boosting log loss - we do not see an improvement. Since the best standalone solutions are from the same family (gradient boosting), we do not see an improvement in results since they have highly correlated strengths and weaknesses in detecting fraud. There is no benefit from diversifying across models because the best models are basically the same.

Figure 2-35. Overall log loss of the ensemble

As shown in figures 2-36 and 2-37, the precision-recall curve, the average precision, and the area under the receiver operating characteristic also corroborate the lack of improvement.

Figure 2-36. Precision-recall curve of the ensemble

Figure 2-37. Area under the ROC curve of the ensemble