Zero to Agile Data Science

Hi everyone! Welcome to the Credit Card Fraud Detection business problem.

Assume you work for a financial institution that issues credit cards to customers. Once the credit card is in the hands of the customers and being used regularly, it is important that the customer is alerted, and the card blocked upon the detection of fraudulent transactions. We will be analyzing anonymized data from a Kaggle competition that simulates credit card transactions and your job as a Data Scientist is to build a binary classifier to predict fraudulent transactions.

Now what does the dataset contain?

The dataset is available for download as a csv file. The content section of the Kaggle site says that,

"The datasets contains transactions made by credit cards in September 2013 by European cardholders.

This dataset presents transactions that occurred in two days, where we have 492 frauds out of 284,807 transactions. The dataset is highly imbalanced, the positive class (frauds) account for 0.172% of all transactions.

It contains only numerical input variables which are the result of a PCA transformation. Unfortunately, due to confidentiality issues, we cannot provide the original features and more background information about the data. Feature 'Class' is the response variable, and it takes value 1 in case of fraud and 0 otherwise."

There are 30 features and 1 target variable in our dataset. I have deliberately chosen this dataset to keep the initial problem space simple (no dirty features, no categorical variables to be encoded, no nulls, etc.) and also to illustrate that it is possible to get good results from the very first iteration and any further complexity may not improve the performance.

Let us get started with the first iteration for detecting Credit Card Fraud.

You will find this notebook within the notebooks subfolder inside the Predicting-Credit-Card-Fraud folder.

To give a brief overview, we will load the data, check it out briefly and build a baseline model using sklearn’s RandomForest Classifier.

Here is the table of contents corresponding to the above plan

Sections 1,2 and 3 are about introducing the problem, loading the packages, setting the file names and loading the csv file as a dataframe. Sections 4, 5 and 6 are where the core of the analysis takes place. Section 4 will be where we will be doing some preliminary EDA to get a feel for the data and also to make sure there is nothing that we need to clean out. Section 5 will be the optional data cleaning phase in case we identify in the previous phase whether we need to clean it. Section 6 is where we will be getting a baseline binary classification model and comparing with the naïve predictions. Section 7 is where we can wrap up our analysis and Section 8 is where we have the reference articles for further reading.

So, lets begin!

In the Intro section, I have repeated the problem description. This is a good practice in case you want to share the notebooks with other colleagues to check

In the next code cell, I have imported the minimum number of libraries to get started. This not only reduces the size of your code base but also allows you to scrutinize what is needed and what is not needed. Notice how I used sys.path.append("/home/paperspace/Zero to Agile DS"): this is for bringing my utils function to the namespace. This utils function contains a lot of useful functions that I have put together over the years.

After the import, I setup all the global params like the filepaths, filenames, and I have defined the target variable that we are trying to predict as ‘Fraud’. This will make our life easier later in this notebook when are labelling the graphs.

In the next code cell, I am checking my data/raw folder to make sure that the creditcard.csv file is indeed present

Now that we have all the metadata in place, we can begin the analysis.

Import the csv files as a pandas dataframe and then check out random samples from the dataset along with the head and tail (this is to ensure there is nothing unusual at the top or bottom of the input files like stray headers).

In this dataset, there are not anything unusual in the top or the bottom of the csv file.

Data Insights

Data Structure

This is a fairly standard looking dataframe with nothing untowards at first glance. Given that there are a lot of tutorials out there that do EDA, I am going to skip that step and only do very basic probing of the data.

The following code snippet allows you to see the number of rows and columns along with the datatypes for each column header.

Luckily for us, all the columns have numbers (either floats or ints) which reduces the number of steps before we get a baseline performance. Visually inspecting the outputs also suggests that there are no columns where there are nulls. We will confirm this programmatically further along the analysis.

Summary stats

You can get the summary stats for the numerical columns using the describe() method to get a feel of the distribution. Note that some of these will be less meaningful if the underlying values are discrete. For example, the target class is a binary variable, so the mean of 0.017 should be interpreted differently than if it were a continuous variable. Since these are anonymized columns, we are not going to spend time on analyzing the implications

Unique Value Checking

You can check out if certain columns have a lot fewer values compared to the overall number of rows in the dataframe. It seems all the PCA features have nearly the same number of values as the number of rows.

Identify ‘bad’ columns

Next, I am going to find out if there are features that need any data cleansing. I am going to use a custom function that I previously defined in the helper_functions_comprehensive.py file

If you would like to understand the source code in more detail, I encourage you to check out the relevant video

This function has subroutines that allow the easy identification of those columns with nulls, blank spaces, constants and duplicates.

As seen from the data output, the input data doesnt have any features that we need to worry about — no null value removal or imputation, data type conversions, etc.

Data Cleansing

This section is useful when the previous step generated lists that need to be removed. But since we have an extremely clean dataset, we are going to skip this section. Otherwise, you will need to reduce the dataset by removing the list of columns identified in the previous step. Then there might be situations where the original data types identified by pandas is incorrect and we might need to change from string to float. But we don’t have that scenario either. Lastly, we don’t have any categorical columns and therefore, the encoding step is also not necessary.

It’s a good practice to export the dataset that will get utilized for modelling, so that you have a track record of what got used for the results in case your colleagues want to replicate the modelling without having to go through the ETL steps. I prefer to use the python library, pickle, to make any objects in the RAM go to the hard drive for storage.

Data Prep

Before we build a predictive model, lets prepare our dataframe so that we are not doing any conceptual mistakes.

Do we need to do any normalization or scaling of numerical variables?

No because in this series of notebooks, I will be only using decision-tree based models which don’t require the use of any scaling or normalization. So we are going to skip this section

In the next 2 code cells, I am going to separate out the feature and the target variables followed by the train test split. This is standard procedure in Machine Learning, nothing special here except that I have taken care to stratify our splits.

I have set the random_state so that the results are repeatable. Also, even for other classification datasets, its good practice during train test split to stratify according to the class distributions. It becomes even more important for heavily imbalanced datasets like the one above. Note how the last parameter is stratify = y. This param indicates to the train test method that ‘during the splitting, please ensure the class ratio stays the same in the train and test folds”.

Baseline Estimation

This is an essential step before modelling to get a feel of what will a non-performant model do and which metric to select. When the dataset contains equal instances of each class (ie 50:50), then a classifier predicting everything as 1 will get 50% correct. For skewed datasets, the biggest problem is that machine learning models will not learn the underlying pattern but will blindly predict everything to be the majority class (0). To recognize the situations when that happens, we need to know how the distributions look like

Turn out this dataset has less than 0.2% positive instances (492 rows). If you visualize as a histogram, the minority class will be barely visible.

This immediately tells us that we should not rely on headline accuracy but instead go deeper into the classification metrics.

The Precision, Recall and F1 score would be a good place to start because the dumb classifier which predicts everything as belonging to the majority class will have exactly 0 True Positives which will cause the above metrics to be 0. We will also rely on a custom metric to define our classifier.

Now that we know what a noisy prediction will look like for this dataset, in the next lecture we will begin the modelling process.

Predictive Modelling

Now lets get to the fun part !

I have instantiated a Random Forest Classifier from sklearn and then called the fit method on the Train data. Notice that the random seed has been set to 42. You can choose any random seed but make it explicit so that others can replicate your results.

Display performance metrics

I am going to call the custom_classification_metrics_function.py file that I have put together to clean up the main jupyter notebooks. This prints the Classification report and the underlying data needed to display the confusion matrix.

We are going to ignore the classification report and confusion matrix for the Training data because it is showing perfect metrics.

Lets direct our attention to the Test data:

If you look at the headline accuracy, it looks too good to be true at 100%. To find out if the model is really performing as expected, we need to deep dive into the classification report.

How do we interpret the classification report? In imbalanced datasets, its usually much more valuable to look at the metrics for just the minority class (in this case the 2nd row corresponding to the Fraud ) as opposed to the overall accuracy, which despite being close to 100% is not useful for reasons stated earlier. Lets also analyze the Confusion Matrix for good measure:

Interpret Metrics

At first glance, the performance seems pretty decent. Out of the 98 actual frauds, our model predicts 80 of them as frauds to give a True Positive Rate (ie Recall) of 82%. The Precision isnt too bad either at 94% meaning only 6% (in this case 5 samples on the top right) of the Frauds predicted by the classifier are wrong (ie False Positives). You will also get analogous results in the confusion matrix below.

Now given that the classification report already generates an F1 score, you might be wondering why would I go for a different metric. Although, (unlike Accuracy) the F1 score is sensitive to imbalance, it is unable to differentiate between a good recall or a good precision due to the symmetrical nature of the formula

F1 = 2*(Precision * Recall)/(Precision + Recall)

In scenarios where the business problem requires us to weight the Recall at a higher weightage than the Precision (like in our credit card fraud detection), the default probability threshold of 50% assumed by the sklearn’s RandomForestClassifier will not give us the desired performance metric. If detecting the Fraud cases are more important at the cost of letting in a few False Positives, then its better to lower our probability cutoff to below 50%.However if you start tuning the probability threshold for each classifier, it creates an issue: how do we go about comparing model performance if the probability cutoff itself can be tuned to what we want? The Precision, Recall, F1 scores are all probability-dependent and can’t directly be used.

One solution is to come up with a custom metric that takes into account the probabilistic aspects of the predictions and I will be covering in the next lecture:

Welcome back everyone! In this lecture, I will be introducing a new concept that allows you to compare the performance of models trained on imbalanced datasets.

In this code cell, I am utilizing the precision_at_recall_threshold_function imported from my utils file. I will quickly hop into the relevant section within the utils function.

This function takes in the true labels, predicted probabilities and the threshold for which recall needs to be computed. It makes use of sklearn’s Precision Recall Curve which is analogous to the ROC.

Using the Precision Recall Curve, various points along the curve for each probability cutoff are outputted into lists. Then the value just above the Recall threshold we have specified is taken and used. In this case the cutoff used is 85%

Now lets go back to our Jupyter notebook.

The above function is called using the following code snippet which uses the previously fitted classifier to generate the probabilities rather than the hard class labels. These probabilities are then passed into our precision_at_recall_threshold function.

When we run this code, we get an impressive 93% precision_at_recall metric meaning that there is some probability threshold that will allow you to obtain a precision = 93% and a Recall = 85%.

This might be a fairly good score depending on the business requirements.

If the business were to get back to check up on us at the end of Day 1, then we would present the model along with the findings. If you knew what the costs of misclassifying were for FN and FP, then you can figure out the cost-benefit tradeoff of using this Day 1 model versus whatever is being used currently.

Now the question remains: is there a way to improve it further? How do we find out ? One option is to dumb down the model into just 2 features and see how the class separation looks like in 2D space (rather than the original 30 feature space).

In this code block, what we are doing is we are instantiating and fitting 3 different dimensionality reduction techniques: T-SNE, PCA and SVD.

For every subplot, I am creating 2 scatterplots (one for each of the classes) and making them overlap. The blue dots represent the majority class (non-fraud) and the red dots represent the minority class (fraud). From the graphs, its not exactly obvious how the class separation is because the minority class members are so few.

In the subsequent days, we will explore more and more sophisticated approaches including Upsampling, Feature Engineering, Hyperparameter tuning and Ensembling to see if the performance is superior to what we achieved in the first day.

Hi everyone! Welcome to Day 2 of our Credit Card Fraud Detection series of notebooks

Better data will usually beat fancy algorithms! As part of the analysis in Day 2, lets focus on correcting the imbalance in the dataset and then correct any outliers.

Here is the table of contents corresponding to the above plan

Sections 1,2 and 3 are about introducing the problem, loading the packages, setting the file names and loading the data. Sections 4, 5 and 6 are where the core of the analysis take place. Section 4 will be where we are selecting the best data augmentation technique. Section 5 will introduce techniques that will help us to do basic EDA, detect, correct any outliers in the numerical data and test performance. Section 6 is where we can deal with null values and categorical variables but we will skip it in this business problem because this dataset is clean. Section 7 is where we can wrap up our analysis

Lets get started!

As before, lets import the libraries, set some of the variables & global parameters, then load the raw data. I am going to scroll through these sections because it’s the same as the previous notebooks.

Next I am going to introduce some theory on data augmentation.

Data Skewness correction

While we recognized in the previous analysis that the dataset is heavily imbalanced, we didn’t do anything to correct it before fitting the classifier on the training data. While the model performance wasn’t too bad, its a risky approach because the overwhelming amount of the majority class can drown out signal. In the current analysis, we will attempt 4 techniques to rebalance the dataset, check our performance through the precision_at_recall metric, then pick the best techniques.

The theoretical reason for favoring the 50:50 ratio is that the results will be no worse than what was obtained with natural class distributions.

Broadly there are 2 approaches to data skewness correction: undersampling and oversampling

In Undersampling, we throw away the majority class until the remaining data is balanced.

In Oversampling, we sample repeatedly the minority class until the synthetic data is balanced

There is also the hybrid approach where we sample both classes simultaneously until the minority class reaches a threshold and I will also introduce a custom function to generate samples.

But first, lets do some groundwork ….

We can continue to use the raw dataframe because its not in need of any cleaning. If there are situations where you need to use the cleaned dataframe, then import the dataframe that fed into the modelling workflow in Day1

Feature Target and Train Test Split

After loading the dataframe, lets split the feature from the target class and then do the Train Test split.

Why should we do the train test split here rather than just before the modelling?

Because the data augmentation techniques should leave the Test set alone and only resample the train set. The Test set should be a valid representation of how the unseen unlabelled data will occur during Production. Because one the model is live and is inferring, we wont have the luxury of correcting the imbalance because we wont have the class labels!

Initialize dataframe to hold metrics

The sampling_strategy_metrics_df dataframe will hold several key metrics including the precision_at_recall which we discussed in the previous article

In the next code cell, I am going to import some functions that will be used to automate the testing of the resampled data for performance comparison. This is basically a wrapper around the instantiation, fitting and testing of the classifier. The resample and test performance will also do the data augmentation and call the test rf performance within the function.

In the next lecture we will run each of the 4 resampling methods and capture the above metrics for each of them using the automated test functions above

There are many techniques for undersampling, oversampling and the hybrid approach. One popular API is imblearn available here https://imbalanced-learn.readthedocs.io/en/stable/api.html

In this notebook, I am going to pick one implementation for each approach and compare results. The reader is encouraged to try multiple variants and see what works best for their datasets

Instead of repeating prediction and testing part each time, I am going to call the custom functions that I imported above that take in a resampling method and automatically test the performance against a RandomForestClassifier. Note that I assume the resampling methods have a fit_sample method that can be called to either upsample or downsample the training dataset. This is true for those that are based out of imblearn

The detailed explanation of the resample_and_test_performance_function is given in Appendix section 5. In brief, you pass in the resampling method, train, test data and the labels. You will receive the resampled X_train and Y_train which I have choosen to throw away by storing them in a dummy variable underscore. The 3rd returned value is the dictionary containing all the classification metrics which can be stored in the dataframe

RandomUndersampler

In the first implementation, I am going to downsample my majority (negative) class using the RandomUnderSampler method as the name suggests, the algorithm randomly selects the minority class until the class ratio is the same.

I have imported the RandomUnderSampler method from the imblearn.undersampling class.

The Undersampling reduces the size of the dataset to 788 instances by eliminating all but 394 of the (previously) majority class. We are going to skip analyzing the output in detail and move onto the sampling_strategy_metrics_df which got updated with the metrics as seen below:

The precision_at_recall_threshold metric is poor at only 71%. Though the Recall is better than our baseline, the fact that the Precision is so poor makes this model unusable ie though we capture nearly 90% of our Frauds, we have too many False Positives.

If the modelling efforts were to stop now, business will choose the baseline model from day1 because the False Positives are significantly lower and likely outweigh any benefits from increased capture of True Frauds.

SMOTE

Next, I am going to try a popular oversampling technique using SMOTE. SMOTE stands for Synthetic Minority Over-sampling Technique and was presented in a 2002 paper here https://jair.org/index.php/jair/article/view/10302/24590.

Theory: This technique is theoretically superior to just oversampling with replacement for the minority class (the latter causes the decision tree to overfit). In SMOTE, instead of sampling with replacement of the original minority class, synthetic new samples are generated. The audience is encouraged to check out this article for more information on how the synthetic samples are created.

Next, lets move onto the implementation. In this code block, after importing the SMOTE method, I am passing in the instantiated SMOTE method into the resample_and_test_performance_function

If I scroll down to the sampling_strategy_metrics_df results are being shown, we can see that the precision_at_recall metric has now increased to 80%.

SMOTEENN

Next, we are going to explore how the performance is affected if we go with a combination of upsampling and undersampling. We are going to explore a strategy known as SMOTEENN which combines the now-familiar SMOTE upsampling technique with ENN undersampling. The API docs can be found here https://imbalanced-learn.readthedocs.io/en/stable/api.html#module-imblearn.combine

In this code block, I imported the SMOTEEN method. Then I called the resample_and_test_performance_function. Unlike the previous 2 resampling methods, SMOTEENN comes with the param for the sampling_strategy. The default is auto' which will select the minority class only but I chose to resample both classes. I have set the random_state to be 42 which will enable the audience to repeat the results.

As seen from the 3rd row where the Precision_at_Recall is 78%, SMOTEEN is slightly worse than SMOTE for this dataset but better than Undersampling

Custom Data Augmentation

The generalized version of the oversampling is when we simultaneously upsample both the classes at different rates. I coded out a function called augment_data_function which takes in the original datasets and the positive and negative upsampling ratio desired. This function returns the upsampled datasets and is explained in great detail in the appendix section. For now, you can assume that it can upsample the minority and majority class independently in arbitrary ratios (in this case by 4 and 2 respectively)

Because this function doesn’t have a fit and sample method (unlike the imblearn ones from before), I have chosen to manually call the test_rf_performance function rather than through the resample_and_test_performance function.

Homework to the audience is to create a class that wraps around this function and has the fit_sample method. Then you can refactor the code to be consistent with the above.

As seen from the sampling_strategy_metrics_df: The last row shows that the Precision_at_recall metric is 88%. The custom function's results are superior any of the other resampling techniques seen so far and close to the baseline model.

Notice that even the best resampling method gives a Precision_at_recall metric lower than the Day1. But in the interest of the analysis, we will assume that future iterations will require the resampling to take place. The viewer is encouraged to try out the future iterations without the resampling just to see the effect. If the data scientist had directly jumped onto the more complex methods, then he would have lost their chance to simultaneously have good performance and keep the model (like in Day1)

Lets try to further improve this performance by removing Outliers and rerunning the training.

Pre cleaning setup

Some datasets require the data scientist to clean out any blanks, special characters, etc. However, this dataset is clean and there isnt anything that needs to be applied to the whole dataset. So we will skip this section

EDA & Handling Outliers

This is the section where we deep dive into the data and do EDA. However I am only going to focus on the top features in the interests of time. The idea is to identify patterns in the data that will eventually make our model more powerful (either through Feature Engineering done in Day3 or removing data points because they are outliers which are unlikely to occur in Production data

Selecting top features through feature importances

Which features do we focus our efforts on for identifying and removing the outliers? Instead of all the 30 features, aim to pick the top 5 based on the feature importance and get them cleaned up.

There are many methods to select the top features including using a Correlation Matrix with the Target Variable. Here we want to use something familiar and simple like the RandomForestClassifier we have been using all along

After fitting a random forest, we use the feature_importances_ method to obtain the feature_importances and convert into a dataframe where we can visualize it easily.

As seen above, the top 5 features are V14, V17, V10, V12 and V16. Because this is an anonymized dataset from Kaggle, we dont really know what these are. In real life, you would ask the business to double check if these being the top 5 makes sense. For now, proceed with the rest of the analysis.

Isolating the Fraudulent datapoints

Given that our dataset is skewed, if we were to include the whole dataset in identifying the outliers, the concern is that the outliers from the non-fraud cases will swamp the signal from the fraud cases. Hence the right thing to do will be to focus on the subset of the data that is labelled as frauds (fraud_resampled_train_df), remove those and merge it back with the overall dataset.

In this code cell, as a first step, I joined our features and target variable. After concatenating, I rename the target column because the default is 0.

Subsequently, I separate out the fraud and non-fraud data into their own

Using the numerical_distribution_function, I have plotted the Histogram, BoxPlot and Quantile plots for the top 5 features that we previously selected.

If you scroll through some of these graphs, you will notice that the the distribution is sort of gaussian with a smattering of outliers on either edges.

Now how do we identify and remove these outliers?

Outlier Definition, Detection & Removal

There are various qualitative and quantitative methods of defining outliers depending on the use case. Given that the data feature column headers have been anonymised, we are constrained to use only the quantitative techniques for outlier detection.

Even among the quantitative techniques, there are several approaches, the most popular one being the use of Inter-Quartile Range to define the range in which the ‘usual’ number of points lie. Other methods include the use of Z scores to identify the boundaries beyond which the data points can be removed. The comparison of these 2 is given here http://colingorrie.github.io/outlier-detection.html

Both are fairly easy to implement and I am going to use the IQR method here

In this code cell, I have used the quantile method to extract the 25th and 75th quantile for the top 5 columns and transposed them so that the rows are the feature names and the columns are the summary stats. Next I have defined the lower and upper cutoff based on the formula for IQR and included it in the dataframe.

In the next code cell, I have gone through the important columns in a for loop and filtered for those values that are within my IQR. Using the print statement, I keep track of how many outliers I have removed for each column. Just to make sure that I am not deleting way too many values, I have printed the total number of rows deleted.

In the subsequent code cell, now that we have cleaned out the minority_resampled_train_df of any outliers, I am going to join it back with the other dataframe.

Finally, the performance is tested with the now familiar test_rf_performance_function. As seen in the output here, the precision_at_recall metric is slightly worse than with just the resampling implying that we might be losing some crucial information when we deleted the outliers. So I am not going to use the dataset obtained by the Outlier detection. Instead use the best resampling results for future iterations.

In case your colleagues want to replicate the results, it’s a good idea to export the datasets. I have coded out a custom function, data_export_function, to reduce the coding effort.

We are going to skip the Data Signal Amplification section for this business problem because we don’t have any null values or categorical variables that need to be imputed.

the above approach was presented to illustrate what is possible. In the subsequent days, we will start from scratch and explore the effect of feature engineering on the features alongwith the custom augmentation technique defined here.

Hi everyone! Welcome to Day 3 of our credit card fraud detection

Engineering better features with existing data is like dark magic. It takes not only years to master but also an intuition. In the 3rd day of analysis, I will introduce approaches to synthetically enhance the number of features in our dataset without adding too much noise.

This feature engineering step is critical for predictive modelling algorithms when you can’t ask for more features (for example the business may say that its too expensive to collect) and you are not using Deep Learning.

In the previous section, we tried to correct for the imbalance in the class distribution by upsampling. Given that increasing the number of rows wasnt enough to increase our precision_at_recall metric to above the 1st day of analysis, lets increase the number of columns (ie features) to see if it makes a difference in the Test performance.

We will use a mix of custom functions and polynomial feature generation API to expand the feature space. Then once the synthetic features are created, we will reduce the feature space by removing collinear columns and filtering further by the top features from the feature importances. Finally, we will run the Random Forest Classifier to get test performance on datasets that have been resampled using the technique identified previously.

Here is the table of contents corresponding to the above plan

Sections 1,2 and 3 are about introducing the problem, loading the packages, setting the file names and loading the data. Sections 4 and 5 are where the core of the analysis take place. Section 4 will require a round of memory reduction before the generation of features using math functions and polynomial feature generation. Section 5 will identify and eliminate features which carry redundant information. Section 6 is where we will test our performance and check if the preceding steps improved the signal to noise ratio. In Section 7, we will conclude our analysis followed by any links in Section 8

Lets go ahead and start our initialization protocol. As in the previous week, we will load the libraries

Then we will set the file path in which the utils function is found.

Next we will initialize all the metadata.

Finally we will do a quick import of the data and visually inspect. As before, lets reduce the size of our data using the reduce_mem_usage_function.

Data Prep

Since Feature Engineering is a computationally intensive task, its better to optimize parts of the problem beforehand.

Memory footprint reduction: We are going to reduce our memory footprint of the stored dataframe using the reduce_mem_usage_function to ease the burden on the processing.

I encourage the reader to explore the source code in the helper_functions_comprehensive.py. In short, the function parses through the numerical columns in the dataframe and tries to reduce the data type to the lowest possible without losing much of the precision. For example, as seen from the logs above, the final data type is float32 (it was originally float64). The final size is approximately 50% of what it was before!

The final output is the dataframe with the reduced memory footprint and the dictionary whose keys are the columns where some of the values have nulls.

Next we split the feature and the target class.

Selecting top features: In large corporate settings when you have access to big servers, you can consider all the features for feature engineering. But for those who dont have access to a lot of RAM, its better to be selective on which features get worked on. I am going to focus on the top 10 features by fitting a basic RandomForestClassifier, obtaining the feature importances and then selecting the top 10 features.

I have printed out the names of the columns alongwith the relative importances between them. These column names are stored in the imp_column_list. Subsequently, I have split up the Feature dataframe into the important ones that need to be engineered and the unimportant ones where the columns will remain the same.

Welcome back everyone! In this lecture we are going to do a lot of feature engineering, so hang in tight.

Feature Engineering can based on domain knowledge or clever intuition based on the understanding of the data. But given that this is a Kaggle dataset with anonymized features we cant do either. The next best option is to do combinations of features using mathematical operations which is what we are going to attempt here.

I am going to make a copy of the imp_X dataframe and store in the engineered_df dataframe. Here is how the engineered_df looks like before we create any synthetic fields.

In this code cell, I have written math functions to generate the square, log, sqrt and reciprocal functions. Instead of iterating through every row in my dataframe column manually, I have used the apply method along with the lambda function to succinctly achieve the mathematical transformation.

To make sure that the math transformations remain valid even when the underlying numbers are negative, I have created a variable to store the minimum value which then gets utilized in both the np.log() and the np.sqrt()

For example, the log of a number which is equal to or less than 0 will generate a NaN in numpy. Hence subtracting the value with the minimum value and then adding 1 will always ensure that the resulting answer is a valid number

Similarly, we have to ensure that the reciprocal function doesn’t produce infinities when the denominator is 0. So I have an if statement where the reciprocal function is replaced with a hardcoded 0 if the underlying value is 0, ie. I have defined 0/0 = 0 in this particular situation

The reader is encouraged to write more functions for their data.

Once we run the code, we expect to see the additional 40 columns.

As you can see in the output here, the engineered_df now shows you some of the newly engineered columns on the right side. Next we move onto another technique for generating synthetic features

Polynomial Feature Generation

Turns out that s klearn has a good package called Polynomial Features which is good at generating cross terms. How does this work? Basically, the API creates all possible cross terms between the features upto the polynomial degree passed in as a parameter. Lets implement this here:

In this code cell, after importing the PolynomialFeatures API, I instantiated an object. The parameters here specify to sklearn to generate all interaction terms between the features upto the quadratic product terms and leave out the bias terms. If you don’t specify include_bias = False, then you will end up with a column of 1s in addition to the rest of your data.

Now I am ready to generate the synthetic columns on our engineered_df dataframe. I use the poly_object to fit and transform in a single method and pass in the engineered dataframe as the input. The column names are automatically generated based on the existing column names.

Lets run the code and see how the columns look like. As you can see, despite limiting the degree of interaction to 2, the output has 1275 features ! Of course, quite a few of these are redundant or correlated with each other (for example, taking the square() of a sqrt() will give you back the original feature).

This creates a need for a bout of feature reduction shortly. Before that, lets reduce our memory footprint again and concatenated previously filtered away original features 11 -> 30 along with the 1275 above to give a grand total of 1295 features before the next feature reduction step. Why are we combining the previously identified low important features if you know that you will be dropping a significant number of features further down the code?

Because the old unimportant features may actually be higher ranked than some of the newer synthetic features

Welcome back everyone! In this lecture, we will be covering Feature Reduction on our Engineered_df.

We will do multiple rounds of feature reduction starting with removing the constant and duplicate columns

First lets use the custom function from Day 1, to identify ‘bad’ columns. The reason for using this function here is that it will help us to identify those features that are duplicates of each other or contain constant values. We don’t expect null values or blank spaces to show up.

Once we run the cell, it takes some time to complete. Once it finishes running, if you look at the logs, it turns out we have a few columns that can be safely removed without affecting the predictive power of our downstream model.

Lets do the removal in the next code cell

(Explain verbally)

Identify Collinear columns

Why doe we need another round of feature reduction?

Instead of columns which are exact duplicates of each other, what if the columns are scaled versions or somehow correlated with each other? This is where identifying collinear columns will be ideal.

These highly correlated variables can decrease the model's ability to learn, model interpretability, and generalization performance on the test set. What criteria should we use to remove them once identified? We will take features having more than 0.8 collinearity with another variable.

(Some source code has been adapted from Chris Albon https://chrisalbon.com/machine_learning/feature_selection/drop_highly_correlated_features/

and Will Koehrsen https://www.kaggle.com/willkoehrsen/introduction-to-feature-selection)

In this cell, we set the cutoff threshold at 80%. We then calculate the pearson correlation and then take the absolute value of it. Because the correlation matrix so generated is symmetrical across the main diagonal, we create a mask to extract only the upper triangular matrix and store it in upper_df. Running this cell takes a while because of the sheer number of pairwise comparison that needs to happen.

Note that the above method only captures linear correlations amongst the features and treats correlation in each direction the same (This assumption may not be true when there are variables that influence each other in non-linear ways. See this article for more advanced techniques https://towardsdatascience.com/rip-correlation-introducing-the-predictive-power-score-3d90808b9598).

In this code cell, lets identify the number of columns that have been flagged because they have greater than 0.80 correlation. Turns out there are 1085 columns and these will be eliminated in the next step. In the next code cell, we will drop the above columns. Despite this massive reduction, we still have 190 features.

Lets do another round of feature reduction using feature importances.

Use Feature Importances

Use our now-familiar approach to identifying feature importances and store the results in feature_df

When you have an immense number of features, then its possible that many of these are just noise and don’t add to the final predictions. In this code cell, there are no features with exactly 0 feature importance

Depending on the problem we can choose to remove just those features with 0 importance or formally establish a threshold for identify and drop those with low feature importance. In this business problem, I am going to pick the features that cumulatively explain 90% of the variance. In this code cell, we will plot these features in a horizontal bar chart and obtain the number of columns required to retain 90% of the signal. The plot_feature_importances function takes in the dataframe that contains just the ‘feature’ and the ‘importances’ column. It returns norm_feature_importances_df which has the feature names in descending of feature importances and the no_of_important_col which gives us the index position in the dataframe beyond which we can throw away the features.

This function generates the 2 plots as shown below:

The first one is just a horizontal bar chart with all the features names in the y axis and the feature importance strengths along the X axis. Notice how many of the top features are our synthetically generated columns.

The second plot is the Cumulative Importance Plot where the Y axis is the sum of the preceding list of feature importances. The cumulative plot can help the business decide on the number of features to keep. The curve rises steeply until the 75-80% with less than 50 features. The remaining 20% requires an additional 150 features. The previous function call assumed that the business requirement is to keep the features that cumulatively explain 90% of the variance. Depending on the business problem this can still give an enormous number of features (in this case we have 96 features remaining) in which case we can either reduce our threshold or do a custom filter to keep the top 25 features I will be doing the latter later in the Jupyter notebook. But first lets display some of the selected features.

Seems like our synthetic features came out on top.

Its good practice to record the data in case you want others to replicate your results. I am going to pickle the engineered_df before feature reduction.

In the next cell, relevant_feature_list is used to store the column names which explains the 90% of the variance (ie 96 columns). We will then use this list to filter the engineered_df. The top few rows are displayed.

Custom Feature Removal

If at this stage, you find that you have a lot of features, one option is to go back and reduce the information content to be preserved. The other option is to have an arbitrary threshold. I am going to pick the top 25 from our relevant_feature_list later in the next section.

Welcome back everyone !

The time has come to see if all the hard work in creating these synthetic columns has any impact on the metrics. Lets do our usual train test split. After upsampling the subset of the above with the top 25 features, lets fit the RandomForest classifier and test.

I am going to run the custom_classification_metric function on both the train and the test set. Lets Check the classification report to see how the metrics relate to the previous 2 days of modelling. Seems the confusion matrix gives us identical performance metrics as our model in Day 1. The AUC ROC shows a slightly lower value of 0.94 compared to the Day performance of 0.96. Lets bring in the precision_at_recall_threshold metric to see if its actually lower.

Turns out our suspicions were correct. The Precision_at_recall metric is indeed lower than Day 1.

All the hard work in feature engineering made the model performance slightly worse than what we had originally in Day 1!

This is typical in Data Science where sophistication doesnt necessarily imply greater performance. But we have more tricks up our sleeve in the subsequent days for Hyperparameter tuning and Ensembling. In the interests of time, I have not played around with how many features and subsequently didnt use the feature engineered dataframe in further weeks. But the reader is invited to try them out with hyperparam tuning. In the next section, I will be covering Grid Search for tuning hyperparameters in our RandomForest Classifier.

Hyperparameter tuning using GridSearchCV is the next step towards increasing the performance of our RandomForestClassifier. Tuning these requires reading through the documentation, patience to experiment with different combinations and either lots of time or powerful servers to run the tuning.

Implementations of ML algorithms (including our RandomForestClassifier) will inevitably have ‘knobs’ that can be used to tune their performance to the dataset at hand. This is analogous to tuning a guitar before a performance.

While the default implementation of RandomForestClassifier has given us decent performance, we will attempt to find a better combination of parameters using an implementation of brute-force search through the sklearn’s GridSearchCV. Because grid search uses accuracy by default which we know ahead of time that its not the right metric for imbalanced datasets, we will also be passing in a custom scoring function. Once the best hyperparameters are found, we will instantiate a classifier with these params and then test the performance.

I assume readers are comfortable with initializing all the metadata, loading the csv file as a dataframe, reducing the memory footprint, splitting the dataframe into train & test, splitting the train again and then upsampling one of the folds. So I am going to jump straight to the newer concepts.

After reading through the sklearn documentation on RandomForestClassifier (https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html ) and based on the understanding of how they are constructed, I picked 4 parameters to tune.

Lets briefly explain what the hyperparameters are in this code cell

n_estimators gives you the number of decision trees that will be built as part of the ensemble. Higher the number, the more the model is prone to overfitting.

max_depth constraints the algorithm from constructing individual decision trees too deep. At each node, I have constrained the splitting to happen only if the number of samples remaining exceed the range given in the min_samples_split param. Similarly, min_samples_leaf will decide what is the lowest number of samples in each of the leaves. If you dont set any boundaries, the algorithm will continue going until the individual leaves are pure! The audience is encouraged to try both a large set of parameters and broader search space if you have access to more powerful servers.

Now how do we feed in these combinations to our RandomForestClassifier? One option is to manually code out a for loop that iterates through every combination of the above hyperparameters, feeds into the RandomForestClassifier and validates against a validation set and then picks the combination that has the best performance.

GridSearchCV from sklearn already allows you to search through potential parameter combinations and find the best one. It accomplishes this by splitting the train data into different stratified cv folds. Then fitting one parameter combo 3 times each (in this case cv=3 to speed up tuning) and taking the average ‘score’ to compare performance across different combinations.

I created a custom function, tune_grid_search_function, to wrap around sklearn’s GridSearchCV.

I encourage the audience to go to section 5.9 and watch the video where I explain the tune_grid_search_function in more detail along with ways to pass in your custom_scoring_function. sklearn also gives you other built-in scoring metrics you can choose from here. I encourage the readers to try out a generalized version of the F1 score called the fbeta score given here, in addition to the precision_at_recall threshold used in this tutorial.

We call this function using the resampled training data, the validation data, the previously initialized random forest parameters, the RandomForest Classifier as our classifier type alongwith the 3 keyword arguments for setting the is_custom_scorer flag, custom_scoring function and the recall_threshold of 85%

Running this cell takes a really long time and depends on how many parameters you have passed in to the function. Here in our case, tuning takes a few hours to execute given that there are 144 combinations to try out with 3 fold cv giving 432 runs. Because of the verbosity you can see the logs for every iteration of the gridsearch. Lets take a break and come back

(transition)

Once the search for the optimum parameters is complete, we can print some of the classification metrics to see how they looked like on the training data.

To see how the results improved over time, I have plotted each of the parameter combination (taken from the random forest grid) against its cv scores. If you have more time, you can run a more refined gridsearch on the combinations that seem to have a higher gridsearch score.

After upsampling the training set with the best augmentation technique from Day 2, we are ready to fit our RandomForestClassifier with the best parameters from our GridSearch. Notice how I have used double asterisk to get the best_params out of the nested dictionary.

Subsequently, we pass in our fitted model through the custom classification metrics function to generate the usual classification report, confusion matrix and ROC AUC curve.

If you inspect the confusion matrix, you will notice that the numbers are not an improvement over Day1. Lets confirm if this is indeed the case by using our precision_at_recall_threshold function.

As seen from the result, the hyperparameter tuning didn’t generate better results from before, implying that the simple model from Day 1 should have sufficient performance in production. The viewer is encouraged to try out a wider search space (perhaps with more trees, deeper depth, lower number of leaves) and see if it improves the test performance.

In the next lecture, I will introduce ensemble techniques to scale up the hyperparameter tuning across multiple classifiers to give one last shot at improving the performance.

Building sophisticated ensemble models using tuned base classifiers is how a lot of Kaggle competitions are won. In our final Day 5 of analysis, I will introduce an approach in combining the predictions of multiple classifiers.

The fundamental idea is to take the predictions of multiple tuned classifiers and have a ‘voting’ layer that creates the final predictions. Get ready for an election!

Today, we will be going beyond RandomForestClassifiers and explore other classifiers for hyperparameter tuning. Key thing to note is that we will be diving into the documentation to change the default eval_metric (for XGBoost) to make sure its compatible with imbalanced datasets. Once the best hyperparams are found for each classifier, they will be fitted with the upsampled test data and prediction probabilities generated. The custom precision_at_recall metric will continue to be used in judging the performance on the Test data.

Here is the table of contents corresponding to the above plan

Sections 1,2 and 3 are about introducing the problem, loading the packages, setting the file names and loading the data. Sections 4 and 5 are where the core of the analysis take place. Section 4 will require a round of memory reduction before we can proceed with the GridSearch for the best hyperparameters. The same custom tuning function will be used but this time for 5 tree-based classifiers. In Section 5, we will identify the top performing classifiers based on our custom precision_at_recall threshold metric, then use it to create an ensemble and test our performance.

I assume readers are comfortable loading the csv file as a dataframe, reducing the memory footprint, splitting the dataframe into train & test, splitting the train again and then upsampling one of the folds.

Next we are going to import 4 other decision tree based classifiers. Amongst the 5 classifiers, RandomForestClassifier and ExtraTreesClassifier are kind of similar. While the 3 variations of the gradient boosting decision trees are supposedly similar. We will later find out that despite the theory, the performance is not similar. I am going to briefly describe what these are

ExtraTreesClassifier stands for Extremely Randomized Trees. Its meant to reduce overfitting. Compared to Random Forest, Extremely Randomized Trees further randomizes how trees are built. Extremely Randomized Trees have even the splits being randomized unlike RF where the feature splits are deterministic. Also, unlike RF, sampling happens without replacement (ie Bootstrapping = False on average). ET have better performance in the presence of noisy features vs RF. When all the variables are equally important, then it doesnt matter what you use. Here is the paper which proposed this algorithm. https://orbi.uliege.be/handle/2268/9357

GradientBoostingClassifier as the name suggests, this algorithm iteratively constructs weak learners on only the incorrectly classified instances. A more detailed explanation can be found here https://stackabuse.com/gradient-boosting-classifiers-in-python-with-scikit-learn/

LGBMClassifier is a variation of the gradient boosting machines that is supposedly faster because it uses leaf-wise tree growth algorithm rather than depth-wise. It was developed in Microsoft and has an alternative syntax to call the APIs which you can read in the docs here https://lightgbm.readthedocs.io/en/latest/Parameters-Tuning.html

XGBoost is another variant of the gradient boosting algorithms above but with lots of bells and whistles to improve performance (both speed and accuracy). The original implementation has a slightly different API syntax than usual but I have used a wrapper makes the syntax compatible with the other classifiers. XGBoost is commonly used amongst kagglers as the go-to choice to win competitions. Feel free to check out this excellent resource that describes the evolution of decision-tree based algorithms https://towardsdatascience.com/https-medium-com-vishalmorde-xgboost-algorithm-long-she-may-rein-edd9f99be63d

So why did I only pick only Decision-tree based classifiers?

My reasoning for picking these instead of SVMs, KNN or Neural Nets is purely from a data pre-processing perspective. Distance based algorithms will require you to scale the data so that all the features are in the same range which is not required for Decision Tree based algorithms. [Homework to the viewers is to try out the ensemble with other ML algorithms and use appropriate scaling]

In the next code cell, I created a dictionary to hold all the initialized classifier objects which we will later pass onto the tune_grid_search_function. Note that there is no parenthesis () for any of them because I will be feeding in the parameters inside the GridSearchCV.

Next I am going to initialize the search space for each of the ML algorithms

Now the question comes down to which parameters do we choose?

For the random forest, I am going to continue keeping the same 4 we chose before n_estimators, max_depth, min_samples_split and min_samples_leaf.

Next we turn our attention to ExtraTreesClassifier. Since ETC is already supposed to reduce the overfitting, I am going to make the parameters to err on the side of overfitting. For example, I increased the max_depth and reduced the number of leaves. Other than that I have chosen the same parameters as our random forest

Next, lets define the GradientBoosing Classifier search space. In addition to our usual decision tree related params like n_estimators, max_depth, min_samples_split, min_samples_leaf, we also introduce sub_sample. This hyperparameter allows one to control the fraction of samples to be used for fitting the individual base learners. Note that the n_estimators here are built sequentially rather than concurrently like in RandomForest. I have also specified that early stopping needs to be done if after 5 iterations, there is no change in the validation data (specified to be 10% of the training set). For imbalanced datasets, we need to watch out if the split is stratified and thankfully the documentation says it is.

Lets move onto the 4th classifier search space

The 2 main differences you will notice in LightGBM compared to the earlier classifiers are

a) the min_samples_in_leaf is now labelled as min_data_in_leaf and performs a similar role

b) that we introduced 2 additional hyperparameters that are specific to LightGBM 'objective':['binary'],

'eval_metric':['binary_logloss'],

The objective parameters being binary tells the classifier to expect a binary classification problem. Why did I set the eval_metric as logloss? Because logloss works on predicted probabilities rather than hard predictions. It penalizes those predictions that are far away from the actuals. Note that since this is a loss metric, the lower the better. This is equivalent to the cross-entropy which is a generalized version for multiclass classification. See this article for more information https://stackoverflow.com/questions/50913508/what-is-the-difference-between-cross-entropy-and-log-loss-error.

Finally, lets define the hyperparam search space for the most popular non-Deep Learning algorithm – XGBoost.

n_estimators: As the name suggests, it is the number of trees built. I have kept it at the default value of 100 because I dont want the simulation to run too long.

max_depth: A good range will encompass the default (=6). max_depth tells the algorithm how many levels deep a tree is allowed to go. The interaction between the previous split and a new variable can be captured in each level.

colsample_bytree: All colsample_by* parameters have a range of (0, 1], the default value of 1. This parameter specifies the fraction of columns to be subsampled in each tree. In our case, colsample_bytree is the subsample ratio of columns when constructing each tree. Subsampling occurs once for every tree constructed.

I could have also chosen to tune other subsample params which will affect what proportion of the training features gets picked. But have chosen not to in the interest of simulation time.

There is nothing to tune in the following but I have specified it as arguments to the model:

random_state: I always set the random seed explicitly to any function that has the parameter

objective: Because ours is a binary classification problem, I have set it as binary:logistic and it will generate an output probability.

eval_metric: Changing the eval_metric to something meaningful for imbalanced dataset is critical because by default XGBoost will make use of #(of wrong cases)/(Total # of cases), which is analogous to using accuracy. If we had not changed this to aucpr, then the evaluation will regard the instances with prediction value larger than 0.5 as positive instances, and the others as negative instances.

early_stopping_rounds, as the name suggests will stop the search sooner. I set it as 5 because I want the model to not waste time if the Validation metric doesnt change beyond 5 rounds.

XGBoost is a really powerful technique and should be part of your arsenal. I encourage all readers to spend time in going through the documentation for XGBoost here (https://xgboost.readthedocs.io/en/latest/parameter.html ) and here (https://github.com/dmlc/xgboost/blob/master/doc/parameter.rst) . Refer to this kaggle (https://www.kaggle.com/c/santander-customer-satisfaction/discussion/20662) discussion for insights on how to pick some of the hyperparams.

[HW: In all of the above classifiers, its possible to tell the classifier that the underlying data is imbalanced and therefore to deliberately overweight the minority classes. Homework to the audience is to use the class_weight hyperparameter instead of the custom upsampling method ]

In this section, we are going to go through all the previously defined classifiers and run it through the custom tune_grid_search_function.

The main difference in this code cell compared to the previous iteration of the hyperparameter search in Day 4 is that we are storing the results in a hyperparam_results_dict dictionary. I am going to run this and come back later to explain the results

Before we proceed further, lets export the results as a pickled object

Lets check out the performance of the individual tuned models from the step above. Lets first upsample the training set using the custom augmentation approach from Day 2

In the next code cell, lets create 2 dictionaries – one to hold the precision_at_recall threshold for each of our individual models and the other to hold the probability scores that can subsequently be used for ensemble level.

As you can see from the above output, the test performance is noticeable different between the various classifiers. Note that at this stage, you might find out that one of your tuned models yields superior performance and decide to stop here rather than proceed to the Ensemble stage. If performance is a requirement, then you might want to pick the best model from above and productionalize it. A valid reason for stoppping now would be that it requires greater engineering effort to maintain production pipelines for an Ensemble model vs going to production with a single model like XGBoost. I have continued on with the analysis for illustrative purposes.

Our approach would be to take the best models from the above and create an ensemble which is what we will be doing in the next section

Now we have the best hyperparams for all the different classifiers. You can check the individual model performance and pick the top 3 for feeding them into the Voting classifier.

Initialize Base Classifiers

I am going to pick the RandomForestClassifier, and XGBClassifier as base models for later ensembling. Depending on the dataset, you will find that other tuned models will have better performance and hence need to be picked.

In this code cell, I have used the best params obtained during the tuning to initialize the 3 classifiers that I previously noticed have the best performance. You can decide to choose all of them if you wanted.

In the next code cell, I have imported the VotingClassifier from sklearn. I have initialized a VotingClassifier() from sklearn with the 3 base classifiers above passed through the estimators argument.

Theory: But what does a VotingClassifier() actually do?

You dont actually need sklearn to accomplish this ensembling for you: you can code it out in Python. Its just more convenient because the .fit(), .predict(), .predict_proba() and other methods are already implemented and you dont have to code a class from scratch.

Ensembling using the above methodology is meant to smooth out variations and idiosyncracies in the way individual models would predict. For each class, the weighted sum is calculated across all the classifiers of either the probabilities (voting=’soft’) or labels (voting=’hard’).

For the parameters, I have set the voting=’soft’ to indicate that I am interested in the Voting Classifier looking at the base classifier’s probabilities rather than the hard thresholded predictions. By default, all the classifiers are equally weighted but since I noticed that XGBoost has the highest score, I have deliberately given it a weight of 3. The flatten_transform=True allows one to obtain the individual base classifier’s predictions if needed through the .transform() method (but we are not going to do that.)

Anyway, now that we have fitted the VotingClassifier(), lets go ahead and test the performance.

Lets run the fitted VotingClassifier through the usual gauntlet of custom_classification_metrics_function to generate the classification report, confusion matrix and the ROC AUC curve.

From the classification report, we notice that the performance metrics are pretty decent.

Lets check out how the Precision_at_recall_threshold metric. Turns out that the Day 5 performance is good but the Day 1 performance is still the leading contender

How exactly are the 2 models in Day 1 and Day 5 different from each other? To get a feel for it, here is a Precision Recall curve that can be used to visualize.

An ideal Precision Recall curve will look like a square with a near perfect sharp 90 degree bend going from the top right to the bottom right. Real ML algorithms will have this part smoother. The higher the top right is, the more likely it is that the area under the precision recall curve will be larger. However, there may be slight differences in how long a given fitted model’s precision-recall curve ‘stays’ horizontal before falling down to the X axis. This can cause the all too crucial distinction between an algorithm that gently tapers downwards vs something that stays flat but quickly ‘falls’. This difference can be detected through the precision at recall metric mathematically or visually through inspection of the individual curves at the top right.

In our case, Day 1 fitted model has a better curve than the Day 5 model. Of course, this is all based on the limited subset of hyperparameters that I have chosen to tune and also the limited range. As a homework, I encourage the viewers to increase the grid search range and note down any differences in the performance

Congratulations for finishing all the 5 days worth of iteration for Detecting Credit card fraud!

We have gone through the credit card dataset over 5 days and gradually increased the complexity of the modelling process.

Here are the key concepts we covered

1. Automated detection of bad columns in our raw data (Day1)

2. Building a naïve model using class distributions (Day1)

3. Custom metric suitable for imbalanced datasets (Day1)

4. 4 Data Augmentation Techniques (Day2)

5. 2 Feature Engineering Techniques (Day2)

6. 4 Feature Reduction Techniques (Day2)

7. Memory footprint reduction techniques (Day3)

8. Setting a custom scoring function inside the GridSearchCV (Day4)

9. Finding suitable hyperparameters to tune for 5 decision-tree based classifiers (Day5)

10. Building an ensemble of classifiers (Day5)

While we should be proud of having tried these complex ideas, what we have found it is that improving performance over baseline models is hard and the business may yet decide to go ahead with the RandomForestClassifier without any tuned hyperparameters for its simplicity. Which is why we tackled the RF first without any bells and whistles to illustrate that you can get ROI out of a project right from Day 1. But of course there are business situations when you do indeed need more complex models in which case its better to go all the way from Day 1 to Day 5.

So that’s it for the credit card fraud detection!

Please refer to the attached Python file in the Resource section for the functions mentioned in this section

Before we get started with describing the helper functions, I want to reiterate some of the best practices in defining a Python function. Its good to have a description underneath the function definition using Triple Quotes. This allows the tab key to bring up the description in your script after you import it. Anyway, lets move to the find_bad_columns_function itself.

This function is used to find out whether the dataframe has columns that contain nulls, blank spaces, constant values throughout or those columns that are duplicates of each other.

The null_col_list will contain the names of those columns that have the nulls.

As you can see from the argument of dataframe.columns, I am indexing based on the presences of nulls in any of the rows. The .to_list() makes sure that the final values are stored as a list.

Next, we are going to find out if any of our dataframe columns have blanks.

Blank spaces can occur in only those columns that have been interpreted as object columns. So to identify columns with blanks, the first step is to only focus on those columns that are interpreted by pandas as being of object type. Next I run a for loop to go through each of the object column and check against the presence of a blank. If any blanks exist in the column, then the if condition is true and that column name gets stored in the blank_space_col_list

Third, I have the logic for detecting columns that have too low of a variance. The basic idea is that the entire column has the same number throughout, then that column will carry no information.

I have split up the columns into numerical and non-numerical columns separately because the logic is slightly different

For numerical columns, I rely on the use of standard deviation to find out those columns that have it below 1%

For non-numerical columns, I rely on the most frequently occurring value occupying greater than 99% of the storage space.

Caveat here is that for extremely imbalanced datasets, the rare level can potentially indicate the presence of the minority class, so before removing quasi-constant columns, please make sure the presence of this level doesn’t coincide with the minority class

The last check will be for identifying those columns that are duplicates of each other. The rationale for removing it is to keep only 1 item when there is a pair of strongly correlated columns (otherwise the Decision trees will make incorrect splits)

Finally we return the lists for the user to make a decision on whether to drop it or keep it.

Here is how the sample output looks like

The first output is the test accuracy, which is the headline number that many business executives care about. I advise the audience to take this number with a pinch of salt because it is meaningless for imbalanced datasets and in those scenarios where the cost of false positive and false negatives are different

The 2nd section is the Classification report which gives a lot more information about the classifier including the breakdown of the metrics per class type.

The 3rd output is the value for the area under the ROC curve. Again a high number can be misleading for those situations where the data is skewed.

The 4th output is the graph for the ROC graph

And finally the 5th output display is the confusion matrix which can tell us what the actual number of true and false positives are in the dataset. The y axis shows the actual labels and the x axis shows the classifiers predicted labels.

Now that we understand the context in which the function is being used, let me go back to the helper_functions_comprehensive python script and explain how the function was created.

The function has the following arguments,

At a high level there are 6 main sections in this function:

· printing the accuracy,

· printing classification report,

· printing roc_auc_value,

· displaying the roc_auc_curve

· displaying the confusion matrix

· calculating and returning dictionary containing the key metrics

Lets go through them one by one:

Accuracy

The starting point for all downstream calculations is the fitted classifier object that has been previously trained on the training data. This is used to generate predictions on the Test set.

The predictions are then compared with the actual y_test values using the .score() method to get the accuracy.

Classification report

Sklearn has a built in method to calculate the classification report which just needs the y_test and y_pred

ROC AUC score

To get the area under the curve, we start by generating the prediction probabilities using the predict_proba() method on the trained classifier. The output probabilities are then fed into the sklearn method, roc_auc_score to obtain the actual value. Note that I have truncated the display to 2 decimal places before printing it

ROC AUC Curve

Next lets generate the Receiver Operating Characteristic curve. This will be a 2-step process where I first generate the true-positive rate, false positive rate and the threshold using sklearn’s roc_curve function. Using these tpr and fpr, I pass them to another function which generates the plot. This function is defined separately. If I were to scroll upto where the function is defined, you will see how the roc_curve_function is defined. The 2 lines containing the .plot methods are the core of this function

The roc_curve_function plots 2 plots on top of each others. Now if I go back to the Jupyter notebook that contains this plot, you will notice the dotted red line and the solid blue line: the diagonal line represents random guessing. The blue line represents the roc auc plot for the fitted classifier. The ROC curve is plotted with TPR against the FPR where TPR is on y-axis and FPR is on the x-axis. These were previously obtained using the sklearn method roc_auc.

Confusion Matrix

The most informative graph is the confusion matrix. The first step is to generate the 2x2 arrays of numbers corresponding to the 4 quadrants in the confusion matrix. This is accomplished by the sklearn method confusion_matrix. These numbers are then plotted using the plot_confusion_matrix_function. Note that this function has been adapted from the sklearn page https://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html

Output Metrics

There will be situations when we will need some of the numbers from the confusion matrix, classification report and the roc_auc curve. So I have created an if statement that gets triggered when the user explicitly sets the output_metrics flag to be true. I initialized a counter dictionary to store the 8 types of metrics I am interested in. 4 of these are from the confusion matrix, 3 are from the classification report and the last one is from the ROC AUC curve

Hi everyone, welcome to section 5.3. In this lecture, I am going to describe where the resample_and_test_perfomnce_function is used and how it is created. As we saw in Day 2 of our credit card fraud detection notebook, we tried resampling methods to rebalance our skewed datasets when the majority class far exceeds the minority class. When we are trying out different resampling methods in Day2, it becomes easier if there is a function that allows one to quickly generate the rebalanced data and also test it out using the classification metrics that we defined earlier.

Now lets go back to the helper function script where the resample_and_test_performance function is defined. The arguments are the Imbalance learn object, the splitted train and test data along with the labels.

Note that even though we have primarily used only Imblearn as the input arguments, the way I have created the function allows for any resampling method that has the fit_sample method to call upon.

In the first line of code, we see that the fit_sample method is called upon the train dataset to generate the resampled train dataset for the features and target respectively.

These resampled data are then passed into the test_rf_performance_function which generates the classification metrics of interest.

Now what does the test_rf_performance_function do?

If I were to scroll up in the helper script where this is defined we see that this is just a wrapper around our Custom_classification_metrics_function with an additional step for fitting a RandomForest classifier on the resampled dataset and finally storing the precision_at_recall_function’s output in the metrics_dict. The key thing to note is that we set the output _metrics flag so that we can generate the metrics_dict dictionary and pass it along to the calling function.

Hi everyone, welcome to lecture 5.4. In today’s lecture, I will be describing where the augment_data_function is used in Day2 and how the function was constructed.

This function was first used in Day2 credit card fraud detection when we needed to do custom upsampling. This is an alternative resampling methodology you want to explore in addition to the built-in resampling methods in imblearn.

Lets go to the helper script to see how this was created.

# This function was inspired by the notebook from Kaggle found in the links given https://www.kaggle.com/niteshx2/beginner-explained-lgb-2-leaves-augment

I have modified the original function to make the upsampling ratios for the minority and majority class independently tweakable.

We start off with setting the random seed and converting the input dataframes into arrays

Then initialize a list for holding the empty rows for the subset of the features corresponding to the positive and negative labels

We create 2 separate for loops, one for each class type

The number of times the for loop runs is 1 less than the respective sampling ratios that we passed into the function. This is so that the total number of items in the class after the augmentation aligns with the ratios that we desire

Within each for loop, we start off with getting the subset of the features for which the label is positive or negative.

We get the indices of the subset temp_array and store in the ids variable.

Then in the inner for loop, we go through every feature column and randomly assign it a value from the same column of a different row. This adds some new information to the dataset, since the feature column values of the created sample are a new combination of known data rather than introducing new fictitious information.

Outside the inner for loop, we append the shuffled temp_array to the xs or xn

Once the upsampling is done for each class, we unfold the xs, xn arrays using vstack

We create the y variable corresponding to the lengths of the xs and xn

Then we merge all the features together into x and all the labels into y

These arrays are converted back into pandas datastructures for the features and labels. We shuffle them again so that the 1s and 0s are randomly distributed rather than right below one another.

Hi everyone, welcome to lecture 5.5. In today’s lecture, I will be describing where the numerical_distribution_function is used in Day2 and how the function was constructed.

This function was first utilized in the Exploratory Data Analysis section where we plotted the histogram, boxplot and probability plots for some of the important columns. These plots allow you to quickly visualize and infer potential outliers. Now lets head over to the helper scripts to figure out how it was written.

This function allows you to automate the plots for each column in your dataframe. We pass in the dataframe and the number of bins which we want in the x axis. There is a for loop that iterates through each column and then plots the histogram, Boxplots and Quantile plots respectively for the viewer to then visually inspect.

Note that the Histogram’s X axis can be tuned and it is typical to bin it so that there are atleast 30 data points on average per bin.

Hi everyone, welcome to lecture 5.6. In today’s lecture, I will be describing where the data_export _function is used in Day2 and how the function was constructed.

This function is useful when you want to capture what dataset was used for a particular analysis either for yourself or for sharing with a colleague.

Lets head over to the helper scripts to see how it was created.

The function is fairly straightforward to understand as it comprises of building blocks seen before in Day1. You pass in the train and test sets for the features and labels. Additional arguments are needed for the file path, file name and the date which will be used as a suffix.

First you reset the indices, so that during the concatenation step, the rows are not interleaved.

Subsequently, you attach the label column to the features. Then you concatenate the test with the train dataframes (along the rows which is indicated by axis=0). Lastly you export the dataframe as a csv file

This concludes our short lecture on how to automate the data_export function.

Hi everyone, welcome to today’s session where I will be describing where the reduce_mem_usage_function is used and how the function was constructed.

We first used it in the Day 3 notebook where we needed to reduce the dataset size before feature engineering. As seen in the output, using this function on the original dataframe reduced the memory footprint by 50% primarily by converting the float64 into float32. The output is the same dataframe but with the datatypes being a reduced version wherever possible.

Lets head over to the helper scripts to see how this is achieved. Note that this function is based off of a Kaggle solution given in the link above.

The basic idea is that the numerical columns containing integers and floats are reduced to the lowest possible datatype while preserving the precision. Floats are also converted to integers if it turns out that there is no need for the extra decimal precision.

The function is initialized with 3 arguments with the input dataframe being mandatory. The initial line is to tell the user the starting memory footprint of the dataframe.

We initialize a dictionary outside the for loop to store the column names that have nulls along with the imputed value. The for loop iterates through every column and only applies the datatype change to those columns that are not objects.

We initialize a flag for IsInt to be False. Also store the minimum and maximum values for each column

Now you might be wondering what does the IsInt flag do?

It recognizes situations where the decimal value is really small compared to the integer part of it (notice the threshold is 10). In those scenarios, the numbers are rounded off. The np.abs() is just to make sure that the formula is equally valid for both positive and negative values. I have defaulted the percentile_threshold to be 0.01 meaning that the rounding off is only triggered if even the smallest number is greater than 10.

The next line has the if statement which checks if the IsInt flag is set for the particular column we are considering.

The null imputation function is used to find and replace the nulls with the minimum value we identified earlier minus 1. The function returns 4 objects as follows

a) The dataframe with the nulls imputed for that specific column

b) A flag indicating whether nulls existed or not

c) A dictionary dictionary to keep track of what was imputed value in a particular column so that we can replace them again with NaN later.

d) The null_indices_list which contain the row numbers where the

Q. Why do we bother with null imputation?
Because in Python, integer datatypes donot support nulls. Therefore we will impute them with a number and then replace it with the nulls once the data conversion to Integer is successfully completed.

Once we are sure that the nulls have been imputed, we can convert some of the floats to int. This is done through another custom function called the float_to_int_conversion_function.

Lets scroll up to where the function is defined.

The if-else statements that you see here correspond to the core functionality where the min max ranges are compared with the maximum possible numerical representation within numpy for either signed or unsigned integers. For example, the first if statement checks if the minimum is positive in which case, its safe to assume that the numbers in that column can be represented by an unsigned integer. Next, we check if the maximum is less than the max value possible for 8 bit representation (which is 255) and so on.

Now lets go back to (line 292) where we left off in the reduce_mem_usage_function.

We bring back the nulls once the conversion is done

The else statement here gets activated if the values in the dataframe column are floats and we convert them to 32 bit representation. The reader is encouraged to tinker further with this function and optimize the float part similar to what we did for the integer.

Hi everyone, in this session, I will be describing where the plot_feature_importances function is used and how the function was coded out in the helper functions script.

We first used this function in Day 3 where we did feature engineering and needed to cut down a substantial number of features based on their feature importance. As you can see in this line, the function returns 2 objects – a dataframe where the features are normalized and the variable containing the number of features to keep assuming the cumulative signal we want to keep is 95%

There are 2 different plots generated by this function along with a printed output. The first one is the horizontal bar chart with the features on the y axis and the normalized feature importance on the x axis. To make it easier to visualize, only the top 15 features are plotted.

The 2nd plot contains the cumulative feature importance for all the features. The y axis contains the feature importance and the x axis contains the feature number after sorting in the descending order.

With this in mind, it becomes a bit easier to understand how the plots were generated. Lets head over to the helper scripts where this function is located.

The arguments consist of the dataframe containing only 2 columns – the feature name and their importance. The 2nd parameter is the threshold at which we want to cut off.

We first sort the dataframe’s rows based on the feature importance values so that they are in descending order. Next we create 2 additional columns for the normalized importance and the cumulative features.

Subsequently, we plot the bar charts with the feature names on the y axis and the normalized feature importance on the x axis. The key thing to note here is that you can tune the number of features to plot. Homework to the readers will be to parameterize this by inserting as another argument in the function definition.

The next function is cumulative importance plot generated by cumulative importance on the y axis and the feature number on the x axis.

In addition to the graphs, we are also interested in how many features does it take to reach a certain cumulative importance. This line achieves this by comparing the cumulative feature importance with the threshold that we provided outside the function. Subsequently we print this output

Finally we return the dataframe (which now has the 2 extra columns for normalized feature importance and cumulative importance) alongwith the number of features we want to keep that allows for 95% of the signal to be captured.

Hi everyone, in this session, I will be describing where the precision_at_recall_threshold_function is used and how the function was coded out in the helper functions script.

We first encountered this in Day1 when we finished training our model but were not satisfied with the out-of-the box capabilities provided by sklearn for computing classification metrics. We discussed that though the F1 score is sensitive to imbalanced data, it does not differentiate between a model with a poor Recall vs a poor Precision. As you know, the Precision and Recall vary depending on the binary classifier’s probability threshold.

For the credit card fraud detection example, we are interested in minimizing the False Negatives. Now once the model achieves a certain False Negative goal which can be equivalently phrased as achieving a certain Recall goal, the next question is how good is the model in keeping the False Positives low (which can be equivalently phrased as a high Precision)?

What we want to evaluate is the Precision at a given Recall threshold (assuming that business was previously able to define a target Recall). As we start comparing models, the question turns to “can we automatically generate the precision given a recall threshold and the predicted probabilities on the test data?” The single number can then be used to compare models against each other, analogous to how we compare using Accuracy. This is where the precision_at_recall_threshold_function comes in.

Lets go to the helper script to see how the function outputs the single number. The function takes in the target column, predicted probabilities and the Recall threshold for which we want the Precision of the model to be calculated. At its core, is the sklearn precision_recall_curve which computes the precision-recall pairs for different probability thresholds. It returns the precision and recall lists. The values are such that they are in ascending order for precision from 0 to 1.

The next line of code just filters the output list to figure out what is the highest Precision value at which the Recall exceeds 0.85 and returns it. This corresponds to the highest value in the truncated list which is why you see the -1 indexing.

Hi everyone, in this session, I will be describing where the tune_grid_search_funtion is used and how the function was coded out in the helper script.

We first encountered it in Day 4 of the credit card fraud detection notebook.

At a high level, this function generates the best hyperparameters along with associated metrics for a given classifier, hyperparameter search space, train/validation data and any keyword arguments.

Let me go to the helper script where this function is defined.

Let me dive into this function. I initialize a counter object called the hyperparam_results to store

a) internal_grid_search_scores (List of CV scores)

b) best_params (optimized hyperparams for the given classifier)

c) classification_metrics (binary classification metrics such as Accuracy, Precision, Recall, F1, AUC ROC, fbeta score and the precision_at_recall_threshold metric)

Next, you will notice the kwargs variable. In case you are unfamiliar with the keyword arguments syntax, Python function allows you to pass in additional arguments not specified at the time of the function definition. This property allows the programmer to make any future changes to the contents of the function while maintaining backwards compatibility.

In this case, my keyword arguments will contain the following params

i) is_custom_scorer (which is a binary flag)

ii) custom_scoring_function (which in this course will be the precision_at_recall_threshold_function)

iii) recall_threshold (which is set to 85% and can be any other value between 0 and 1 depending on the business requirements)

The if else construct gives the data scientist the flexibility to pass in the custom scorer if available. If not, then the usual GridSearchCV will work just fine.

I will start with the else statement which is easier to explain.

The search_object is an instantiation of the scikit learn’s GridSearchCV method. The latter takes in the classifier, the search space, the number of cross validation folds upon which the average performance will be taken and some optional parameters such as the verbosity (which prints out some logs) and the number of cores to use. I set the number of cores as 1 because I was getting some errors but I encourage the audience to set it as n =-1 to utilize all the cores effectively.

Now the default case is clear, lets go back to the if statement. The kwargs is checked for the ‘is_custom_scorer’ parameter. If the flag is set to True, then the custom_scoring_function is extracted from the kwargs. As seen above in the highlighted section of the code, I have deliberately created a custom scoring function instead of relying on GridSearchCV’s default scoring (which is accuracy) which wont work for imbalanced datasets. Notice that within the make_scorer() arguments, I have passed in 2 additional params to indicate to sklearn that my precision_at_recall_threshold_function needs probabilities instead of hard labels. And unlike loss functions (where greater_is_better = False), this metric needs to increase to signify improvement.

Now we are ready to leverage the GridSearchCV. The only difference this time (compared with the else statement) is that we are passing in scoring=custom_scorer in addition to the usual parameters previously discussed in the else statement.

After the search_object is instantiated above, we fit on the training data. Here the training set will be internally split into 3 folds with 2 of the folds used for training and the other fold used for internal cross validation.

Any scikit learn model object will contain several methods including mean_test_scores, best_estimator_ and the best_params_ which we will use outside of the function to predict on the test set. The mean_test_scores contains the average of the metrics over the 3 cross validation folds we have chosen.

The best_estimator_ will give you the optimized model itself which we can then pass onto our metrics_store_function whose only job is to generate the binary classification metrics on our validation set.

If I quickly scroll upto that function, you will notice all its doing is calling the respective sklearn methods on the optimized model and then returning the dictionary which contains these 5 metrics.

Now let me go back to the tune_grid_search_function

Once the classification_metrics dictionary is returned, I have added in a couple of additional parameters for the fbeta score and the precision_at_recall metric. The F2 score is a generalized version of the F1 score which I encourage the audience to check out.

The following if statement checks if the kwargs’ custom_scoring_function is the precision_at_recall_threshold function. If so, then the recall_threshold is fetched (in our case its 85%)

The classification_metrics dictionary then stores the result with the appropriate suffix by calling the precision_at_recall_threshold_function

Finally, we will be storing the classification_metrics dictionary in the hyperparam_results dictionary to be returned by our function

Congratulations on finishing the course! You should now be able to take any binary classification dataset and build a series of predictive models of increasing complexity over 5 iterations.

In addition to a systematic approach to developing Data Science projects, you also learnt the following

1. Automated detection of bad columns in our raw data (Day 1)

2. Creating your own metric for imbalanced datasets (Day 1)

3. Four Data Resampling techniques (Day 2)

4. Handling Nulls (Day 2)

5. Two Feature Engineering techniques (Day 3)

6. Four Feature Reduction techniques (Day 3)

7. Memory footprint reduction (Day 3)

8. Setting a custom scoring function inside the GridSearchCV (Day 4)

9. Changing the default scoring metric for XGBoost (Day 5)

10. Building meta-model (Day 5)

In many industries where the application of data science is fairly new, business stakeholders are happy to get anything to work and may even ask you to wrap up the modelling part after the very first week because the performance maybe good enough to get the project going. By iteratively adding complexity to the model, you make sure that the incremental effort is worth the extra accuracy.

This course has provided you with a framework where you cover all the bases in the shortest possible time while keeping the final model fairly simple. Hope you found the materials useful. Please feel free to use the source code in your own projects. If you have any questions, please feel free to post in the comments section below the video and I will do my best to answer you. If you enjoyed the course, please spread the word on social media! Thank you for your time.