What does model deterioration look like?

Kirill Eremenko
A free video tutorial from Kirill Eremenko
Data Scientist
4.5 instructor rating • 118 courses • 1,546,052 students

Lecture description

In this lecture, you will see how models are deteriorating over time. You can find visuals and explanations on how it is going.

Learn more from the full course

Data Science A-Z™: Real-Life Data Science Exercises Included

Learn Data Science step by step through real Analytics examples. Data Mining, Modeling, Tableau Visualization and more!

21:12:10 of on-demand video • Updated September 2020

  • Successfully perform all steps in a complex Data Science project
  • Create Basic Tableau Visualisations
  • Perform Data Mining in Tableau
  • Understand how to apply the Chi-Squared statistical test
  • Apply Ordinary Least Squares method to Create Linear Regressions
  • Assess R-Squared for all types of models
  • Assess the Adjusted R-Squared for all types of models
  • Create a Simple Linear Regression (SLR)
  • Create a Multiple Linear Regression (MLR)
  • Create Dummy Variables
  • Interpret coefficients of an MLR
  • Read statistical software output for created models
  • Use Backward Elimination, Forward Selection, and Bidirectional Elimination methods to create statistical models
  • Create a Logistic Regression
  • Intuitively understand a Logistic Regression
  • Operate with False Positives and False Negatives and know the difference
  • Read a Confusion Matrix
  • Create a Robust Geodemographic Segmentation Model
  • Transform independent variables for modelling purposes
  • Derive new independent variables for modelling purposes
  • Check for multicollinearity using VIF and the correlation matrix
  • Understand the intuition of multicollinearity
  • Apply the Cumulative Accuracy Profile (CAP) to assess models
  • Build the CAP curve in Excel
  • Use Training and Test data to build robust models
  • Derive insights from the CAP curve
  • Understand the Odds Ratio
  • Derive business insights from the coefficients of a logistic regression
  • Understand what model deterioration actually looks like
  • Apply three levels of model maintenance to prevent model deterioration
  • Install and navigate SQL Server
  • Install and navigate Microsoft Visual Studio Shell
  • Clean data and look for anomalies
  • Use SQL Server Integration Services (SSIS) to upload data into a database
  • Create Conditional Splits in SSIS
  • Deal with Text Qualifier errors in RAW data
  • Create Scripts in SQL
  • Apply SQL to Data Science projects
  • Create stored procedures in SQL
  • Present Data Science projects to stakeholders
English Model Deterioration. This is going to be an exciting topic, maybe even unique On the whole internet I'm not going to claim that but maybe because I haven't seen it anywhere else . We're going to find out how models deteriorate What does it look like when a model deteriorates All right So we have our cup curve our cup curve is a very powerful tool that allows us to see our model to have a graphical representation of our model which includes as well how well the model is performing. And remember we had this rule of thumb that you look at the 50 percent and you what crosses your model line where your cap line is crossed by this vertical 50 percent line So and that allows you to determine how well your model is performing how good this model is And we had this rule of thumb where anything below 60 percent is a rubbish model Anything between 60 and 70 is a poor model 70 to 80 is good, 80 to 90 very good, 90 to 100 too good practically Very unlikely that that can be achieved. So let's observe a model over time and see how it will deteriorate. It's going to be fun. All right So we start off with our model and at the top in the top left corner we're going to say how old it is this a brand new model you just delivered it You're very excited about it Very proud And it's performing at 81 percent at the 50 percent mark So it's a very good model And let's see what happens to it next, six months down the track. How well is it performing a bit worse. 75 percent So it is just a good model it's not a very good model anymore. It's just a good model but still still pretty solid. It's giving good results good predictions you can derive value from it. Then 12 months down the track. What do we see here - 69 percent It just crossed into the poor path or the poor thresholds so as you remember a 60 to 70 is kind of like a poor model but it's still around the 70 percent mark still may be driving insides not as good as when it was brand new 81 percent but it's kind of still do. alright, fairly alright right but venturing into the poor territory, another six months down the track after 18 months after deployment holes are moldering only 63 percent. That is definitely a poor model can really derive a lot of value from that. But you know we we are not maintaining it so let's see what happens six months later 24 months down the track. Fifty five percent. And that is already a red flag a warning. Fifty five percent is not much better than a random coin toss just a 5 percent increase on top of a coin toss. And like what's the point of having that model was the point of wasting processing power and actually using it in business processes. This is kind of already useless. That's why we call them rubbish models. Total rubbish model. And believe it or not six months later even this can happen. And I've actually seen one of these models where it's driving a negative driving negative value for the business 42 percent when if you use a random coin toss you'll actually get a better result. Definitely this model isn't just rubbish it's actually doing worse for the business. So that's how a model can ditcher it over time. The time frames used over here are completely out of the blue so it might not be a six month period between each one of these might be 12 month period might be a three year period depends on the industry it could be faster, it could be a month could be two months depends on the dynamics in the industry in the company in the population in the country and the economic environment and so on. So it depends on what you're modeling as well so if you're not modeling populace like people and your customer base and so on if you're modeling defects in a factory then your model might last longer but then if something changes, for instance, the plastic with the materials change then your model start might start deteriorating. So it all depends on the circumstances. But models do deteriorate over time and we'll actually find out why in the next tutorial Hope you enjoyed this one. I definitely find it quite visual and hopefully it illustrates the concept of moral degeneration quite well.