Udemy
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
Development
Web Development Data Science Mobile Development Programming Languages Game Development Database Design & Development Software Testing Software Engineering Development Tools No-Code Development
Business
Entrepreneurship Communications Management Sales Business Strategy Operations Project Management Business Law Business Analytics & Intelligence Human Resources Industry E-Commerce Media Real Estate Other Business
Finance & Accounting
Accounting & Bookkeeping Compliance Cryptocurrency & Blockchain Economics Finance Finance Cert & Exam Prep Financial Modeling & Analysis Investing & Trading Money Management Tools Taxes Other Finance & Accounting
IT & Software
IT Certification Network & Security Hardware Operating Systems Other IT & Software
Office Productivity
Microsoft Apple Google SAP Oracle Other Office Productivity
Personal Development
Personal Transformation Personal Productivity Leadership Career Development Parenting & Relationships Happiness Esoteric Practices Religion & Spirituality Personal Brand Building Creativity Influence Self Esteem & Confidence Stress Management Memory & Study Skills Motivation Other Personal Development
Design
Web Design Graphic Design & Illustration Design Tools User Experience Design Game Design Design Thinking 3D & Animation Fashion Design Architectural Design Interior Design Other Design
Marketing
Digital Marketing Search Engine Optimization Social Media Marketing Branding Marketing Fundamentals Marketing Analytics & Automation Public Relations Advertising Video & Mobile Marketing Content Marketing Growth Hacking Affiliate Marketing Product Marketing Other Marketing
Lifestyle
Arts & Crafts Beauty & Makeup Esoteric Practices Food & Beverage Gaming Home Improvement Pet Care & Training Travel Other Lifestyle
Photography & Video
Digital Photography Photography Portrait Photography Photography Tools Commercial Photography Video Design Other Photography & Video
Health & Fitness
Fitness General Health Sports Nutrition Yoga Mental Health Dieting Self Defense Safety & First Aid Dance Meditation Other Health & Fitness
Music
Instruments Music Production Music Fundamentals Vocal Music Techniques Music Software Other Music
Teaching & Academics
Engineering Humanities Math Science Online Education Social Science Language Teacher Training Test Prep Other Teaching & Academics
AWS Certification Microsoft Certification AWS Certified Solutions Architect - Associate AWS Certified Cloud Practitioner CompTIA A+ Cisco CCNA Amazon AWS CompTIA Security+ AWS Certified Developer - Associate
Photoshop Graphic Design Adobe Illustrator Drawing Digital Painting InDesign Character Design Figure Drawing Canva
Life Coach Training Neuro-Linguistic Programming Personal Development Mindfulness Meditation Personal Transformation Life Purpose Emotional Intelligence Neuroscience
Web Development JavaScript React CSS Angular PHP WordPress Node.Js Python
Google Flutter Android Development iOS Development Swift React Native Dart Programming Language Mobile Development Kotlin SwiftUI
Digital Marketing Google Ads (Adwords) Social Media Marketing Google Ads (AdWords) Certification Marketing Strategy Internet Marketing YouTube Marketing Email Marketing Google Analytics
SQL Microsoft Power BI Tableau Business Analysis Business Intelligence MySQL Data Modeling Data Analysis Big Data
Business Fundamentals Entrepreneurship Fundamentals Business Strategy Online Business Business Plan Startup Blogging Freelancing Home Business
Unity Game Development Fundamentals Unreal Engine C# 3D Game Development C++ 2D Game Development Unreal Engine Blueprints Blender
30-Day Money-Back Guarantee
Development Data Science Machine Learning

Testing and Monitoring Machine Learning Model Deployments

ML testing strategies, shadow deployments, production model monitoring and more
Rating: 4.5 out of 54.5 (169 ratings)
2,467 students
Created by Christopher Samiullah, Soledad Galli
Last updated 10/2020
English
English [Auto]
30-Day Money-Back Guarantee

What you'll learn

  • Machine Learning System Unit Testing
  • Machine Learning System Integration Testing
  • Machine Learning System Differential Testing
  • Shadow Deployments (also known as Dark/Decoy launches)
  • Statistical Techniques for Assessing Shadow Deployments
  • Monitoring ML System with Metrics (Prometheus & Grafana)
  • Monitoring ML Systems with Logs (Kibana & the Elastic Stack)
  • The Theory Around Continuous Delivery for Machine Learning

Course content

12 sections • 92 lectures • 8h 16m total length

  • Preview02:35
  • Preview01:36
  • How to Approach This Course (Important)
    03:26
  • All Notes & Slides For This Course
    00:03
  • FAQ: I would like to learn more about the topics not covered
    00:42

  • Deploying a Model to Production
    08:31
  • Course Scenario: Predicting House Sale Price
    09:27
  • Setup A: Python Installation (Important)
    03:47
  • Setup B: Git and Github Setup (Advanced users can skip)
    03:02
  • Course Github Repo & Data
    02:38
  • Download dataset and Github repo: links and guidelines
    00:57
  • Setup C: Jupyter Notebook Setup
    02:13
  • Setup D: Install Notebook Dependencies
    02:19
  • Introduction to the Dataset & Model Pipeline
    13:21
  • Preview05:51
  • Additional Links and Resources
    00:28

  • Section Overview
    00:48
  • Testing Focus in This Course
    01:26
  • Preview03:44
  • The Value of Testing
    1 question
  • Testing Theory
    03:47
  • Testing Machine Learning Systems (Important)
    06:31
  • Setup A: Install Requirements
    00:13
  • Hands-on Assignment: Unit Testing Input Data
    1 question
  • Hands-on Assignment: Unit Testing Data Engineering Code
    1 question
  • Assignment 3: Hands-on Assignment: Unit Testing Model Quality
    1 question
  • Assignment 4: Hands-on Assignment: Unit Testing Model Config
    1 question
  • Wrap Up
    00:26

  • Section Overview
    00:45
  • Code Conventions
    02:26
  • Pytest
    11:49
  • Setup - Kaggle Data
    03:22
  • Download the data set - Text Summary
    00:28
  • Setup 2 - Tox
    05:47
  • Code Base Overview
    13:41
  • Preprocessing & Feature Engineering Unit Testing Theory - Why Do This?
    03:24
  • Preprocessing & Feature Engineering Unit Testing
    11:06
  • Quick note on git hygiene for the course
    00:12
  • Model Config Unit Testing Theory - Why Do This?
    03:00
  • Model Config Unit Testing
    09:57
  • Input Data Testing Theory - Why Do This?
    03:06
  • Input Data Unit Testing
    08:35
  • Model Quality Unit Testing Theory - Why Do This?
    02:19
  • Model Quality Unit Testing
    10:10
  • Quick Lecture on Tooling Improvements
    02:41
  • Wrap Up
    01:41

  • Section Overview
    00:45
  • Quick Docker Recap
    06:09
  • Why Use Docker?
    07:24
  • Introduction to Docker Compose
    04:28
  • Docker Quiz
    3 questions
  • Docker & Docker Compose Installation
    05:56
  • Windows Specific Docker Issue
    03:48
  • Hands on Exercise: Basic Docker Compose
    1 question
  • Docker Space Consumption Tips
    00:18

  • Section Overview
    00:40
  • API Conceptual Guide
    02:16
  • Overview of the Codebase
    06:47
  • Using our Open API Spec Part 1
    01:55
  • WINDOWS SPECIFIC SETUP
    00:03
  • Using our Open API Spec Part 2
    02:56
  • Integration Testing Theory
    01:52
  • WORK AROUND LECTURE - 32 bit Operating Systems
    00:14
  • Integration Testing Hands-On Code
    10:21
  • A note on benchmark integration tests
    01:33

  • Section Overview
    00:32
  • Differential Testing Theory
    03:19
  • Differential Testing Implementation
    07:37

  • Section Overview
    00:44
  • Shadow Mode Theory
    04:23
  • Testing Models in Production
    09:32
  • Tests in Shadow Deployments
    15:08
  • Code Overview - DB Setup
    13:13
  • WINDOWS port mapping
    00:12
  • Gotcha: breaking changes in sqlalchemy_utils
    00:10
  • Setup Tests for Shadow Mode
    11:40
  • Shadow Mode - Asynchronous Implementation
    04:25
  • Populate Database with Shadow Predictions
    05:22
  • Jupyter Demo - Setup
    05:02
  • Jupyter Demo - Tests in Shadow Mode
    14:18

  • Section Overview
    01:36
  • Why Monitor?
    05:34
  • Monitoring Theory
    08:29
  • Metrics for Machine Learning Systems
    06:03
  • Prometheus & Grafana Overview
    06:42
  • [WINDOWS ONLY] Additional Setup
    02:28
  • Basic Prometheus Setup - Hands-on
    05:33
  • Adding Metrics - Hands-on
    08:22
  • Adding Grafana - Hands-on
    07:21
  • Infrastructure Metrics - Hands-on
    06:44
  • Adding Metrics Monitoring to Our Example Project
    07:30
  • Creating an ML System Grafana Dashboard
    15:44

  • Monitoring Logs for ML - Theory
    04:03
  • The Elastic Stack (Formerly ELK) - Overview
    04:41
  • Kibana Hands-on Exercise
    09:43
  • Integrating Kibana into The Example Project
    09:36
  • Setting Up a Kibana Dashboard for Model Inputs
    14:03

Requirements

  • Comfortable with Python
  • Familiar with Scikit-Learn, Pandas, Numpy
  • Comfortable with Data Science Fundamentals
  • Can use Git version control
  • Basic knowledge of Docker
  • This is an advanced course

Description

Learn how to test & monitor production machine learning models.


What is model testing?

You’ve taken your model from a Jupyter notebook and rewritten it in your production system. Are you sure there weren’t any mistakes when you moved from the research environment to the production system? How can you control the risk before your deployment? ML-specific unit, integration and differential tests can help you to minimize the risk.


What is model monitoring?

You’ve deployed your model to production. OK now what? Is it working as you expect? How do you know? By monitoring models, we can check for unexpected changes in:

  • Incoming data

  • Model quality

  • System operations

When we think about data science, we think about how to build machine learning models, which algorithm will be more predictive, how to engineer our features and which variables to use to make the models more accurate. However, how we are going to actually test & monitor these models in a production system is often neglected, . Only when we can effectively monitor our production models can we determine if they are performing as we expect.


Why take this course?

This is the first and only online course where you can learn how to test & monitor machine learning models. The course is comprehensive, and yet easy to follow. Throughout this course you will learn all the steps and techniques required to effectively test & monitor machine learning models professionally.

In this course, you will have at your fingertips the sequence of steps that you need to follow to test & monitor a machine learning model, plus a project template with full code, that you can adapt to your own models.


What is the course structure?

Part 1: Testing

The course begins from the most common starting point for the majority of data scientists: a Jupyter notebook with a machine learning model trained in it. We gradually build up the complexity, testing the model first in the Juyter notebook and then in a realistic production code base. Hands-on exercises are interspaced with relevant and actionable theory.

Part 2: Shadow Mode

We explain the theory & purpose of deploying a model in shadow mode to minimize your risk, and walk you through an example project setup.

Part 3: Monitoring

We take you through the theory & practical application of monitoring metrics & logs for ML systems.


Important:

  • This course does not cover model deployment (we have a separate course dedicated to that topic)


Who are the instructors?

We have gathered a fantastic team to teach this course. Sole is a leading data scientist in finance and insurance, with 3+ years of experience in building and implementing machine learning models in the field, and multiple IT awards and nominations. Chris is a tech lead & ML software engineer with enormous experience in building APIs and deploying machine learning models, allowing business to extract full benefit from their implementation and decisions.


Who is this course for?

  • Data Scientists who want to know how to test & monitor their models beyond in production

  • Software engineers who want to learn about Machine Learning engineering

  • Machine Learning engineers who want to improve their testing & monitoring skills

  • Data Engineers looking to transition to ML engineering

  • Lovers of open source technologies


How advanced is this course?

This is an advanced level course, and it requires you to have experience with Python programming and git. How much experience? It depends on how much time you would like to set aside to go ahead and learn those concepts that are new to you. To give you an example, we will work with Python environments, we will work with object oriented programming, we will work with the command line to run our scripts, and we will checkout code at different stages with git. You don’t need to be an expert in all of these topics, but you need a reasonable working knowledge. We also work with Docker a lot, though we will provide a recap of this tool.

For those relatively new to software engineering, the course will be challenging. We have added detailed lecture notes and references, so we believe that those missing some of the prerequisites can take the course, but keep in mind that you will need to put in the hours to read up on unfamiliar concepts. On this point, the course slowly increases in complexity, so you can see how we pass, gradually, from the familiar Jupyter notebook, to the less familiar production code, using a project-based approach which we believe is optimal for learning. It is important that you follow the code, as we gradually build it up.


Still not sure if this is the right course for you?

Here are some rough guidelines:

Never written a line of code before: This course is unsuitable

Never written a line of Python before: This course is unsuitable

Never trained a machine learning model before: This course is unsuitable. Ideally, you have already built a few machine learning models, either at work, or for competitions or as a hobby.

Never used docker before: The second part of the course will be very challenging. You need to be ready to read up on lecture notes & references.

Have only ever operated in the research environment: This course will be challenging, but if you are ready to read up on some of the concepts we will show you, the course will offer you a great deal of value.

Have a little experience writing production code: There may be some unfamiliar tools which we will show you, but generally you should get a lot from the course.

Non-technical: You may get a lot from just the theory lectures, so that you get a feel for the challenges of ML testing & monitoring, as well as the lifecycle of ML models. The rest of the course will be a stretch.


To sum up:

With more than 70 lectures and 8 hours of video this comprehensive course covers every aspect of model testing & monitoring. Throughout the course you will use Python as your main language and other open source technologies that will allow you to host and make calls to your machine learning models.


We hope you enjoy it and we look forward to seeing you on board!


Who this course is for:

  • Data Scientists who want to know how to test & monitor their models beyond in production
  • Software engineers who want to learn about Machine Learning engineering
  • Machine Learning engineers who want to improve their testing & monitoring skills
  • Data Engineers looking to transition to ML engineering
  • Lovers of open source technologies

Featured review

João Santos
João Santos
52 courses
7 reviews
Rating: 5.0 out of 53 months ago
Great content! nice Explanations, great code. This is not a beginners course and there is so much information trough the course... Either on the slides or on the lecture notes there is really great much information that brought my knowledge to the next level.

Instructors

Christopher Samiullah
Machine Learning Engineer
Christopher Samiullah
  • 4.5 Instructor Rating
  • 2,690 Reviews
  • 16,007 Students
  • 2 Courses

My name is Chris. I'm a professional software engineer from the UK. I've been writing code for 8 years, and for the past three years, I've focused on scaling machine learning applications. I've done this at fintech and healthtech companies in London, where I've worked on and grown production machine learning applications used by hundreds of thousands of people. I've built and maintained machine learning systems which make credit-risk and fraud detection judgements on over a billion dollars of personal loans per year for the challenger bank Zopa. I currently work on systems for predicting health risks for patients around the world at Babylon Health.


In the past, I've worn a variety of hats. I worked at a global healthcare company, Bupa, which included being a core developer on their flagship website, and three years working in Beijing setting up mobile, web and IT for medical centers in China. Whilst in Beijing, I ran the Python meetup group, mentored a lot of junior developers, and ate a lot of dumplings. I enjoy giving talks at engineering meetups, building systems that create value, and writing software development tutorials and guides. I've written on topics ranging from wearable development, to internet security, to Python web frameworks.


I'm passionate about teaching in a way that minimizes the time between "ah hah" moments, but doesn't leave you Googling every other word. Complexity is necessary for application in the real world, but too much complexity is overwhelming and counter-productive. I will help you find the right balance.


Feel free to connect on LinkedIn (very active) or Twitter (getting more active in 2021)

Soledad Galli
Lead Data Scientist
Soledad Galli
  • 4.6 Instructor Rating
  • 5,857 Reviews
  • 25,312 Students
  • 6 Courses

Soledad Galli is a lead data scientist and founder of Train in Data. She has experience in finance and insurance, received a Data Science Leaders Award in 2018 and was selected “LinkedIn’s voice” in data science and analytics in 2019. Sole is passionate about sharing knowledge and helping others succeed in data science.

As a data scientist in Finance and Insurance companies, Sole researched, developed and put in production machine learning models to assess Credit Risk, Insurance Claims and to prevent Fraud, leading in the adoption of machine learning in the organizations.

Sole is passionate about empowering people to step into and excel in data science. She mentors data scientists, writes articles online, speaks at data science meetings, and teaches online courses on machine learning.

Sole has recently created Train In Data, with the mission to facilitate and empower people and organizations worldwide to step into and excel in data science and analytics.

Sole has an MSc in Biology, a PhD in Biochemistry and 8+ years of experience as a research scientist in well-known institutions like University College London and the Max Planck Institute. She has scientific publications in various fields such as Cancer Research and Neuroscience, and her research was covered by the media on multiple occasions.

Soledad has 4+ years of experience as an instructor in Biochemistry at the University of Buenos Aires, taught seminars and tutorials at University College London, and mentored MSc and PhD students at Universities.

Feel free to contact her on LinkedIn.


========================


Soledad Galli es científica de datos y fundadora de Train in Data. Tiene experiencia en finanzas y seguros, recibió el premio Data Science Leaders Award en 2018 y fue seleccionada como "la voz de LinkedIn" en ciencia y análisis de datos en 2019. A Soledad le apasiona compartir conocimientos y ayudar a otros a tener éxito en la ciencia de datos.


Como científica de datos en compañías de finanzas y seguros, Sole desarrolló y puso en producción modelos de aprendizaje automático para evaluar el riesgo crediticio, automatizar reclamos de seguros y para prevenir el fraude, facilitando la adopción del aprendizaje de máquina en estas organizaciones.


A Sole le apasiona ayudar a que las personas aprendan y se destaquen en ciencia de datos, es por eso habla regularmente en reuniones de ciencia de datos, escribe varios artículos disponibles en la web y crea cursos sobre aprendizaje de máquina.


Sole ha creado recientemente Train In Data, con la misión de ayudar a las personas y organizaciones de todo el mundo a que aprendan y se destaquen en la ciencia y análisis de datos.


Sole tiene una maestría en biología, un doctorado en bioquímica y más de 8 años de experiencia como investigadora científica en instituciones prestigiosas como University College London y el Instituto Max Planck. Tiene publicaciones científicas en diversos campos, como la investigación contra el Cáncer y la Neurociencia, y sus resultados fueron cubiertos por los medios en múltiples ocasiones.


Soledad tiene más de 4 años de experiencia como instructora de bioquímica en la Universidad de Buenos Aires, dio seminarios y tutoriales en University College London, en Londres, y fue mentora de estudiantes de maestría y doctorado en diferentes universidades.


No dudes en contactarla en LinkedIn.

  • Udemy for Business
  • Teach on Udemy
  • Get the app
  • About us
  • Contact us
  • Careers
  • Blog
  • Help and Support
  • Affiliate
  • Impressum Kontakt
  • Terms
  • Privacy policy
  • Cookie settings
  • Sitemap
  • Featured courses
Udemy
© 2021 Udemy, Inc.