Data Science Methodology in Action using Dataiku

This lecture provides three options for setting up your Dataiku Sandbox.  You will be using this sandbox for course exercises.

In this lecture, we will over our data science methodology

We have augmented existing data science methodology to include steps needed for optimizing your machine learning model.

In this lecture, we will provide overview of our 7 steps of data science methodology

In this lecture we will provide an overview of Data Science Methodology Step 1 - Define Project. Here we identify the use case, user persona, scope, deliverables, exit criteria and project plan.

In this lecture, we will introduce our COVID analytics project

Data understanding is very critical steps in driving data science methodology. You have heard the term – Garbage In and Garbage Out.

In this section, we will review various characteristics of big data and how one should go about understanding and selecting right data sources and data contents for a use case from data science perspective for our COVId Use case

In this lesson, we will cover  activities required to complete Step 2 - Describe Data

This lecture will provide details on Task 1 on how to  Load data  using Dataiku,  We will use Covid Analytics project as an example.

This lecture will provide details on Task 2 on how to classify data  using Dataiku,  We will use Covid Analytics project as an example.

This lecture will provide details on Task 3 on how to  get description of columns in your dataset using Dataiku,  We will use Covid Analytics project as an example.

This lecture will provide details on Task 4 on how to  verify data quality  using Dataiku,  We will use Covid Analytics project as an example.

Many data sources provide data at high volume or velocity, making it hard to model.  This lectures covers how data can be reduced for modeling.

Feature engineering enables us to create meaningful features that can be used as input for modeling and visualization. 

This lecture provides an overview of how data is synthesized into Analytics Base Table.

This lecture shows selection of datasets for Covid Analytics Project.

In this lecture, we will cover  Task 2 – Reformat and Filter Dataset of our Step 3 – Prepare dataset of our data science methodology

Filter – helps you select rows you will use in your model. Reformat allows you to change the format in which a specific column is stored.

In this lecture, let’s move to Task 3 – Transform data of step 3- prepare data

In this lecture, we will cover Task 4 – Group of step3 - prepare data.

Here we will learn how to use Dataiku to group a set rows and compute aggregations for the grouped rows.

Let’s move to Task 6 – Engineer features of our Step 3 – Prepare dataset of our data science methodology

Task Engineer features– modeling requires a new set of attributes which are computed using existing attributes, for example, a model of COVID spread using incremental deaths as indicator of COVID severity. We need to compute incremental deaths by subtracting yesterday’s total deaths from today’s total deaths.

Let’s move to Task 6 – Merge dataset Dataset of our Step 3 – Prepare dataset of our data science methodology

Here we will discuss the Merge function

Merge function– allows us to merge two files with complimentary data using a common index. For example, we are collecting social mobility from Google and COVID indicators from USA_Facts. The two files need to be merged on respective indices for modeling.

In this lecture, we will introduce various modeling techniques.

In this lecture, we will cover  task 1 where we set up the dataset for classification.

In this lecture, we will cover Task 2. classification using clustering techniques.

In this lecture, we will cover Task 3. Prediction set-up activity.

In this lecture, we will cover various prediction model development. Prediction techniques will help us predict a future value based on historical data.

In this lecture we will provide overview on Prediction evaluation measures

In the lecture we will provide demonstration on how to evaluate prediction models

In this lecture we will cover Step 6 – deploy model of our data science methodology on how to engage users to support deployment process.

We will be covering how you can engage users to validate your model results and utilizing the results of AI models, you can incorporate decisions in your business processes and automate them.

With the explosion of smart devices and the IoT ecosystem, the emphasis is rapidly shifting from business intelligence to intelligent action. How do we deploy the models we developed in the previous sections. In this lecture we will discuss deployment options, emerging standards and associated Python libraries.

In this lecture, we will provide overview of Step 7 – Optimize Model, specifically, what guard-rails should be in place for continuous monitoring and learning of AI system for production use

How do you monitor your model for incorporating user feedbacks and incorporate field learning.