Data Science in Action using Python

In this lecture, we will provide overview of two we will be using for all class and project  assignments. These environments include  - Anaconda or Google Colaboratory

In this lecture, we will provide overview of how to set-up Anaconda environment on your laptop

In this lecture, we will provide overview of how to set-up Google Colab environment. 

In this lecture, We will cover how various Python data and notebooks are organized to support development of our COVID Use case.

We will also cover some basics on Python notebooks.

In this lecture, we will provide overview of our 7 step data science methodology

In this section we will provide overview of our 7 steps based enhanced data science methodology.

We will be covering key tasks and deliverables associated with each of  these 7 steps of our methodology.

In this lecture we will provide an overview of Data Science Methodology Step 1 - Define Project.  Here we identify the use case, user persona, scope, deliverables, exit criteria and project plan.

In this lecture, we will describe how to provide overview of  Task 1 of Step 1: Describe use case of our data science methodology

Data understanding is very critical steps in driving data science methodology. You have heard the term – Garbage In and Garbage Out.

In this section, we will review various characteristics of big data and how one should go about understanding and selecting right data sources and data contents for a use case from data science perspective for our COVId Use case

This lectures describes the tasks and deliverables associated with Describe Data

Step 2 of our methodology for understanding Data includes 5 major tasks.  In this lecture, we will cover Task 1. How to describe data sources

In this lecture we will cover Step 2 - Task 2. How to Describe Data using 4 major "V" of Big Data

In this lecture, we will cover Step 2 - Task 3. How to Explore /Classify Data into different categories like structured or unstructured, numeric or alpha, cumulative or incremental, date or any other factor

In this lecture, we will cover Step 2 - Task 4. Collect basic statistics on datasets such as min, max, average, median, mode to understand data distribution

In this lecture, we will cover Step 2 - Task 5. how to verify Data Quality for your datasets, for example, how many records are missing or invalid in the data set

Purpose of the Step 3 – prepare data is multi-fold

•Identify initial issues

•Analyze issues

•Repair or reject the data-fields with Documented assumptions and thereby

•Improve Data Transformation & Quality issues.

There are many key tasks associated with the Step 3 of Data Preparation process, which we will cover in this lecture. There are many advanced tasks in a typical data science project. As you get comfortable with these basic tasks, you can do many projects with simple and intermediate complexity, and then you can explore and add other advanced tasks to your toolkit.

In this lecture, we will cover Task 1.  Select Datasers associated with Step 3 - Prepare data of our data science methodology. Select Task helps you select columns you will use in your model

Task 2 – Filter Dataset of our Step 3 – Prepare dataset of our data science methodology helps you select rows you will use in your model.  We will first describe the assignment and then take you through the details.

Sometime the data is not correctly formatted. Task 3. Reformat data  helps you reformat the data in order to use it correctly in your model. For example, when we ingested the data, countyFIPS was read as a decimal in one of the files. In order to match it to other files, we need to drop the decimal.

Task 4. Transform Datasets is used when the data is not correctly organized. You may need to move your fields from columns to rows, or vice versa. For example, USA_Facts data has dates in the columns, and we will need them as rows to match other sources.

If the data is not sorted in the order needed for new attribute creation or modeling, it may need to be sorted, Task 5. Sort Datasets allows you to sort dataset in correct order, for example for incremental deaths computation, the data should be in chronological order.

Task 6. Feature Engineer allows you to create new set of attributes which are computed using existing attributes, for example, a model of COVID spread using incremental deaths as indicator of COVID severity. We need to compute incremental deaths by subtracting yesterday’s total deaths from today’s total deaths.

Task 7.  Merge datasets allows us to merge two files with complimentary data using a common index. For example, we are collecting social mobility from Google and COVID indicators from USA_Facts. The two files need to be merged on respective indices for modeling.

In this section, we will provide overview of Step 4- Develop Model of our data science methodology.  We will introduce various modeling techniques to you and Discuss how to build / execute those modeling techniques on our COVID use case.

Task 1. Prediction set up of Step 4 - Develop Model  introduces important set up step before running complex statistical algorithms. Prediction is a comparatively a simpler task if it was providing evidence in support of a well-formed hypothesis. As we formulate this hypothesis, data visualization is a major part of hypothesis confirmation and formulation. While visualization does not deliver the results, it provides a significant hunch that can then be further solidified via statistical processes.

In this lecture, We will cover Task 2 - Time series forecasting of our Step 4- Develop Model.   We will take all the datasets built so far and will apply time series forecasting modeling techniques. We will introduce the libraries and how to invoke the libraries. However, we will not detail how to evaluate, revise and optimize the model. When we get to Step 5. Evaluate model, we will revisit these models and show how they can be evaluated and revised.

In this lecture, we will cover various data modeling regression techniques. We will revisit the causal model and the associated hypotheses and use modeling to either find meaningful relationships in support of our hypotheses or reject the hypotheses. Assuming we have a set of validated hypotheses, we will use the resulting model to generate forecasts.

In this lecture, we will cover Task 4. classification using clustering techniques. These techniques offer ways to develop classification models using unsupervised learning. 

In this lecture, we will cover Task 5. classification using clustering techniques. These techniques offer ways to develop classification models using unsupervised learning.

In this lecture, we will cover Task 6. connecting the dots with illustration using network graphs.. These techniques offer ways to synthesize insights.  Once we create the graph, these graphs can be used for analyzing patterns of connectivity.

In this lecture, we will introduce a couple of graph analysis techniques and show you how these techniques can be implemented using Python libraries.

Step 5 of our our data science methodology involves providing measurements for various AI models we covered in previous section.  In this lecture, we will cover measurements for Regression models.  We will use county-level Covid data to illustrate how Regression models can be evaluated.

In this lecture, we will first discuss how classification is evaluated. Then we will pick up the classification model we developed in the Step 4- develop model section and go through associated evaluation metrics.

In this lecture, we will provide evaluation for  Forecasting models. We will resume our forecasting modeling activities, picking up ARIMA model and will focus this time far more on evaluating the accuracy of prediction. We will start with an overview of the ARIMA model and then describe how its parameterized and evaluated. Next, we will use an illustrative example from Covid data set to apply ARIMA model and evaluate its performance. Finally, we have a class assignment for you to try developing your ARIMA model and its evaluation.

In this lecture we will cover Step 6 – deploy model of our data science methodology on how to engage users to support deployment process.  We will be covering how you can engage users to validate your model results and utilizing the results of AI models, you can incorporate decisions in your business processes and automate them.

With the explosion of smart devices and the IoT ecosystem, the emphasis is rapidly shifting from business intelligence to intelligent action. How do we deploy the models we developed in the previous sections. In this lecture,  we will discuss deployment options, emerging standards and associated Python libraries.

In this lecture, We will cover Step 7 – Monitor model on how to engage users to support continuous deployment process.