Data Science Methodology

This lecture accomplishes three important objectives. It reviews the Business Intelligence methodology and challenges associated with using it in the context of big data and AI. We offer important revisions to the methodology in response to these challenges.

In this lecture we will illustrate the first step - Define Project using Class project. 

As Data is the most important resource in today’s enterprise, in this lecture, we will cover what is Big Data and how do you categorize big data

In this lecture, we will cover

•What are the different types of data and Data Sources and what are the options for

•Data Deployment in today’s explosive data intensive environment.

In this lecture, we will conclude the section with an example to illustrate how one goes about data understanding using COVID as an example use case

Data reduction deals with taking big data and reducing its size and velocity for data science project.

In this lecture, we will define feature engineering and will discuss some of the operations needed to perform feature engineering.

By now, you have seen how input data sources can be manipulated to create a series of reduced tables, with new variables created via feature engineering. Before moving to modeling or visualization, this data needs to be consolidated into a smaller set of tables, also known as analytics base tables. In this lecture, we will discuss how the data we have manipulated can be synthesized into an analytics base table.

The focus of the section is to discuss various tasks associated with running your model  to support Step 4- Develop Model of our Data Science methodology.  We will also  introduce you to various AI and machine learning modeling techniques

In this lecture, we will cover Time series forecasting - where the variables vary over time and our task is to find patterns across time periods

In this lecture, we will cover Classification – where the purpose of modeling is to identify classification of a dependent variable based on a set of independent variables.

In this lecture, we will cover Regression modeling techniques.  Regression extends classifications to establish a mathematical relationship between a set of dependent variable to a set of independent variables.

In this lesson, We will define  Model measurements – what measurements do you need on your model and how do you select the appropriate measurement for your model.

In this lecture, we will cover various user personas needed for model management and evaluating results of your AI models,

In this lecture, We will start with Deployment framework, how one goes about building/deploying the Big Data driven AI engagement

There are several deployment options for you AI models

• First we will cover how to use Queries and Dashboards for displaying results of your AI models. Users can be engaged for viewing results of your data models and highlights issues with data or model results as needed.

• Then we will cover next deployment options known as Scoring Engines. A scoring engine can can use or aggregate results from many deployed AI models to provide insights on incoming data sources

• Next, we will cover how do you automate your business processes through Intelligent Robotic Process Automation commonly known as RPA utilizing results from a scoring engine

• Finally, we will cover Conversational AI, as Conversational AI is rapidly becoming a very popular way of engaging your users. Here the focus is on using conversational AI as a user interface for the model that you developed in the previous sections. We will cover basic concepts behind a conversational AI engine.

In this lecture, we will provide overview of Step 7 – Monitor Model, specifically, what guard-rails should be in place for continuous monitoring and learning of AI system for production use.  How do you monitor your model for incorporating user feedbacks and incorporate field learning.

In this lecture, we will use a couple of slides to remind you of some of the key learnings in this class