
Meet the tutor Bharani Kumar de Peru, a chief data scientist and industrial revolution 4.0 implementer, with IIT and ISB degrees, leading analytics and edtech ventures.
Explore supervised learning concepts, including binary, multi-class, and multi-label classification, use naive Bayes for text data, and integrate unsupervised clustering before predictive modeling.
Explore agenda and stages of analytics within this data science training program. Discover how project management methodology guides real-world analytics projects and reveals finer concepts in each module.
Explore diagnostic analytics by asking why something happened, such as spikes or drops in Covid-19 cases, and relate them to factors like lockdowns and vaccination.
Explore predictive analytics by using current data to forecast future outcomes across chosen time horizons. Consider how changing factors affect the validity of these predictions.
Prescriptive analytics selects actions via what-if analysis after predictions, guiding decisions through descriptive, diagnostic, and predictive stages to prevent failures.
Explore crisp-ml(q), a cross-industry standard for data science, outlining six phases—business and data understanding, data preparation, model building and tuning, evaluation, deployment, and monitoring and maintenance.
Define the scope of application, set objectives to minimize loan defaulters under constraints, and use survival analytics with input features to predict defaults and maximize profits.
Define the business success criteria to align machine learning projects with business problems, set KPIs like loan default rates, and weigh accuracy, performance, and ROI when evaluating solutions.
Understand business use cases with fraud detection in credit card transactions, balancing fraud minimization with customer convenience, and explore precision farming using drones to maximize yield and minimize cost.
Explore data understanding by measuring data, analyzing it to build predictive and optimization models, and supporting management decisions through what-if analysis of sales levers like marketing spend and outlets.
Discover the difference between continuous and discrete data types, with examples like time, money, height, and weight, and distinguish discrete counts from decimal representations and categorical data.
Explore categorical data and count data, distinguishing binary and multiple categorical data, with churn, defaults, claims, resignations, and other count-based targets across business contexts.
Explain nominal, ordinal, interval, and ratio data with real examples—flight numbers, gate numbers, temperatures, and money—highlighting absolute zero.
Explore the scale of measurement, distinguishing nominal data (counts), ordinal data (rank), interval data (add/subtract), and ratio data (multiply/divide) to guide data collection and statistical analysis.
Compare quantitative data, numeric and continuous or count data, with qualitative data, which is categorical. Use examples like temperature and distance to illustrate how data type informs decision making.
Differentiate structured (tabular) data from unstructured data like videos, images, audio, and text, and note that semi-structured data such as HTML, XML, and JSON can be converted to structured formats.
Explore balanced, imbalanced, and rare-event datasets in supervised learning, including two-class and multi-class outcomes, with examples like employee attrition and credit card fraud, and how distribution shapes analysis.
Compare batch offline processing with live streaming online processing through loan default prediction and real-time fraud detection, showing dashboards, automation rules, and cost-aware compute choices.
Identify primary and secondary data sources and how datasets use rows and columns to structure data for analysis, and recall common input and output terminology such as predictors and response.
Understand primary data sources and how external data, like social media and IoT sensor data, enhances loan default prediction. Recognize privacy risks and the move from secondary to primary data.
Compare primary and secondary data sources, showing how combining customer data with open source maps and drone analytics informs 5G planning for rural villages.
Learn end-to-end data collection with surveys, linking business reality to root cause analysis, define research objectives, and craft constructs split into time, constraint, and strength with targeted questions.
Learn how design of experiments guides data collection for marketing trials, optimizing discount levels, expiry timing, and customer radius to reveal how factors influence coupon redemption.
Identify and mitigate random and systematic errors in the data collection stage, implement standard operating procedures, and apply gauge R&R and attribute agreement analysis to ensure measurement quality.
Learn to identify and prevent bias, ensure fairness in data collection, and avoid using sensitive variables in loan decisions, while prioritizing understanding the business problem and data before modeling.
Introduce crisp data preparation within the CRISP-ML(Q) framework, detailing six phases—from business and data understanding—to record objectives, constraints, success criteria, project charter, secondary and primary data sources, and data errors.
Master the probability formula, number of interested events over total events, and apply it to dice outcomes like bigger than three or smaller than four.
Define a random variable (uppercase) as a variable that can take multiple values (lowercase outcomes) with probabilities. Learn how distributions allocate 0–1 across outcomes using coin flips and dice.
Explore probability concepts, probability distribution, and random variables, and distinguish discrete versus continuous data using practical examples like daily iPad sales and distribution graphs.
Explore normal distribution as a continuous probability distribution, illustrated by heights. The area under the curve equals one, and a single value has zero probability, spanning minus to plus infinity.
Explore inferential statistics by sampling a population, using simple random sampling and sampling frames, to draw inferences about the population and understand hypothesis testing, parametric and nonparametric methods.
Explore the standard normal distribution and z-scores, relate mean and standard deviation to probability ranges, and examine sigma levels from one to six sigma.
Explore mean, median, and mode as first moment decision measures, and learn how outliers influence the mean, why median is robust, and when to use mode for categorical data.
Explore measures of dispersion, the second moment of data, and how dispersion reveals profit variation across markets, guiding forecasts and identifying outliers through control charts.
Learn how box plots use percentiles, quantiles, and quartiles, distinguishing their ranges and mapping 0, 25th, 50th, 75th, and 100th points to Q1–Q4.
Use the normal q-q plot to assess normality by comparing theoretical and sample quantiles, with histogram and box plots for distribution shape.
Learn to create a bivariate scatter plot in Python, load the wc80 dataset, and interpret correlation coefficient, covariance, outliers, and clusters.
Download and install Python from python.org, using the latest 3.10.7 version, which runs on Windows, Linux, or Macintosh. Python is open source and free for individuals and organizations.
Install Anaconda, a cross-platform distribution with pre-installed, compatible libraries to save setup time. Discover how it streamlines data science work by avoiding library version conflicts and using the Navigator.
Learn to set up Anaconda Navigator and Spyder, compare IDEs, and use key Python libraries like numpy and pandas to read_csv and work with data frames.
Explore Jupyter and Google Colab for Python data analysis, run code, import pandas, and load csv datasets, with insights on using Spyder for practice.
Recap key concepts from scatterplots: correlation direction and strength, covariance versus correlation, and outliers and data preparation techniques like typecasting and missing values.
Learn data cleansing and preprocessing, including organizing data, handling duplicates, and typecasting between integer, float, and object types in Python for structured data.
Master data cleansing and typecasting with Python to prepare supervised learning datasets, using pandas to inspect data types and convert columns with astype.
Explore how to handle duplicates in data using master data management and data quality concepts, consolidating records and removing duplicate rows or columns to ensure accurate, efficient machine learning insights.
Identify and manage duplicate records in a dataset using pandas in Python, exploring keep options (first, last, false) and applying drop_duplicates to cleanse the data.
Data Mining Supervised Learning - Traditional ML Models.
Supervised Learning is a sub-division of the Model Building step of CRISP-ML(Q)
Methodology. Supervised learning is a type of Predictive Modeling that involves
Classification Models, Shallow Machine Learning Models, Ensemble Models, Regression
Models, and the Black Box technique. We have numerous divisions of each modeling
technique.
We thoroughly discuss Probability, Joint Probability, Bayes Rule, and Naive Bayes using a
use case. Naive Bayes is not ideal for larger numeric features because numeric features
must be converted into categorical ones through discretization or bining. It allows the
deletion of missing value entries. This algorithm assumes class-conditional independence.
The probability is zero for new words not seen in training data, making the entire calculation
zero. To encounter this problem, we use Laplace Estimator. French Mathematician
Pierre-Simon Laplace created this algorithm. The default value of the Laplace estimator is 1.
Any value can be used for the Laplace estimator.
K-Nearest Neighbor Classifier is also called Lazy Learner, Memory-Based Reasoning,
Example-Based Reasoning, Instance-Based Learning, Case-Based Reasoning, Rote
Learning, etc. We understand the differences between the k-means algorithm and kNN. We
then understand 1, 2, 3, and 7 Nearest Neighbors. The minimum k value equals 1, and the
maximum equals the number of observations. k is a hyperparameter. We then understand
what a baseline model is, where accuracy is equal to the majority class, and for prediction
models, accuracy is greater than 80%. We further understand the Bias-variance trade-off.
We jump into the applications and importance of k-NN at the end.
The Decision Tree algorithm is a Rules-based algorithm. We understand what a decision
tree is, followed by learning how to build decision trees, then we dive into the greedy
algorithm, building the best decision tree and attribute selection. A decision tree is a tree-like
structure in which an internal node represents an attribute, each branch represents the
outcome of the best and each leaf node represents a class label. There are 3 types of nodes
a root node, a branch node, and a leaf node. So how do we build a decision tree? First, we
use training data to build a model, and then the tree generator determines the following-
- Which variable has to be split at a node, and the value of the split
- The decision to stop or split again has to be made
- Assigning terminal nodes to a label
- A basic or Greedy algorithm is a tree constructed in a top-down recursive
divide-and-conquer manner.
- Further, we analyze Greedy Approach, Entropy, and Information gain.