
Meet the tutor: a data mining expert, chief data scientist and Industrial Revolution 4.0 implementer with advanced analytics, IIT, Indian School of Business, and edtech ventures, including interlocking director roles.
Explore the agenda and stages of analytics and learn how project management methodology shapes real-world data science projects.
Discover diagnostic analytics, the next stage of analytics that explains why something happened by identifying reasons behind events, such as covid case increases or drops linked to lockdown and vaccination.
Explore how predictive analytics uses current data to forecast future outcomes, such as COVID-19 cases and vaccination rates. Assess forecast validity by considering the chosen time horizon and evolving conditions.
Prescriptive analytics uses predictions to explore what-if scenarios and decide actions, such as boosting vaccines or automatically shutting down at-risk equipment, bridging descriptive, diagnostic, and predictive stages to prescriptive.
Describe crisp-ml(q) as the cross-industry standard process for machine learning, covering six phases: business and data understanding, data preparation, model building and tuning, evaluation, deployment, and monitoring and maintenance.
Define the scope of application by framing the business problem and objectives, including minimizing loan defaulters under practical constraints, and apply survival analytics to balance risk and profits.
Define business success criteria and KPIs, such as loan defaults, and align machine learning accuracy and performance with return on investment expectations, using concise objective statements.
Explore business understanding and use case framing in data mining. Balance fraud prevention with user convenience and examine drone-driven precision farming use cases to optimize yield and cost.
Explore the agenda of data understanding by identifying data types and scales of measurement, and review key terms and terminologies, along with primary and secondary data collection techniques.
Learn how measuring data enables analysis, modeling, predictions, and optimization for data understanding, guiding management decisions on sales and resources.
Explore data types by comparing continuous and discrete data, defined by decimal representation and sense-making. Learn examples like time, money, height, weight, and the distinction between numeric and categorical data.
Learn how categorical data differs from count data within discrete data, including binary and multiple categorical types, and distinguish nominal, ordinal, interval, and ratio data.
Learn practical data understanding with real-time examples by distinguishing nominal, ordinal, interval, and ratio data, and explore how absolute zero and subjectivity affect interpretation.
Explore the scale of measurement across nominal, ordinal, interval, and ratio data, showing how counts, proportions, and modes arise, and why ratio data enables broader analysis.
Explore the differences between quantitative and qualitative data, including continuous, count, and categorical data, and learn how numeric data informs decision making in data mining.
Explain the differences between structured, unstructured, and semi-structured data, and show how raw media and text transform into structured formats using frames, pixels, MFCC, and bag-of-words representations.
Learn data collection basics, including primary and secondary data sources, and clarify input and output variables (response, dependent, target) with terms like explanatory predictors, covariates, rows, and columns.
Understand primary data sources and how external data, including social media sentiment and IoT sensor data, improve loan risk prediction and production quality analysis.
This lecture contrasts secondary and primary data sources, with telecom data, Google Maps, and drone analytics illustrating how combining sources yields stronger insights.
Explore data collection using surveys to diagnose business reality, perform root cause analysis, and translate decision problems into research objectives and construct-driven survey questions across time, constraint, and strength dimensions.
Design of experiments to optimize data collection and marketing tests, examining discount levels, expiry timing, and customer distance to predict coupon redemption.
Identify and mitigate data collection errors in unsupervised learning by examining random and systematic errors, measurement device reliability, gauge R&R, SOPs, attribute agreement analysis, and bias to ensure representative data.
Develop fair machine learning by mitigating bias with representative data, avoiding race or gender signals, and prioritizing business understanding, data collection, and data preparation.
Explore the crisp data preparation framework CRISP-ML(Q) and its six phases. Understand business and data understanding, objectives, constraints, success criteria, data types, and data collection methods (secondary first, then primary).
Explore the probability formula, defined as the number of interested events divided by the total events, and apply it to die-roll examples like greater than three or smaller than four.
Define and distinguish random and variable, explain that a random variable has variable outputs with associated probabilities, illustrate with coin flip and die, and describe probability distribution.
Understand probability and its applications, learn about probability distributions and random variables, and distinguish discrete versus continuous data using iPad sales examples.
Understand normal distribution as a continuous probability distribution from minus infinity to plus infinity; the area under the curve equals one, and the probability of any single value is zero.
Explore inferential statistics by learning how to infer population sentiment from a sample, using simple random sampling and a sampling frame (optional), while understanding bias, sample size, and hypothesis testing.
Explore the standard normal distribution and z scores, their symmetry and sigma levels, and how standardization with z = (x−μ)/σ characterizes the standard normal curve.
Explore mean, median, and mode as first moment business decision measures, comparing population parameters and sample statistics, and learn how outliers affect the average while mode handles categorical data.
Explore measures of dispersion and the second moment through real-world profit data across locations, using control charts to assess variation and identify outliers.
Clarify box plots by distinguishing percentile, quantile, and quartile, and relate the quartiles from first to third to min, median, and max.
Explore graphical techniques for assessing distribution shapes, including histograms, box plots, and the qq plot to determine normality from sample and theoretical quantiles.
Explore how a bivariate scatter plot reveals the direction and strength of the correlation between two numerical variables, showing linear and non-linear patterns, outliers, and clusters.
Install python from python.org on Windows, Linux, or Macintosh using the latest 3.10.7 version, then explore its open source, free nature and options like Anaconda for a better look.
Master the Anaconda installation across Windows, macOS, and Linux, and understand its distribution with pre-installed, compatible libraries that are free for individuals but paid for commercial use.
learn to use Jupyter and Google Colab for interactive Python coding, import pandas, use read_csv to load datasets, and leverage gpus and tpus with Spyder interfaces.
Explore Anaconda Navigator, Spyder, and Python libraries such as pandas, numpy, scipy, matplotlib, and seaborn, and learn to read_csv into a data frame.
Recap core concepts from scatterplots, correlation direction and strength, and the difference between covariance and correlation. Preview phase two data preparation techniques like typecasting, handling missing values, standardization, and binning.
Explore data cleansing and organizing, transforming unstructured logs into structured data, and use typecasting to ensure correct Python data types (integers, floats, or objects) during data pre-processing.
Identify and consolidate duplicated customer records using master data management and data quality concepts, creating a single view of spending across accounts while removing duplicate rows and columns.
Master data cleansing and typecasting in Python with pandas to clean and transform data types. Inspect dtypes, detect misclassified columns, and use astype to convert to integers, objects, or floats.
The Data Mining - Unsupervised Learning course is designed to provide students with a comprehensive understanding of unsupervised learning techniques within the field of data mining. Unsupervised learning is a category of machine learning where algorithms are applied to unlabelled data to discover patterns, structures, and relationships without prior knowledge or guidance.
Throughout the course, students will explore various unsupervised learning algorithms and their applications in uncovering hidden insights from large datasets. The emphasis will be on understanding the principles, methodologies, and practical implementation of these algorithms rather than focusing on mathematical derivations.
The course will begin with an introduction to unsupervised learning, covering the basic concepts and goals. Students will learn how unsupervised learning differs from supervised learning and semi-supervised learning, and the advantages and limitations of unsupervised techniques. The importance of pre-processing and data preparation will also be discussed to ensure quality results.
The first major topic of the course will be clustering techniques. Students will dive into different clustering algorithms such as hierarchical clustering, k-means clustering, density-based clustering (e.g., DBSCAN), and expectation-maximization (EM) clustering. They will learn how to apply these algorithms to group similar data points together and identify underlying patterns and structures. The challenges and considerations in selecting appropriate clustering methods for different scenarios will be explored.
The course will then move on to dimensionality reduction, which aims to reduce the number of features or variables in a dataset while retaining relevant information. Students will explore techniques such as principal component analysis (PCA), singular value decomposition (SVD), and t-distributed stochastic neighbour embedding (t-SNE). They will understand how these methods can be used to visualize high-dimensional data and extract meaningful representations that facilitate analysis and interpretation.
Association rule mining will be another key topic covered in the course. Students will learn about the popular Apriori algorithm and FP-growth algorithm, which are used to discover interesting relationships and associations among items in transactional datasets. They will gain insights into evaluating and interpreting association rules, including support, confidence, and lift measures, and their practical applications in market basket analysis and recommendation systems.
The course will also address outlier detection, a critical task in unsupervised learning. Students will explore statistical approaches such as z-score and modified z-score, as well as distance-based approaches like the Local Outlier Factor and Isolation Forest. They will understand how to identify anomalies in data, which can provide valuable insights into potential fraud detection, network intrusion detection, or system failure prediction.
Evaluation and validation of unsupervised learning models will be an essential aspect of the course. Students will learn about internal and external evaluation measures, including silhouette coefficient, purity, and Rand index. They will gain skills in assessing the quality of clustering results and measuring the performance of dimensionality reduction techniques.
Throughout the course, students will be exposed to various real-world applications of unsupervised learning. They will discover how market segmentation can be achieved through clustering, enabling businesses to target specific customer segments effectively. They will also explore image and text clustering, which has applications in image recognition, document organization, and recommendation systems. The course will highlight anomaly detection, which plays a crucial role in identifying fraudulent transactions, network intrusions, or manufacturing defects. Lastly, students will learn how unsupervised learning powers recommender systems, providing personalized recommendations based on user behaviour and preferences.
Hands-on experience will be a significant component of the course. Students will work on practical exercises and projects, applying unsupervised learning algorithms to real-world datasets using popular data mining tools and programming libraries such as Python's scikit-learn or R's caret package. They will gain proficiency in pre-processing data, selecting appropriate algorithms, fine-tuning parameters, and interpreting and visualizing the results.
By the end of the course, students will have a solid understanding of unsupervised learning techniques, their practical applications, and the ability to leverage these methods to discover valuable insights and patterns from unlabelled data.