CRISP-ML(Q)-Data Pre-processing Using Python(2026)

Define the scope of application by clarifying the business problem, objectives, inputs, and constraints to build a predictive model that minimizes loan defaulters while balancing profits.

Define the business success criteria and KPIs, such as reducing loan defaulters under 5%, achieving machine learning accuracy above 85%, and ensuring responsive performance, with clear return on investment.

Define business objectives for fraud detection in credit card operations, balancing minimize fraud with customer convenience, and illustrate with precision farming use cases like drone-driven analytics and cost constraints.

Explore data understanding by examining data types and scales of measurement, along with key terms and terminologies. Learn about primary and secondary data collection techniques.

Explore how measuring sales data enables analysis, builds models for predictions and optimization, and supports what-if analysis for informed management decisions.

Explore the difference between continuous and discrete data, learn how decimal representation defines continuous data, and identify numeric versus categorical data with real-world examples.

Explore categorical data and count data, including binary (boolean) versus multi-category variables, and nominal, ordinal, interval, and ratio data, with churn, default, and loan repayment examples.

Explore practical data understanding for pre-processing using Python by distinguishing nominal, ordinal, interval, and ratio data through a real-world example illustrating absolute zeros and subjective perceptions.

Master scale of measurement from nominal to ratio data, learn which operations apply: counting, rankings, additions, multiplications, and why ratio data enables the richest statistical analysis for data pre-processing.

Define the difference between quantitative and qualitative data, and distinguish continuous, count, numerical, and categorical data to guide data pre-processing decisions.

Store structured data in tabular formats; unstructured data such as videos, images, and text cannot be stored in tables. Convert semi-structured data such as JSON into structured form.

Explore big data versus non big data through the three v's—volume, velocity, and variety—and learn storage and compute decisions for structured, semi-structured, and unstructured data using SQL, NoSQL, and Hadoop.

Identify cross-sectional, time series, and panel data and know when date and time matter. Distinguish data structures—multiple columns versus a single time column—to determine appropriate techniques.

Master data pre-processing by recognizing balanced and imbalanced datasets and rare events, including binary and multi-class classification and normal and non normal distributions.

Compare offline batch processing with online streaming, showing how historical loan data supports batch predictions at set times, while real-time fraud detection requires instant scoring and higher resources.

Identify primary and secondary data sources, distinguish their roles, map input variables to output and dependent variables, and convert unstructured data into structured formats for machine learning analysis.

Explore secondary data sources and how they differ from primary data sources. See telecom customer data, open data like Google Maps, drone analytics, and syndicated data to enrich insights.

Harness primary data sources by combining bank data with outward data from social media to improve loan default predictions, and distinguish primary data from secondary data in IoT contexts.

Identify business reality and root causes, define decision problems and research objectives, and design multidimensional survey constructs to measure customer preferences, advertising effectiveness, and price elasticity.

Explore how design of experiments informs mobile promotion strategies by testing discount levels, expiry dates, and customer radius to reveal how proximity and timing affect coupon redemption.

Identify and mitigate random and systematic errors in data collection, including faulty devices, harsh-environment sensors, and bias, using standard operating procedures and gauge repeatability and reproducibility analysis.

Ensure models are fair and unbiased by careful data collection and avoiding sensitive variables such as gender or race. Focus on understanding the business problem and thorough data pre-processing.

Introduce the CRISP-ML(Q) data preparation framework, emphasizing business and data understanding, recording objectives, constraints, and the project charter, defining success criteria, and choosing secondary then primary data sources.

Master the probability basics using the dice example and the formula: favorable events over total events, with patterns like greater than three or smaller than four.

Define random variable as a variable whose outputs vary with probabilities, represented by uppercase letters, with outcomes in lowercase; explore probability distributions using coin flips and dice.

Explore how probability informs decision making, illustrate probability distribution with iPod sales data, and distinguish discrete versus continuous random variables and distributions.

Explore inferential statistics by examining sampling from a population, using a sampling frame and simple random sampling to draw inferences about the population, and introduce hypothesis testing.

Clarify key terms such as probability, joint probability, and conditional probability. Explain continuous and discrete probability distributions, inferential statistics with populations and sampling, and introduce exploratory data analysis.

Explore the normal distribution as a continuous probability model, using histograms and density plots to check normality, noting area under the curve from minus infinity to plus infinity equals one.

Explore the normal distribution’s symmetry and Six Sigma concepts, the standard normal distribution, and the Z-score formula Z = (X - mu) / sigma.

Explore the first moment of business decision by analyzing mean, median, and mode, differentiating population parameters from sample statistics, and noting how outliers influence the mean versus median.

Explore measures of dispersion and the second moment to quantify variation across profits in Malaysia and Singapore, interpret dispersion bands, compare forecasting confidence, and identify outliers.

Explore how box plots use percentiles, quartiles, and quintiles to display results, with min and max and 25th, 50th, and 75th percentiles (Q1, Q2, Q3) and 100th percentile (fourth quartile).

Assess normality with graphical techniques, including Q-Q plots, histograms, and box plots, and understand standardized values and theoretical quantiles to determine if data are normally distributed.

Explore bivariate scatter plots to understand correlation direction and strength, distinguish linear from non-linear relationships, identify outliers and clusters, and interpret correlation coefficient r in data analysis with Python.

Download and install Python across Windows, Linux, and macOS from the source, noting version 3.10.7. Adopt open source and OS agnostic Python, with Anaconda available for a nicer setup.

Learn how to install Anaconda across Windows, Macintosh, Linux, or Unix, understand its OS-agnostic distribution with pre-installed libraries, and launch the Anaconda Navigator for data scientist workflows.

Explore how to use Anaconda Navigator and Spyder for Python development, load datasets with pandas, and leverage key libraries such as NumPy, SciPy, Matplotlib, Seaborn, and scikit-learn for data pre-processing.

Learn to use Jupyter as the key launcher and Google Colab for Python coding, including loading datasets with pandas, running cells, and choosing hardware accelerators like GPUs and TPUs.

Navigate the six-phase CRISP-DM process from business understanding to data preparation and evaluation, defining objectives, constraints, and success criteria for data science projects.

Learn core python data pre-processing with numpy, pandas, seaborn, and statsmodels to load education data, compute mean, median, mode, standard deviation, range, and explore data visually.

Explore the third moment, skewness, and its formula using (x minus mean) cubed over sigma cubed to reveal non-normal, positively or negatively skewed data; learn through histogram examples.

Explore how the fourth moment guides business decisions using python, linking skewness, normal distribution, and excess catharsis to actionable retail and e-commerce insights.

Explore univariate visualizations, including bar plots and histograms, to interpret single-variable data through bins, frequency distributions, and differences between normal and non-normal patterns.

Explore univariate plots in Python with histograms, density plots, and box plots, and interpret skewness and distribution using pandas, numpy, seaborn, and MATLAB.

Explore univariate box plots in Python to identify outliers, understand the five-number summary (min, q1, median, q3, max), and compare with histograms for distribution shape.

Understand univariate q-q plots in python to assess normality with pandas and statsmodels. Load the education dataset, compare sample and theoretical quantiles, and note data preprocessing if needed.

Load a dataset in python, read csv with pandas, and create a bivariate scatter plot of waist circumference versus adipose tissue, interpreting correlation and covariance for direction and strength.

Explore scatterplots to assess correlation direction, strength, and linear versus nonlinear relationships. Learn data pre-processing essentials—from typecasting and handling duplicates to outlier analysis, missing values, and standardization.

Learn to clean and organize raw data and apply typecasting to correct data types in Python, covering data cleansing, organizing, duplicates, and outlier handling for effective preprocessing.

Master data pre-processing in python with pandas: perform data cleansing, fix misclassified types via typecasting, and convert employee id, zip, salaries, and ages to appropriate formats.

Review data understanding and cleansing alongside exploratory data analysis, covering univariate and bivariate plots, correlation versus covariance, and Python typecasting with astype.

Explore handling duplicates in data pre-processing with Python, consolidating multiple records into a single customer view, and removing duplicate rows or columns to improve data quality.

Identify and manage duplicate records in a dataset using Python pandas read_csv, duplicate detection, and drop_duplicates, exploring keep options first, last, and false to clean data.