Data Pre-Processing for Data Analytics and Data Science

Define the scope of application and business problem to minimize loan defaulters. Use inputs x and outputs y to build a model predicting default while respecting constraints to maximize profits.

Define business success criteria by aligning problem solving with business needs and reducing loan defaulters under 5%. Measure machine learning success by accuracy, performance, and return on investment.

Learn how business understanding and use cases shape data science objectives, balancing fraud prevention with convenience via alerts, otp verification, and sms, while exploring precision farming with drones and sensors.

Discover the foundations of data understanding by exploring data types, measurement scales, essential terms, and primary versus secondary data collection techniques.

Learn how data understanding drives analysis, modeling, and predictions to enable optimization and informed management decisions, using what-if scenarios and constraint-based sales examples.

Learn to distinguish continuous versus discrete data types, including numeric and categorical data, with examples like time, money, height, and weight; continuous data can be decimals, discrete cannot.

Explain categorical data and count data within discrete data, contrasted with continuous data, covering binary and multiple categorical data (nominal/ordinal) and interval/ratio scales, with churn and default examples.

Distinguish nominal, ordinal, and interval data using real-world examples from travel, such as flight numbers, gate numbers, and temperatures, and explain subjective versus objective measurements and absolute zero.

Identify scale of measurement types: nominal, ordinal, interval, and ratio, with respective operations from counting to ranking, additions, and multiplications. Ratio data enables extensive data analysis.

Compare quantitative and qualitative data, with examples like temperatures and distances, and identify continuous, count, and categorical data to guide decision making.

Differentiate structured, semi-structured, and unstructured data, and explore how raw media like videos, images, audio, and text can be transformed into tabular formats.

Explore data collection, differentiate primary and secondary sources, and map data terms such as response variables, explanatory predictors, and rows and columns for structured analysis.

learn how to leverage primary data sources beyond internal data, including social media and IoT sensor data, to improve credit risk models while navigating data privacy and quality concerns.

Understand secondary data sources, including internal and external data, and contrast with primary data. Learn to blend Google Maps data with drone analytics and syndicate data for richer insights.

Identify business reality and root causes to inform survey-based data collection, translate decision problems into research objectives, and decompose multidimensional constructs into focused survey questions across time, constraints, and intent.

Apply design of experiments to data collection by testing marketing promotions, expiry dates, and customer distance to reveal how timing and reach influence coupon redemption.

Identify bias and ensure fairness in data collection and modeling, avoid race or gender as predictors, and emphasize pre-processing and business understanding.

Identify and mitigate data collection errors, including random and systematic errors, faulty devices, and bias, using gauge R&R, attribute agreement analysis, and standard operating procedures to ensure data quality.

Introduce crisp data preparation within CRISP-ML(Q), outlining six phases from business and data understanding to data errors, covering objectives, constraints, project charter, and secondary-then-primary data collection.

Learn the basics of probability using the formula probability = number of interested events divided by total events, with die examples such as bigger than three and smaller than four.

Define a random variable as a variable whose outputs vary and carry probabilities. Illustrate with coin toss and die outcomes, note 0–1 probabilities and distribution for uppercase variables, lowercase values.

Explore core probability concepts, applications, and probability distribution through real-world examples. Learn how random variables are modeled as discrete and continuous data, with tables and graphs.

Explore the normal distribution as a continuous probability distribution of a random variable, illustrated with heights, and confirm the shape with histograms, noting the area under the curve equals one.

Learn how inferential statistics draw inferences about a population from a sample using simple random sampling, sampling frames, bias-free selection, and hypothesis testing.

Explore the standard normal distribution and z scores, mu and sigma, standardization, and Six Sigma contexts to understand the bell curve and central tendency.

Explore the three measures of central tendency: mean, median, and mode, and their use in first moment business decisions, highlighting population parameters, sample statistics, outliers, and data types.

Compare profits across malaysia and singapore to measure dispersion and variation from the average, using second moment concepts to assess forecast confidence and identify outliers via control charts.

Explore box plots and the differences between percentiles, quantiles, and quartiles, including Q1, Q2 (median), and Q3, and understand how percentiles map to quartiles.

Assess normality with a normal Q-Q plot by comparing sample quantiles to theoretical quantiles, and interpret a straight line for the middle 50% of data to confirm normal distribution.

Understand bivariate scatter plots to assess direction and strength of correlations between two numerical variables, distinguish linear and non-linear trends, and identify outliers and clusters.

Install and use Anaconda as an OS-agnostic distribution that provides pre-installed, widely used libraries, saving data scientists from library version conflicts; it's free for individuals but licensed for commercial use.

Explore Anaconda Navigator, Spyder, and Python libraries, compare IDEs, and learn to load datasets with pandas read_csv, create data frames, and understand zero-based indexing.

Download python from python.org and install version 3.10.7 on OS; use Python as a calculator, open source and free for individuals and organizations, with Anaconda for the look and feel.

Explore Jupyter and Google Colab to run Python, import pandas, upload and read CSV files, and compare interfaces with Spyder while noting GPUs and TPUs.

This lecture uses scatterplots to reveal direction and strength of relationships (linear or nonlinear), identify outliers and clusters, compare correlation with covariance, outlines data pre-processing techniques and 30-hour training.

Explore data pre-processing fundamentals, including data cleansing, organizing, and typecasting, to convert unstructured logs into structured data and ensure correct data types in Python.

Master data cleansing and typecasting in python with pandas, including reading csv files, inspecting dtypes, and converting columns with astype for accurate data types.

Recaps core statistics and data visualization for data cleansing and exploratory analysis. Covers moments, dispersion, skewness, kurtosis, univariate and bivariate plots, and correlation concepts.

Learn how to identify and handle duplicates through consolidation, duplicate removal, and duplicate column elimination, using master data management and data quality concepts to achieve a unified customer view.

Identify duplicates in the mtcars dataset using pandas and the duplicated function. Experiment with keep options (first, last, false) and remove duplicates with drop_duplicates to preserve a usable feature set.

Learn how to identify and treat outliers using box plots, the three r technique (rectify, retain, remove), winterization, and standard deviation and iqr rules for data pre-processing in analytics.

Explore outlier analysis and treatment in python using pandas, numpy, and seaborn, including calculating IQR, identifying outliers in salaries, and applying trimming, replacing, box plot, and winsorization techniques.