Data Cleaning Techniques in Data Science & Machine Learning

This session demonstrates reading data from csv and excel formats, loading datasets like the breast cancer data into a data frame, with header handling, and renaming columns.

Examine the structure of the dataset by examining size, shape, rows and columns, feature names, data types, and missing or unique values; learn when to transpose for analysis.

Split data into input features and target, filter out personal information and unused columns, and merge datasets on a common idea column to align results for modeling.

Learn to verify data integrity by checking completeness, consistency, accuracy, and duplication avoidance, and up-to-dateness, while validating features, data types, ranges, and unique identifiers to prepare reliable datasets for analysis.

Apply domain knowledge to guide model building and feature selection using data descriptions, such as the auto MPD dataset from the UCI repository, and consult domain experts to interpret features.

Explore how to determine a variable's range with min and max, visualize distributions using bar charts, histograms, and box plots, and assess relationships via scatter plots and trend lines.

Analyze how miles per gallon and car weight relate through a scatter plot, correlation coefficient, and polynomial fits of order 1 and 2 in Python using seaborn.

Define and classify variables as category, descriptive, and numeric variables. Explain binary, nominal, and ordinal types, and show how to use one hot encoding for ML models.

Explore numeric variable types—discrete, continuous, and ordinal—and learn to transform and encode them with democratization and one-hot encoding for machine learning.

Explore how to visualize single-variable data using bar charts, histograms, and density plots, including decisions on bin size, spacing, and when to stack or align multiple variables.

Explore how scatter plots visualize relationships between two variables, reveal distributions, show a trend line, and indicate correlation and the strength of dependence.

Explore how the Pearson correlation coefficient measures the linear relationship between two numeric variables, indicating positive or negative dependence and strength from -1 to 1 via plots and the formula.

Normalize each variable to a 0–1 scale by subtracting the minimum and dividing by the range, and apply log transformation to skewed data to improve knn distances.

Discretize continuous features into intervals to reveal patterns and improve predictions. Select binning strategies (uniform, equal height, or k-means) and choose supervised or unsupervised methods.

Explore how real-world data collection creates missing and absurd values, from placeholders to default entries, and learn data cleaning, type-correcting, and harmonizing features for robust machine learning.

Explore data cleaning techniques to identify and handle missing values, unify their formats, and visualize the distribution of missing values with Python, Pandas, and Seaborn.

Decide to drop rows or columns with missing values or impute them using mean, median, mode, or nearest neighbor methods, guided by thresholds, data size, and feature type.

Detect outliers with z-scores and nearest-neighbor methods, assess their impact on model performance, and decide when to exclude or note them during preprocessing and post-processing.

Learn data cleaning on the Titanic dataset by loading the training data, inspecting shape and missing values, and dropping non-informative columns to predict survival.

Visualize variable distributions with bar charts and histograms, apply log-scale transformations to reveal skewness, then explore survival trends by sex and class using color-coded plots.

Explore data cleaning and feature selection on the breast cancer dataset by applying filter, wrapper, and embedded feature selection methods, handling missing values, and visualizing correlations.

Impute Titanic missing values using mean/median and mode; encode sex, embarked, and cabin to numeric, then apply knn imputation with fancy boot and prepare features with one-hot encoding.