Improving data quality in data analytics & machine learning

Download the Data QC zip via the resources button to access Python notebooks. Open and run them in Colab or locally with Jupyter, PyCharm, or Spyder.

Explore what data really means and what we can and cannot infer from it, and learn how data quality drives the quality of data-based decisions.

Clarify that data is plural and datum is singular, and emphasize correct subject-verb agreement in data analytics and machine learning communication.

Explore how data arise from measurement devices that convert observations into numbers, and how measurement limitations and noise shape interpretation and data quality.

Learn the GIGO concept (garbage in, garbage out) and how bad data drives bad decisions, highlighting the need to collect higher quality data for data-driven analysis.

Data quality drives every data-driven decision; rely on high-quality data rather than anecdotal stories, and collect or clean data to improve decision making.

Improve data quality across all analysis stages by applying context-specific cleaning strategies, evaluating risks, and focusing on planning and strategic data quality actions before, during, and after data collection.

Develop a data management plan by documenting, storing, securing data and processing code, and planning backups to ensure high quality, reproducible results.

Data documentation records metadata, origins, collection details, and processing history, including algorithms and cleaning, and provides data quality metrics, visualizations, and import instructions for diverse audiences.

A data audit reviews the data management plan, storage, documentation, and data security to improve data quality and support accurate decision making in analytics.

Explore four data quality control phases—from before collection to during and after analysis. Maximize quality and minimize bias in data analytics and machine learning.

Anticipate data problems by planning analyses and studying similar datasets before collection. Run pilot studies to test setup and data quality, and seek expert input to improve upfront.

Improve data quality during data collection by monitoring data for cleanliness, addressing noise and missing values, and pausing collection to fix issues before analysis.

Clean and transform collected data to improve quality before analysis. Compute data quality metrics, identify missing, corrupted, or outlier values, and apply removal, interpolation, or scaling as needed.

Improve data quality during analysis by recognizing the risk of cleaning after results, and maintain ongoing data quality control through inspection to prevent biases and ensure generalization.

Explore how improper data cleaning and selective outlier removal can bias results, and learn ethical, independent strategies to improve data quality and preserve generalizability in analytics and machine learning.

Assess data quality with qualitative and quantitative methods, learn what to look for in data, visualize data, and explore coding videos for both non-experts and data practitioners.

Explore qualitative and quantitative data quality assessments, compare their strengths and limitations, and learn how a hybrid approach leverages visual inspection with algorithmic checks to improve data quality.

Explore how to use data visualization to identify quality issues in data, distinguishing numerical and categorical data, and interpreting bar plots, pie charts, box plots, histograms, and scatter plots.

Learn to visualize data distributions with numpy, pandas, seaborn, and matplotlib. Create data, convert to a dataframe, and plot bar charts, pie charts, box plots, histograms, and scatter plots.

Explore how variance measures dispersion and data variability, compare conditions, and apply normalization and standard deviation to assess data quality and potential effects on statistical models.

Explore how correlation measures linear and monotonic relationships with the coefficient R, interpret correlation matrices, check data quality, and distinguish correlation from causation.

Explore the data error rate as a simple, versatile measure of data quality. Compute global or feature-specific error rates and distinguish errors like not a number, outliers, and missing data.

Learn how sample size shapes data-based decisions and why random sampling and per-group sizes matter. Understand how statistical power drives or limits inferences, with guidance on when to grow samples.

Explore general data quality features for any dataset with pandas and seaborn, including missing values, duplicates, and correlations, and learn practical cleaning with drop duplicates and dropping missing values.

Explore data transformations, including linear and nonlinear techniques such as scaling and normalizing, and learn when to apply them before analysis to improve interpretation and satisfy algorithm requirements.

Learn how z-score normalization standardizes data by subtracting the mean and dividing by the standard deviation, enabling cross-scale comparisons in data analysis and machine learning.

Learn min max scaling to normalize data to a 0–1 range, preserving relative distribution while transforming every value by (x−min)/(max−min). Extend to any [A,B] range for machine learning applications.

Binning transforms continuous data into discrete bins by rounding, balancing histogram readability, noise reduction, and information loss. Choose bin counts carefully to preserve distribution and data quality.

Explore unit normalization, or unit scaling, by defining unit vectors and magnitudes, computing vector length with Euclidean distance, and applying scaling to align data across different scales while preserving sign.

Explore the rank transform and tied rank transform, key nonlinear, lossy, non-convertible methods used in correlations, non parametric statistics, and ANOVA, including handling ties by averaging ranks.

Apply nonlinear data transformations, such as log, square root, rank, and Fisher, to normalize non-Gaussian data and improve linear method validity.

Apply z-score scaling and min-max scaling to align variable ranges, explore binning and unit normalization, and perform rank transforms in pandas and NumPy for cleaner data quality.

Explain outliers and missing data, why they matter for analysis, and how to identify and handle them, noting that the best remedy is early data quality control.

Define outliers and their ambiguity in univariate and multivariate data. Explain how they arise from noise, error, or natural variation. Compare removal with robust analysis approaches that handle leverage.

Identify and remove outliers with the Z-score method. Convert data to Z scores, apply a three standard deviation threshold, and use iterative or modified Z-score approaches for non Gaussian data.

Use the modified z-score method to identify outliers in non-Gaussian data by subtracting the median, dividing by the median absolute deviation, and applying a threshold (often three).

Explore four strategies for missing data in data analytics and machine learning, including complete row removal, selective data point removal, mean imputation, and model-based prediction.

Identify and remove outliers using z-scores with a three standard deviation threshold, drop rows with missing values, and impute missing data by column means, illustrating effects on means.

Share personal opinions about the field's state and thriving strategies, based on anecdotal experience and limited data, and invite disagreement through Q&A.

Keep pace with data science developments by understanding the math behind analysis methods and how growing compute power, larger datasets, and better visualizations shape practice.

Balance breadth and depth in data science by building familiarity with many data analysis methods while developing deep expertise in a few. Become the domain expert others consult for insights.

Data scientists want to understand what's happening under the hood of their tools, gaining the computational clarity and confidence to do better data analysis and interpret results accurately.