Data Quality Masterclass - The Complete Course

Define data quality as how well a data set meets a user's needs to support accurate, timely, and duplicate-free data for data-driven decisions and business outcomes.

The lecture outlines five pillars: people, data profiling, defining data quality, data reporting, and data repair, and explains how these drive information quality, return on investment, and business performance.

Explore how poor data quality harms decision making, damages reputation, and leads to duplicate records and missed opportunities. Prioritize data quality to enable accurate insights and cross- and upsell opportunities.

Analyze the cost of poor data quality with IBM and Gartner estimates, revealing billions in lost revenue, wasted time, and the need for a data quality management program.

Analyze the NASA Mars climate orbiter data quality issue where a metric vs imperial mismatch caused a trajectory miscalculation and a multi-hundred-million-dollar project loss.

Master six global data quality dimensions: accuracy, completeness, validity, uniqueness, timeliness, and consistency, and learn how to measure data quality across real datasets.

Explore data accuracy as a core data quality dimension, examining how mismatched names, numbers, and country entries require verification with authentic references to ensure trusted reporting.

Master the data validity dimension by enforcing format, type, and range checks, detecting phone numbers, zip codes, and dates, and applying rules to ignore or resolve invalid entries for completeness.

Data timeliness means accessing data when needed, based on the task, without delays. In emergencies, access should be seconds or minutes via a central system.

Explore the completeness data quality dimension, distinguishing critical from optional fields, and learn how missing data may pass completeness while critical data like allergies or birth date must be present.

Identify duplicates by ensuring data uniqueness across dataset. Recognize that records can be duplicates even with differing fields, and verify completeness and accuracy to avoid trust issues when combining datasets.

Understand data consistency across datasets by ensuring identical employee details in multiple tabs, spotting mismatches like Frank versus Frankie or George Smarts, improving accuracy and completeness.

Participate in a lab to score a customer dataset across the six data quality dimensions: completeness, uniqueness, validity, accuracy, timeliness, and consistency, and build a data quality scorecard.

Data quality rules define what good data looks like within each dimension and are the business rules that drive automated data quality checks across your data systems.

Document data element requirements with technical descriptions, priority, and dependent data quality rules, then translate to code (sql), define test cases, implement automated production checks, and conduct annual reviews.

Explore creating data quality rules for key data elements like date of birth, translate rules into code, test, and implement validation checks across systems.

Explore data quality rules for the employee full name, including not N/A, not blank, no special characters, at least one space, and dependencies on social security number for production validation.

Discover how data profiling analyzes data sources, collects metadata, and reveals duplication, inconsistencies, and missing values to help data stewards investigate error origins and improve quality.

Showcase how data profiling benefits stakeholders—from business users to data stewards—by evaluating contact quality, system data inventories, and rule compliance to save time and boost ROI.

Parse data by separating complex entries into separate fields. Convert data formats to gain more control, such as splitting full names into first and last names.

Explain how data parsing extracts data pieces to validate against patterns and standards, enabling automatic checks of phone numbers, emails, and names within a data quality program.

Convert data to a common format and transform nonconforming records to a single standard across the database. Automated tools streamline data standardization, reducing manual edits.

Identity resolution consolidates data across datasets by checking, validating, and appending information to create a single data-rich profile for customers, employees, or suppliers, removing duplicates to reveal true relationships.

Explore the identity resolution process with a data quality management tool: identify channels and devices, join and match data points, validate confidence, and activate a single customer view.

Identity resolution creates a single customer view from multiple data sources into a single source of truth, enabling personalized experiences, better governance, and data-driven decisions across sales, marketing, and service.

Explore data linkage, the process of identifying, matching, and merging duplicate records across datasets to support data cleansing and master data management.

Data cleansing resolves corrupt, inaccurate, incomplete, or irrelevant data by transforming and standardizing data. It builds on parsing, identity resolution, and record linkage to improve data quality.

Enhance data value by appending information from internal and third-party datasets, integrating parsing, standardization, and record linkage to support smarter sales and marketing decisions.

Learn data inspection and monitoring to track and report data quality progress. Apply data profiling to define business rules, monitor accuracy and blanks with KPIs, and present leadership ready metrics.

Demonstrate data quality techniques: profiling, parsing, standardization, identity resolution, data linkage, cleansing, enrichment, and monitoring, by loading a supplier dataset, performing step-by-step improvements, and measuring outcomes.

Understand how data quality drives AI performance; clean data yields smarter, fairer predictions. Avoid duplicates, inconsistent timestamps, and varying structures that push AI toward wrong, confident decisions.

Explore why data quality matters for AI and how massive data sets magnify learning and risk from small errors, like incorrect formats or swapped labels.

Leverage AI to detect anomalies beyond simple rules, classify unstructured data, and improve matching, deduplication, and cleansing, amplifying traditional data quality practices with pattern-based intelligence.

Develop data quality skills for an AI world with broader profiling. Learn metadata lineage, governance, and AI basics to link data to model behavior and business impact.

Explore AI-driven data quality improvement on a realistic customer dataset of 1,000 records, fixing missing values, inconsistent categories, and incompatible fields across marketing systems, CRMs, and e-commerce platforms.

Identify and address missing and incomplete data using both manual checks and ChatGPT, highlighting gaps such as missing emails and data source fields for clean, marketing-ready datasets.

Apply ai-assisted standardization of categorical fields with ChatGPT, using ISO-3166 country codes, normalization of names, and data cleaning to resolve inconsistencies in country, segment, and phone data.

Identify and highlight duplicate emails using AI-powered deduplication, review full-row and customer ID duplicates, and save the latest cleaned Excel file.

Perform outlier and anomaly detection on numeric columns using AI tools, applying industry best practices; verify data quality with business logic to catch logical inconsistencies beyond statistical outliers.

Explore how ai validates relationships across fields to assess data quality, revealing inconsistencies such as vip spending anomalies, zero orders with revenue, and excel uploads issues, with actionable findings.

Use natural-language data quality checks with AI to flag inconsistencies in identity, emails, phones, addresses, dates, and revenue. Learn to escalate findings to the right teams for review.

Create a data dictionary for a dataset with ChatGPT, export in Excel, include fields, data types, descriptions, rules, constraints, values, and notes, to share with stakeholders.

Create data quality validation rules with ChatGPT for data quality management systems like Collibra, defining per-column rules for uniqueness, completeness, and data type to keep data clean.

Learn how AI tools like ChatGPT translate business rules into SQL and other code, turning validation rules and segment logic into ready-to-run scripts in SQL, Python, and Collibra formats.

Design clear stakeholder communications from a data quality assessment, summarize analysis, fixes like standardizing fields and normalizing contact data, highlight findings, and outline recommended next steps.

Explore how to perform a complete data quality analysis using prompts with ChatGPT, comparing step-by-step versus one-shot prompts, and emphasize careful review and reusable templates.

Join a hands-on data quality audit to detect missing values, duplicates, price outliers, validate relationships, build a data dictionary and audit dashboard, and draft the stakeholder report with AI.

Explore how Copilot in Excel boosts data quality by identifying duplicates and applying AI-driven insights, using the same dataset to compare Copilot with ChatGPT across best practices and tips.

Use Copilot in Excel to detect duplicates by email, phone, and customer ID, create a multi-sheet file, and note that missing values and name matches may not be real duplicates.

Identify missing and incomplete data in the dataset. Create an Excel sheet listing columns with missing values and share the report with the relevant team to improve data quality.

Use Copilot to standardize formats across the dataset, fix capitalization, remove extra spaces, and normalize country names, phone numbers, and date formats, then download the cleaned data with ISO codes.

Leverage Copilot to analyze the customer dataset, identify cross-field inconsistencies like revenue with zero orders and city-country mismatches, and export findings to an Excel workbook with sheets per issue.

Spot outliers in numeric data using Copilot by applying IQR and z-score methods, listing suspicious values with explanations and rerunning analyses as data changes.

Use Copilot in Excel to generate a clear, leadership-ready summary of data quality issues by severity and category, then review it with human expertise.

Discover how Copilot generates an issues log and remediation plan from data quality findings in Excel, and create supporting documents like data quality scorecards and business rule documentation.

Explore how data quality roles, with direct and indirect impacts on data quality management, vary by industry, company size, and budget allocated to data quality.

The data quality manager leads data quality processes, defines critical data and targets, sets escalation thresholds, oversees root-cause analysis, redemption plans, and dashboards, and guides the data quality analysts.

Lead data quality management as a data quality analyst by monitoring quality, performing statistical tests, resolving problems, and collaborating with data quality manager and database developers to prioritize enterprise-wide improvements.

Define the data owner as a senior business executive who sets data quality requirements and oversees datasets, collaborating with analysts to drive a data quality program.

data steward acts as the data owner's right hand, maintaining data across its lifecycle, identifying sources, fixing issues, and collaborating with the data quality team to enforce internal usage guidelines.

Meet the data custodian, the technical owner of the data environment who maintains the setup for data storage and ETL. They ensure data quality, integrity, and safety throughout ETL processes.

Data consumers determine what good data looks like and collaborate with the data owner and data steward to ensure data quality benefits the users.