Automation in ETL /Data Testing

Name: Automation in ETL /Data Testing
Rating: 3.1 (20 reviews)

Learn ETL /Data Testing Automation using a simple and sample ETL Project

Created byDeepak Tiwary

Last updated 9/2025

English

English [Auto],

What you'll learn

Automate your ETL Testing Validation Steps
Reduce the efforts for Retesting and (Non) Regression Testing
Save time for finding Regressions and Retesting
Reuse of same Testing Framework for other ETL Testing Projects

Course content

6 sections • 20 lectures • 1h 55m total length

Introduction1:25
This is an introduction about course!

Connection to the Database MS SQL using Python5:31
I am using MS SQL as a database. You can use any database of your choice. To install there are simple steps. Also, to connect to database through python, every database has its own connection string that can be found through respective database official website. I have shown here how to connect to MS SQL database.
Creation of the Framework to run the sql queries2:56
We need to run sql queries most of the time. To connect to cursor of database, time and again, we have created this framework.

Creating and Executing First Test Case5:36
We are creating a simple test case for validation of presence of table in database.
Parametrizing the test cases5:10
Parametrize test cases with pytest to verify multiple tables (sales, date, customer, products), avoid repetitive definitions, and validate table presence through one streamlined test run.
Validation for Presence of records in the tables3:25
Learn to use parameterized tests to verify table presence and record availability in ETL data testing, update mapping documents, and run 12 test cases across 6 tables.
Validation for Column Names of the tables3:54
Extracting column names from tables5:40
Develop a framework to extract column names from a table using an sql query and Python, transforming results into a clean list for etl data testing.
Extracting column names from excel sheet (Mapping Document)5:29
Validation of Columns Names from Mapping Document with respect to tables present1:23
Parameterize the table name for both tabs in the mapping document, validating that the target name matches the sales fact and sales age tables, then run the test case.
Validation of absence of duplicate records in the tables6:57
Validate absence of duplicates across etl tables by writing parameterized duplicate checks using primary keys; identify duplicates with count greater than one across sales, customer, product, and date dim tables.
Validation for ensuring no records are dropped12:59
Validation of Population of one Column using Sample Data7:51
Validate the population of the total revenue column using sample records, applying the mapping rule purchase quantity multiplied by per unit price, and confirm all test cases pass.
Validation of Population of columns on sample data and full data8:07

Creating pipeline for test case execution10:52
Create a Jenkins pipeline for test execution in ETL testing, wiring GitHub, a Jenkins file, and Pytest with a SQLite database to validate test coverage and show success.
Test Script for validation of migration of data from one DB to other DB9:29
Creation of HTML Report after test case execution5:15
Install pytest-html, run pytest -v to execute all 55 test cases, and generate an HTML report highlighting passes and failures for stakeholders.
Creation of Allure Report after test case execution8:50
Learn how to generate an Allure report after test execution by installing Allure, configuring environment paths, running pytest, and reviewing the index.html with detailed results.

Requirements

ETL Testing
Python
Pytest
SQL

Description

The course is on a basic idea of automation in ETL Testing or Data Testing. We perform Retesting and (Non) Regression tests during a project or release. We perform same set of validation steps to ensure there are no regression introduced due to recent changes. If one has the idea on automating those repetitive task, it helps a lot to save our efforts and time.

I have used a simple ETL project to understand the automation steps. In this course, I have used Python and Pytest to organize the test cases. The project used is just to give an idea how can we connect to actual requirement when we perform actual testing during our professional work. The project is not ideal and actual ETL project can vary to a greater extent.

To validate the business rules, I have used SQL queries in the course. Few frameworks are created to use it for further test cases and execution. Most of the test cases include here are not exhaustive and are just for understanding purpose. Please do not take it as a reference for ETL Testing. Only take away from this course is how to automate your same set of validation steps.

It is recommended to have experience on ETL Testing and basic knowledge of SQL, Python and Pytest. However, if you don't have much experience on these, it won't be very difficult to understand it. I have tried to make the course as simple as possible.

Who this course is for:

Who are new to Automation in ETL/Data Testing
Have an idea for automation in ETL/Data Testing
Perform same set of validation steps in ETL/Data Testing
Run same set of SQL queries
Want to create automation test suite in ETL/Data Testing but lack the idea

Automation in ETL /Data Testing

What you'll learn

Explore related topics

Course content

Introduction1 lecture • 1min

Connection to Database and Creating a Framework2 lectures • 8min

Understanding the Project Requirement1 lecture • 3min

Creation and Execution of Test Cases11 lectures • 1hr 7min

Thank You Video1 lecture • 1min

Additional Videos4 lectures • 34min

Requirements

Description

Who this course is for: