Azure Data Engineering End-to-end Course 2026 (English)

Explore CDU content 3.0 in the Azure data engineering end-to-end course (English), focusing on the key CDU content for data engineering learning.

We updated the Azure Data Engineering course with Databricks and PySpark, adding new interfaces and Unity Catalog, with English and Hindi content, 8.5 hours, and more focus on Spark internals.

Explore the history of SQL as part of the Azure data engineering end-to-end course 2026.

Learn to create and drop databases using SQL queries and the interface. Execute create and drop statements, ensure unique names, handle errors, refresh the list, and save queries as .sql.

Explore SQL Server hierarchy within the Azure data engineering end-to-end course, learning how to represent and navigate hierarchical data.

Explore the four system databases—master, model, msdb, and tempdb—and their roles, from starting the SQL Server to templates, jobs, alerts, and temporary objects, and never touch them.

Learn to use the select statement to fetch data from the employees table, choosing all columns or specific ones like employee ID, name, and salary, while avoiding select star.

Learn to get unique values from single or multiple columns in AdventureWorks using select distinct on dim product, a virtual extraction that does not modify the table.

Learn to use sql comments in tsql on sql server, including double dash single-line and slash star block comments, to document queries and aid end users.

learn to filter data with the where clause in sql, using boolean expressions and comparison operators, and combine conditions with or, and, in, and between through practical examples.

Learn pattern-based filtering using the like operator with wildcards in sql, including percentage and underscore, to filter begins with, ends with, and contains in text and numeric columns.

Learn how to group rows with the group by clause, aggregate sales by product key, and distinguish group by from distinct to calculate total sales per key.

Learn how to enforce uniqueness and non-null constraints using a primary key when creating tables in SQL, preventing duplicate and null employee IDs.

Create a table with a check constraint to enforce salary above 10,000 and gender values m, f, or o; test inserts, generate script, and learn to modify via alter table.

Learn how the SQL default constraint works by adding a city column with a default of London in a create table statement, and see how NYC overrides it.

Learn core string functions in SQL, including upper, lower, left, right, substring, and len, with practical examples like English product name, plus trim, l trim, and r trim.

Explore string functions such as replace, reverse, and stuff to transform text. Apply 1-to-1, 1-to-many, and many-to-many substitutions, including dropping with a blank and inserting characters at specific positions.

Extracts first and last names from a full name using dynamic left, right, and len functions with charindex to handle spaces and middle names, demonstrating nested function patterns.

Explore date and time basics in SQL, including get date, date time, date time two, UTC date time, precision differences, and creating dates with date from parts.

Learn how to extract year, month, and day with date time functions, compare date part and date name outputs, and format dates for display using the format function in sql.

Learn the four integer data types in t-sql—tinyint, smallint, int, and bigint—discover their ranges and storage sizes, and how to choose the right type to prevent overflow.

Explore approximate numeric data types in SQL, including float, real, money, decimal, and numeric; learn precision, scale, ranges, and storage implications with practical table examples.

Identify and compare date, date time, date time two, small date time, and time data types, focusing on their ranges and storage in bytes.

Explore the unique identifier data type in SQL, learn how to generate new ID values with the new ID function, and understand its use as a primary key across tables.

Learn data type conversion in SQL, including implicit and explicit casting with cast and convert, handling string and numeric operations, and using money and int conversions.

Learn to join tables using product key and customer key to show customer names, product names, and sales from dim product, dim customer, and fact internet sales; aggregate by customer.

Explore outer joins, left, right, and full, alongside inner joins, using sales tran and products. Understand non matching rows and why full outer join is rarely used.

Learn to perform a self join by joining a table to itself to reveal each employee's manager name and identify cases where employees earn more than their managers through salary.

Explore the iif function in T-SQL for conditional execution, compare it with the if statement, and build nested conditions to generate color codes and price categories.

Explore how to use iif and case to apply multiple conditions on the dim product table, combining color and list price to categorize red and silver items.

Explore how union and union all combine rows from two tables and handle duplicates. Identify identical column counts and sequences, with nulls to align schemas.

Explore the sql except keyword and its difference from union, union all, and intersect, noting that order matters and except relates to left join and anti-join ideas.

Explore update statements with subqueries and joins in sql to increase employee base rates by 10% and target Canada using sales territory keys.

Explains using subqueries to compute per-customer transaction counts and derive an intermediate table, then calculate the average transactions per customer via a derived table.

Learn to use the top clause in T-SQL to fetch the first n rows (top 100, top 10%), and how bottom results rely on order by and a numeric key.

Explore the basics of window functions with row_number, learn the over clause syntax, and see how partition by and order by produce unique, resettable row numbers across data groups.

Explore how rank and dense_rank, two window functions, rank employees by salary using over and order by, highlighting duplicates, skipping ranks, and partitioning by city to compare within groups.

Classify sql into ddl, dml, dql, tcl, and dcl, and learn how create, alter, drop, insert, update, delete, select, commit, rollback, save point, and grant statements define each category.

Learn how to manage constraints with alter table, including setting not null, adding primary keys and foreign keys with references, and dropping constraints.

Discover how offset skips rows in SQL, requiring an order by, to fetch a specific range such as rows 11 to 20, with an optional fetch first n rows.

Learn how to use the coalesce function to consolidate multiple frequency columns into a single yearly amount by selecting the first non-null value and multiplying by 2, 4, or 12.

Learn to use the sql merge statement to synchronize a target and source stage table by inserting new records, updating existing ones, and optional deletes; understand identity inserts and upsert.

Learn how schema provides a logical boundary to organize tables, with dbo as default, how to create and manage schemas, and why schemas simplify permissions and access control.

Learn how group by with rollup creates grand totals and subtotals for year and month, using isnull and format to display clean summaries of order date and sales amount.

Learn how to use group by with cube in SQL, compare cube and rollup, and generate all dimension groupings, including subtotals and grand totals for multi-column queries.

Learn how the pivot operator transforms rows into columns and aggregates salaries in SQL. Create a two-dimensional view with city and department, using a sub query and dynamic pivoting.

Learn to build multi-part common table expressions using with, define sequential parts, and join them to simplify complex queries, illustrated by a dim product and net sales example.

Explore declaring, assigning, and retrieving variables in T-SQL using declare, set, and select; understand default values, nulls, and handling multiple variables.

This lecture introduces the while loop in t-sql, showing how to declare a counter, implement a condition, increment it, and use break to terminate, with begin and end blocks.

Explore temp tables in SQL, including local (single hash) and global (double hash) types stored in temp db, used for staging intermediate results with automatic cleanup when the connection closes.

Learn to create nested JSON output from SQL queries using master-detail structures, with child fields like product name, price, and dates such as start date, demonstrated on AdventureWorks.

Explore the SQL output clause, using inserted and deleted to return affected rows during insert, delete, update, and merge operations, including table variables and practical examples.

Explore how indexes improve data retrieval in SQL Server, focusing on clustered indexes and the automatic primary key index, and compare table scans with clustered index scans via execution plans.

Explore the theory of SQL Server indexes, including clustered and non-clustered indexes, heaps, pages and extents, IAM, partitions, file groups, and the B-tree structure that enables fast data retrieval.

Explore how index seek, index scan, and table scan influence query performance using the Adventureworks customers table, with clustered and non-clustered indexes, and a composite index on names.

Learn how delete and update cascade enforce referential integrity by automatically deleting or updating child records when the parent key changes, using foreign key constraints.

Explore string_agg and string_split in SQL Server: string_agg concatenates rows into a single string with a separator, while string_split returns rows from a delimited string.

Learn how cross-apply and outer-apply enable row-by-row execution of table-valued functions, overcoming join limitations when joining a table to a TVF.

Explore how sp_executesql, a system stored procedure, safely executes dynamic sql with parameters, demonstrates unicode nvarchar handling, and protects against sql injection.

Explore SQL across 100 videos in the Azure data engineering end-to-end course, delivering practical SQL insights for building robust data solutions.

Explore how untrusted input enables SQL injection through dynamic SQL and concatenation, risks like data leakage and data manipulation, and how parameterized queries and sp_executesql safeguard pipelines and stored procedures.

Learn to build a dynamic pivot in SQL by turning distinct product values into pivoted columns with string aggregate and code name, then execute dynamic SQL to pivot multiple measures.

Explore inner, left, right, full outer, and cross joins on two tables, and learn how nulls behave, including that two nulls are not the same, and affect row counts.

Identify and delete duplicate rows using the row_number window function, partitioning by columns, and execute via subquery or a common table expression to remove duplicates.

Learn how to perform custom merging with a common table expression using accept to identify changed or new rows, and merge from employees_stage to employees updating and inserting by employee_id.

Identify missing departments in cities by generating all city–department combinations with distinct cities and departments, then left joining to employees to flag nulls.

Learn how to retrieve the second largest salary in SQL using multiple approaches, including subqueries, order by with top, and a dense_rank-based common table expression to handle ties.

Partition data by department and rank salaries to identify the second largest in each department. Filter where the rank equals two to return the second largest salary per department.

Find alternate rows in a table by using the modulo operator to pick odd or even rows, and apply row number with cte or subquery when no numeric column exists.

Calculate the month-to-date running total (MTD) using a common table expression and over clause, resetting at each month, and compare with QTD and YTD using Adventureworks net sales data.

Learn to compute quarter-to-date totals by partitioning by quarter and year and using sum(total_sales) over to reset totals across calendar quarters in Adventureworks data.

Learn to compute year to date totals in SQL using sum(total_sales) over partition by year order by order_date, with a CTE named YTD and options for daily or monthly views.

Learn to calculate previous year totals and year-on-year growth using a lag window function in SQL, with the Adventureworks data, grouping by year, and using a multi-part CTE.

Learn to compute a three month moving total with a cte and a window frame rows between two preceding and current row on Adventureworks, using sum, average, min, or max.

Learn the fundamentals of data warehousing, including ETL and ELT, data sources, and common tools, with a view to end-to-end data pipelines and BI integration.

Explore data loads in data warehousing, including full load with truncate, upsert with keys, incremental loads using date columns, and slowly changing dimensions types 1–3 in Data Factory and PySpark.

Understand hash distribution by applying a hash function and modulo operation to assign rows to partitions based on a key column, boosting performance for large fact tables.

Explore acid properties—atomicity, consistency, isolation, and durability—and learn how transactions either complete fully or roll back, with practical money-transfer examples and delta lake relevance.

Explore cloud computing basics by learning how internet-based services enable data lakes storage, sql databases, etl processes, and virtual machines for modern data engineering on cloud platforms.

Explore the top cloud providers—AWS, Google Cloud Platform, and Microsoft Azure—and learn how Azure data engineering drives growing demand for Azure engineers and architects.

Explore Azure as a cloud platform, highlighting its no-installation, pay-as-you-go, and fully managed services, with a focus on Data Factory, Databricks, DevOps, and Synapse in a 28-day free trial.

Sign up for a free Azure trial by creating a Microsoft account, verifying via email, and entering payment details to access the Azure portal and $200 credit.

Create and provision an Azure Data Lake Storage Gen2 account within a resource group, then create containers, upload files, and organize them with directories for scalable data storage.

Provision Azure Data Factory and design, run, and monitor data pipelines. Learn ETL and ELT basics, set up a v2 data factory, and review billing during the free trial.

Upgrade your Azure free trial to pay-as-you-go in the portal, verify address. Verify pan id for Indian residents and choose the basic plan to avoid costly add-ons.

Learn how Azure Data Factory acts as an ETL tool for moving and transforming data through pipelines, activities, and data sets, with 90 plus connectors and secure link services.

Learn the fundamental building blocks of Azure Data Factory, including pipelines, activities, data sets, link services, integration runtime, and how they orchestrate data movement, transformation, and control tasks.

Design a data factory pipeline to copy a CSV file from an input to an output folder in a data lake Gen2, using copy data activity, datasets, and link services.

Use the data factory copy data activity to copy an entire folder or specific files from a data lake, applying wildcard patterns and file-type filters.

Copy data from a data lake to a SQL table, adding a file name column, using wildcard paths and additional columns in Azure Data Factory.

Learn how to create and use set variable activity in Azure Data Factory, including variable types, default values, and serial or parallel execution, with overwriting behavior.

Learn to copy data in Azure Data Factory within a dynamic time frame by filtering files modified in the last hour, 24 hours, or other intervals using expressions and functions.

Use Azure Data Factory to copy each data lake file into a new SQL table using Get metadata and for each, name the table from the file name without extension.

Explore column mappings in copy data activity to align source and target schemas, import schemas, and map or drop columns when moving data from data lake to Azure SQL database.

Execute stored procedures in Azure Data Factory using the stored procedure activity with parameters. Learn why it doesn't return data, compare it to copy activity, and see a truncate-and-load example.

Use the lookup activity to read data from sources such as csv files or database tables and return rows, with options for first row only or all rows.

Learn to use the filter activity in Azure Data Factory to filter an array with expressions and output items greater than three.

Explore the if activity in Azure Data Factory, parameterize expressions, and route to true or false paths. Use dynamic content and equals to debug set variables and SQL-like control flow.

Learn to use the script activity in Data Factory to run ad hoc SQL statements on a SQL database, performing DDL and DML without returning data.

Convert a CSV file to JSON in Azure Data Factory using a copy data activity, with a JSON dataset stored in Data Lake Gen2, and understand JSON structure.

Copy json files to SQL database using the copy data activity in Azure Data Factory, covering simple and nested json with mapping to aligned SQL tables.

Learn how the execute pipeline activity calls a child pipeline from a master pipeline, enabling serial or parallel execution and parameter passing, with debugging via run IDs and outputs.

Evaluate the difference between parameters and variables in data factory pipelines, noting that parameters stay fixed during execution while variables can be updated, with data type differences and usage guidelines.

Delete blank files in a data factory pipeline by listing files with get metadata, filtering to files, and deleting those under 1 kb.

Learn to copy a headerless CSV to a database using Azure Data Factory, configure a CSV dataset and mapping, handle header absence, adjust ordinal mappings, and troubleshoot type conversion errors.

Explore copy behavior in Azure Data Factory's copy data activity, demonstrating preserve hierarchy, flatten hierarchy, and merge files when moving data between input and output folders in a data lake.

Split data by a criterion in azure data factory using a lookup for countries, then loop with for-each to generate country-specific csv files in data lake named after each country.

Learn to split data by multiple criteria in azure data factory, creating dynamic country and region folders and exporting filtered rows to per-country region csv files.

Consolidate data from multiple csv files into a single table using a data factory pipeline. Dynamically derive country names from file names and use for-each looping with a truncate script.

Consolidate data from multiple folders using Azure Data Factory by listing countries with a master pipeline and regions with a child pipeline, then copy data with country and region columns.

Learn to copy data with a pipe delimiter by changing the source dataset column delimiter from comma to pipe, then preview and import schema to fix the error.

Discover how the quote character influences delimited data in a data factory copy activity, and choose double quote, single quote, or no quote to avoid incorrect splits.

Discover how data flows in Azure Data Factory enable data transformations within a pipeline, using sources, data sets, projections, and previews to shape and validate data.

Learn data flow transformations in Azure Data Factory, use the select transformation to choose and rename columns, and preview before publishing to an Azure SQL database sink.

Learn the sort transformation in Data Factory data flows, sorting by country (and sales) with ascending or descending order and default null handling. See previews, debugging, and single-partition behavior.

Master the data flow filter transformation using the expression builder to create boolean conditions, emulating a SQL where clause, preview results, and filter India records.

Learn how to use the derived column transformation to add or update columns, deriving year, month, day, and month name from order dates, and write results to a new table.

Leverage the conditional split transformation in data flows to route rows by expressions into streams like India and France, with dedicated sinks for each output.

Master the cast transformation in Azure Data Factory to change data types after a series of transformations, selecting columns such as date, short, text, and double for accurate casting.

Learn to use aggregate transformations in Azure Data Factory data flows, combining group by and multiple aggregates (sum, average, min, max), with expression builder for custom calculations and sink output.

Learn pivot transformation in Azure Data Factory to convert rows into columns, producing a product-country two-dimensional summary using group by, pivot key, and pivot columns with sales amounts.

Apply window transformation to compute running totals of sales by date, using aggregate then window functions with partition by year and month to reset totals per period.

Learn how the union transformation in Data Factory appends rows from multiple sources, using union by name or by position, handling data type alignment, and writing the combined result.

Learn to use a lookup transformation in data flows to add product and customer names to sales using product key and customer key. Explore lookups, left-join behavior, and duplicates handling.

Learn how to perform inner and left outer join transformations in Azure Data Factory, join two or more datasets, manage keys and mappings, and preview and monitor results.

Apply the exist transformation in data factory to check for product key existence across sales and products, returning true or false and distinguishing exists vs doesn't exist.

Flatten json in azure data factory using the flatten transformation to unroll items into multiple rows. Configure json settings, select array of documents, and map to a sql sink.

Learn how to use the parse transformation to split pipe-delimited and JSON data in a data flow, define output columns, and map split results to a sink.

Learn how to use the stringify transformation in a data flow to convert a nested phone field into a string, flattening complex data for a delimited data lake output.

Explore the integration runtime as the compute infrastructure powering data movement in Azure Data Factory and Azure Synapse, with three types: Azure, self-hosted, and Azure Sis.

Learn how to install and register the self-hosted integration runtime for data migration from on-premises to Azure Data Lake or Azure SQL Database, including setup steps and validation.

Learn to copy data from local or on premises to Azure SQL DB, staging in data lake as a landing zone, and push to Azure SQL DB with 1000-row example.

Discover how to provision Azure Key Vault, store secrets, and grant Data Factory access via access policies; securely retrieve passwords for link services and data transfers.

Create and configure a schedule trigger in Azure Data Factory to run pipelines automatically at designated times, set recurrence and time zone, attach multiple pipelines, then publish and monitor results.

Learn how storage event triggers in Azure Data Factory copy new files as soon as they land in the Data Lake, including parameterized pipelines and Event Grid setup.

Load incremental data from data lake to sql db using data flows in azure data factory, eliminating staging by using max date from a lookup and parameterized filtering.

Load incremental data from Azure Data Lake Storage to sql database for multiple tables using file timestamp, with a mapping table, get metadata, and a max timestamp update workflow.

Learn to perform incremental load without a date or unique key by generating a hash key from concatenated columns using sha-1 in Azure Data Factory data flows.

Build a metadata-driven data integration framework using Azure Data Factory and SQL to process CSV files in Azure Data Lake with auditable file metadata and staging to data warehouse.

Load dimension tables using scd type 2 with staging and final tables in the dw schema. Implement a metadata-driven adf pipeline for full and incremental loads with hash detection.

Discover the Databricks platform, its features like clusters, data engineering, Lakehouse, and PySpark, and how to sign up with Community Edition or paid options across Azure, AWS, and GCP.

Sign up for the Databricks free edition to learn Python, SQL, and PySpark; create a new account, complete a quick puzzle, and access a serverless compute warehouse with no card.

Explore the Databricks platform with a hands-on tour of compute, clusters, notebooks, and workspace in the community edition, and preview Python, PySpark, lakehouse, Delta Lake, and SQL/Scala/R workflows.