Azure Data Engineering End-to-end Course 2026 (English)

Learn to create and drop databases using SQL queries and the interface. Execute create and drop statements, ensure unique names, handle errors, refresh the list, and save queries as .sql.

Explore the four system databases—master, model, msdb, and tempdb—and their roles, from starting the SQL Server to templates, jobs, alerts, and temporary objects, and never touch them.

Learn to use the select statement to fetch data from the employees table, choosing all columns or specific ones like employee ID, name, and salary, while avoiding select star.

Learn to use sql comments in tsql on sql server, including double dash single-line and slash star block comments, to document queries and aid end users.

learn to filter data with the where clause in sql, using boolean expressions and comparison operators, and combine conditions with or, and, in, and between through practical examples.

Explore date and time basics in SQL, including get date, date time, date time two, UTC date time, precision differences, and creating dates with date from parts.

Learn how to extract year, month, and day with date time functions, compare date part and date name outputs, and format dates for display using the format function in sql.

Explore approximate numeric data types in SQL, including float, real, money, decimal, and numeric; learn precision, scale, ranges, and storage implications with practical table examples.

Learn data type conversion in SQL, including implicit and explicit casting with cast and convert, handling string and numeric operations, and using money and int conversions.

Learn to join tables using product key and customer key to show customer names, product names, and sales from dim product, dim customer, and fact internet sales; aggregate by customer.

Learn to perform a self join by joining a table to itself to reveal each employee's manager name and identify cases where employees earn more than their managers through salary.

Explore the iif function in T-SQL for conditional execution, compare it with the if statement, and build nested conditions to generate color codes and price categories.

Explore how to use iif and case to apply multiple conditions on the dim product table, combining color and list price to categorize red and silver items.

Explore how union and union all combine rows from two tables and handle duplicates. Identify identical column counts and sequences, with nulls to align schemas.

Explore the sql except keyword and its difference from union, union all, and intersect, noting that order matters and except relates to left join and anti-join ideas.

Explore update statements with subqueries and joins in sql to increase employee base rates by 10% and target Canada using sales territory keys.

Learn to use the top clause in T-SQL to fetch the first n rows (top 100, top 10%), and how bottom results rely on order by and a numeric key.

Explore how rank and dense_rank, two window functions, rank employees by salary using over and order by, highlighting duplicates, skipping ranks, and partitioning by city to compare within groups.

Classify sql into ddl, dml, dql, tcl, and dcl, and learn how create, alter, drop, insert, update, delete, select, commit, rollback, save point, and grant statements define each category.

Learn how to manage constraints with alter table, including setting not null, adding primary keys and foreign keys with references, and dropping constraints.

Discover how offset skips rows in SQL, requiring an order by, to fetch a specific range such as rows 11 to 20, with an optional fetch first n rows.

Learn how to use the coalesce function to consolidate multiple frequency columns into a single yearly amount by selecting the first non-null value and multiplying by 2, 4, or 12.

Learn how schema provides a logical boundary to organize tables, with dbo as default, how to create and manage schemas, and why schemas simplify permissions and access control.

Learn how group by with rollup creates grand totals and subtotals for year and month, using isnull and format to display clean summaries of order date and sales amount.

Learn to build multi-part common table expressions using with, define sequential parts, and join them to simplify complex queries, illustrated by a dim product and net sales example.

Learn to create nested JSON output from SQL queries using master-detail structures, with child fields like product name, price, and dates such as start date, demonstrated on AdventureWorks.

Explore the SQL output clause, using inserted and deleted to return affected rows during insert, delete, update, and merge operations, including table variables and practical examples.

Explore how index seek, index scan, and table scan influence query performance using the Adventureworks customers table, with clustered and non-clustered indexes, and a composite index on names.

Learn how delete and update cascade enforce referential integrity by automatically deleting or updating child records when the parent key changes, using foreign key constraints.

Explore string_agg and string_split in SQL Server: string_agg concatenates rows into a single string with a separator, while string_split returns rows from a delimited string.

Learn how cross-apply and outer-apply enable row-by-row execution of table-valued functions, overcoming join limitations when joining a table to a TVF.

Identify and delete duplicate rows using the row_number window function, partitioning by columns, and execute via subquery or a common table expression to remove duplicates.

Learn how to perform custom merging with a common table expression using accept to identify changed or new rows, and merge from employees_stage to employees updating and inserting by employee_id.

Identify missing departments in cities by generating all city–department combinations with distinct cities and departments, then left joining to employees to flag nulls.

Learn how to retrieve the second largest salary in SQL using multiple approaches, including subqueries, order by with top, and a dense_rank-based common table expression to handle ties.

Partition data by department and rank salaries to identify the second largest in each department. Filter where the rank equals two to return the second largest salary per department.

Calculate the month-to-date running total (MTD) using a common table expression and over clause, resetting at each month, and compare with QTD and YTD using Adventureworks net sales data.

Learn to compute year to date totals in SQL using sum(total_sales) over partition by year order by order_date, with a CTE named YTD and options for daily or monthly views.

Learn the fundamentals of data warehousing, including ETL and ELT, data sources, and common tools, with a view to end-to-end data pipelines and BI integration.

Explore data loads in data warehousing, including full load with truncate, upsert with keys, incremental loads using date columns, and slowly changing dimensions types 1–3 in Data Factory and PySpark.

Explore cloud computing basics by learning how internet-based services enable data lakes storage, sql databases, etl processes, and virtual machines for modern data engineering on cloud platforms.

Explore the top cloud providers—AWS, Google Cloud Platform, and Microsoft Azure—and learn how Azure data engineering drives growing demand for Azure engineers and architects.

Sign up for a free Azure trial by creating a Microsoft account, verifying via email, and entering payment details to access the Azure portal and $200 credit.

Create and provision an Azure Data Lake Storage Gen2 account within a resource group, then create containers, upload files, and organize them with directories for scalable data storage.

Upgrade your Azure free trial to pay-as-you-go in the portal, verify address. Verify pan id for Indian residents and choose the basic plan to avoid costly add-ons.

Learn to copy data in Azure Data Factory within a dynamic time frame by filtering files modified in the last hour, 24 hours, or other intervals using expressions and functions.

Explore column mappings in copy data activity to align source and target schemas, import schemas, and map or drop columns when moving data from data lake to Azure SQL database.

Execute stored procedures in Azure Data Factory using the stored procedure activity with parameters. Learn why it doesn't return data, compare it to copy activity, and see a truncate-and-load example.

Learn to use the filter activity in Azure Data Factory to filter an array with expressions and output items greater than three.

Learn how the execute pipeline activity calls a child pipeline from a master pipeline, enabling serial or parallel execution and parameter passing, with debugging via run IDs and outputs.

Evaluate the difference between parameters and variables in data factory pipelines, noting that parameters stay fixed during execution while variables can be updated, with data type differences and usage guidelines.

Delete blank files in a data factory pipeline by listing files with get metadata, filtering to files, and deleting those under 1 kb.

Learn to copy a headerless CSV to a database using Azure Data Factory, configure a CSV dataset and mapping, handle header absence, adjust ordinal mappings, and troubleshoot type conversion errors.

Explore copy behavior in Azure Data Factory's copy data activity, demonstrating preserve hierarchy, flatten hierarchy, and merge files when moving data between input and output folders in a data lake.

Learn to split data by multiple criteria in azure data factory, creating dynamic country and region folders and exporting filtered rows to per-country region csv files.

Consolidate data from multiple csv files into a single table using a data factory pipeline. Dynamically derive country names from file names and use for-each looping with a truncate script.

Consolidate data from multiple folders using Azure Data Factory by listing countries with a master pipeline and regions with a child pipeline, then copy data with country and region columns.

Learn to copy data with a pipe delimiter by changing the source dataset column delimiter from comma to pipe, then preview and import schema to fix the error.

Discover how data flows in Azure Data Factory enable data transformations within a pipeline, using sources, data sets, projections, and previews to shape and validate data.

Leverage the conditional split transformation in data flows to route rows by expressions into streams like India and France, with dedicated sinks for each output.

Learn to use aggregate transformations in Azure Data Factory data flows, combining group by and multiple aggregates (sum, average, min, max), with expression builder for custom calculations and sink output.

Learn pivot transformation in Azure Data Factory to convert rows into columns, producing a product-country two-dimensional summary using group by, pivot key, and pivot columns with sales amounts.

Apply window transformation to compute running totals of sales by date, using aggregate then window functions with partition by year and month to reset totals per period.

Learn to use a lookup transformation in data flows to add product and customer names to sales using product key and customer key. Explore lookups, left-join behavior, and duplicates handling.

Learn how to perform inner and left outer join transformations in Azure Data Factory, join two or more datasets, manage keys and mappings, and preview and monitor results.

Learn how to use the stringify transformation in a data flow to convert a nested phone field into a string, flattening complex data for a delimited data lake output.

Learn how to install and register the self-hosted integration runtime for data migration from on-premises to Azure Data Lake or Azure SQL Database, including setup steps and validation.

Learn to copy data from local or on premises to Azure SQL DB, staging in data lake as a landing zone, and push to Azure SQL DB with 1000-row example.

Learn how storage event triggers in Azure Data Factory copy new files as soon as they land in the Data Lake, including parameterized pipelines and Event Grid setup.

Load incremental data from Azure Data Lake Storage to sql database for multiple tables using file timestamp, with a mapping table, get metadata, and a max timestamp update workflow.

Build a metadata-driven data integration framework using Azure Data Factory and SQL to process CSV files in Azure Data Lake with auditable file metadata and staging to data warehouse.

Discover the Databricks platform, its features like clusters, data engineering, Lakehouse, and PySpark, and how to sign up with Community Edition or paid options across Azure, AWS, and GCP.

Sign up for the Databricks free edition to learn Python, SQL, and PySpark; create a new account, complete a quick puzzle, and access a serverless compute warehouse with no card.

Explore the Databricks platform with a hands-on tour of compute, clusters, notebooks, and workspace in the community edition, and preview Python, PySpark, lakehouse, Delta Lake, and SQL/Scala/R workflows.