
Join this Azure Data Engineering course to master SQL, Azure, ETL pipelines with Azure Data Factory, Python, Databricks, and big data tools, starting from scratch with notes and updates.
Explore why most people fail to learn and how this insight applies to the Azure Data Engineering end-to-end course 2026.
Explore CDU content 3.0 in the Azure data engineering end-to-end course (English), focusing on the key CDU content for data engineering learning.
We updated the Azure Data Engineering course with Databricks and PySpark, adding new interfaces and Unity Catalog, with English and Hindi content, 8.5 hours, and more focus on Spark internals.
Explore SQL basics, the structured query language for data warehousing. See how core SQL syntax stays consistent across major databases, with Azure SQL as a practical starting point.
Explore the history of SQL as part of the Azure data engineering end-to-end course 2026.
Download SQL Server 2025 developer edition from the official Microsoft site, install the database engine, then install SQL Server Management Studio 22, restart, and connect by trusting the certificate.
Learn to create and drop databases using SQL queries and the interface. Execute create and drop statements, ensure unique names, handle errors, refresh the list, and save queries as .sql.
Explore SQL Server hierarchy within the Azure data engineering end-to-end course, learning how to represent and navigate hierarchical data.
Explore the four system databases—master, model, msdb, and tempdb—and their roles, from starting the SQL Server to templates, jobs, alerts, and temporary objects, and never touch them.
Create and drop tables in SQL server, defining columns and data types like int and varchar, and practicing table design through both interface and query syntax.
Learn to insert data into a table with an explicit column list, use values, handle quoted strings and dates, and insert multiple rows for verification with a select.
Learn to use the select statement to fetch data from the employees table, choosing all columns or specific ones like employee ID, name, and salary, while avoiding select star.
Dowload the sample database attached.
Learn to get unique values from single or multiple columns in AdventureWorks using select distinct on dim product, a virtual extraction that does not modify the table.
Sort data in SQL using the order by clause with ascending and descending options. Explore multi-column sorting, master and child column behavior, and ASC as the default.
Learn to use sql comments in tsql on sql server, including double dash single-line and slash star block comments, to document queries and aid end users.
learn to filter data with the where clause in sql, using boolean expressions and comparison operators, and combine conditions with or, and, in, and between through practical examples.
Learn pattern-based filtering using the like operator with wildcards in sql, including percentage and underscore, to filter begins with, ends with, and contains in text and numeric columns.
Learn aggregate functions in sql, sum, avg, min, max, and count on fact internet sales table; alias results with as and handle nulls with is null and is not null.
Learn how to group rows with the group by clause, aggregate sales by product key, and distinguish group by from distinct to calculate total sales per key.
Learn how to copy a table with select into, creating a backup table that duplicates data, filters rows, or copies only the structure for safe experimentation.
Learn how to enforce uniqueness and non-null constraints using a primary key when creating tables in SQL, preventing duplicate and null employee IDs.
Apply the not null constraint to table columns to prevent null values and ensure critical fields, like employee name, are always populated.
Discover how the unique constraint enforces uniqueness, differs from primary key and not null, supports multiple constraints (up to 999 columns), and explains null handling.
Create a table with a check constraint to enforce salary above 10,000 and gender values m, f, or o; test inserts, generate script, and learn to modify via alter table.
Learn how the SQL default constraint works by adding a city column with a default of London in a create table statement, and see how NYC overrides it.
Learn how to implement auto increment in a create table script using identity, with starting point and increment by options, including optional primary key and identity insert behavior.
Update records in a table with the update statement, using where clauses to target rows and update multiple columns, including a 10% price increase, with backup via select into.
Learn how to delete rows from a table, use where clauses, and differentiate delete from truncate and drop tables, with notes on identity reset and backup standards.
Learn core string functions in SQL, including upper, lower, left, right, substring, and len, with practical examples like English product name, plus trim, l trim, and r trim.
Explore string functions such as replace, reverse, and stuff to transform text. Apply 1-to-1, 1-to-many, and many-to-many substitutions, including dropping with a blank and inserting characters at specific positions.
Learn to build full names by concatenating first, middle, and last names with plus, space, concat, and concat_ws, handle nulls, and update a table.
Extracts first and last names from a full name using dynamic left, right, and len functions with charindex to handle spaces and middle names, demonstrating nested function patterns.
Explore date and time basics in SQL, including get date, date time, date time two, UTC date time, precision differences, and creating dates with date from parts.
Learn how to extract year, month, and day with date time functions, compare date part and date name outputs, and format dates for display using the format function in sql.
Learn how to use dateadd to add or subtract years, months, and days; compare dates with date difference; and find month ends with eomonth for current or specified dates.
Learn the four integer data types in t-sql—tinyint, smallint, int, and bigint—discover their ranges and storage sizes, and how to choose the right type to prevent overflow.
Explore approximate numeric data types in SQL, including float, real, money, decimal, and numeric; learn precision, scale, ranges, and storage implications with practical table examples.
Identify and compare date, date time, date time two, small date time, and time data types, focusing on their ranges and storage in bytes.
Explore string data types such as care and varchar, including fixed versus variable length, unicode variants like n care and n where care, and their max limits.
Explore the unique identifier data type in SQL, learn how to generate new ID values with the new ID function, and understand its use as a primary key across tables.
Learn how the bit data type stores true or false as 1 or 0 in SQL Server, using a table with an is_active bit column, inserts and select results.
Learn data type conversion in SQL, including implicit and explicit casting with cast and convert, handling string and numeric operations, and using money and int conversions.
Master inner, left, and right joins to combine data across tables using an on condition and common columns, and learn how non-matching rows yield nulls.
Apply inner and left joins on dim product and fact internet sales to combine keys, analyze sales amount, and identify non matched products with left anti join.
Learn to join tables using product key and customer key to show customer names, product names, and sales from dim product, dim customer, and fact internet sales; aggregate by customer.
Learn to write join expressions that align data types between dim products and fact internet sales, using on conditions, functions, and casting. Extract the right four digits from product keys.
Explore outer joins, left, right, and full, alongside inner joins, using sales tran and products. Understand non matching rows and why full outer join is rarely used.
Learn to perform a self join by joining a table to itself to reveal each employee's manager name and identify cases where employees earn more than their managers through salary.
Explore how dimension tables and fact tables model data in data warehousing, with examples from Adventureworks. Visualize star and snowflake schemas connecting dimensions to the central fact table.
Explore the iif function in T-SQL for conditional execution, compare it with the if statement, and build nested conditions to generate color codes and price categories.
Learn to use the SQL case statement to transform colors into codes, compare numeric ranges, handle nulls, and nest cases for robust data categorization.
Explore how to use iif and case to apply multiple conditions on the dim product table, combining color and list price to categorize red and silver items.
Master custom sorting in SQL by using a case expression in the order by clause to prioritize departments, with optional ascending or descending output.
Explore how union and union all combine rows from two tables and handle duplicates. Identify identical column counts and sequences, with nulls to align schemas.
Explore how the intersect keyword returns only the common rows between two tables, unlike union. Compare its behavior to an inner join when all columns are considered.
Explore the sql except keyword and its difference from union, union all, and intersect, noting that order matters and except relates to left join and anti-join ideas.
Learn how the foreign key constraint enforces referential integrity between master (dimension) and transaction (fact) tables, preventing orphan records by linking to primary keys in a star schema.
Learn to use subqueries to dynamically fetch rows with the maximum sales amount, and explore minimums and averages with inner and outer queries and brackets.
Explore how to apply subqueries with in and not in to filter fact internet sales by color red, using join and subquery approaches, and preview common table expressions.
Explore update statements with subqueries and joins in sql to increase employee base rates by 10% and target Canada using sales territory keys.
Explains using subqueries to compute per-customer transaction counts and derive an intermediate table, then calculate the average transactions per customer via a derived table.
Explore exists and not exists in subqueries, compare with in. Learn how correlated subqueries reference the outer table to filter rows using the color red.
Learn how to use subqueries to calculate total sales by year and each year's contribution percentage to overall sales, using a computed column and percentage formatting in SQL.
Learn how the having clause filters aggregated totals after grouping by product key, using total sales, and compare its execution with where before group by.
Learn to use the top clause in T-SQL to fetch the first n rows (top 100, top 10%), and how bottom results rely on order by and a numeric key.
Identify top or bottom selling products by sales amount using group by and order by, with top ten or bottom ten clauses, and join dim product to get product names.
Explore the basics of window functions with row_number, learn the over clause syntax, and see how partition by and order by produce unique, resettable row numbers across data groups.
Explore how rank and dense_rank, two window functions, rank employees by salary using over and order by, highlighting duplicates, skipping ranks, and partitioning by city to compare within groups.
Explore lag and lead window functions to fetch previous and next rows with the over clause, using partition by city and order by date. Handle offset and default for nulls.
Explore window functions in SQL with first_value and last_value, learn how the over clause and partition by city shape results, and see practical examples using order date and quantity.
Classify sql into ddl, dml, dql, tcl, and dcl, and learn how create, alter, drop, insert, update, delete, select, commit, rollback, save point, and grant statements define each category.
Master alter table operations to modify a table's structure by adding, dropping, and altering columns, while understanding data types, constraints, and required permissions.
Learn how to manage constraints with alter table, including setting not null, adding primary keys and foreign keys with references, and dropping constraints.
Discover how offset skips rows in SQL, requiring an order by, to fetch a specific range such as rows 11 to 20, with an optional fetch first n rows.
Learn how to use the coalesce function to consolidate multiple frequency columns into a single yearly amount by selecting the first non-null value and multiplying by 2, 4, or 12.
Learn to use the sql merge statement to synchronize a target and source stage table by inserting new records, updating existing ones, and optional deletes; understand identity inserts and upsert.
Learn how schema provides a logical boundary to organize tables, with dbo as default, how to create and manage schemas, and why schemas simplify permissions and access control.
Learn how group by with rollup creates grand totals and subtotals for year and month, using isnull and format to display clean summaries of order date and sales amount.
Learn how to use group by with cube in SQL, compare cube and rollup, and generate all dimension groupings, including subtotals and grand totals for multi-column queries.
Learn how the pivot operator transforms rows into columns and aggregates salaries in SQL. Create a two-dimensional view with city and department, using a sub query and dynamic pivoting.
Discover how the unpivot operator transforms columns into rows from base pivot data, turning city and sales into department and salary values, without aggregation.
Learn how views create virtual tables in SQL, acting as a security layer over tables and enabling you to join multiple tables, create, alter, or drop views for reusable queries.
Explore common table expressions (CTE) as a temporary named result set; learn the with clause structure and how outer queries use CTEs to improve readability versus subqueries.
Learn to build multi-part common table expressions using with, define sequential parts, and join them to simplify complex queries, illustrated by a dim product and net sales example.
Explore recursive common table expressions in SQL, including anchor and recursive members, and how union all enables the recursion. Learn to limit recursion with a where clause and max recursion.
Explore declaring, assigning, and retrieving variables in T-SQL using declare, set, and select; understand default values, nulls, and handling multiple variables.
Explore the if else statement and its difference from the if function and case expression. Learn to use begin and end to group logic and nest conditions.
This lecture introduces the while loop in t-sql, showing how to declare a counter, implement a condition, increment it, and use break to terminate, with begin and end blocks.
Explore temp tables in SQL, including local (single hash) and global (double hash) types stored in temp db, used for staging intermediate results with automatic cleanup when the connection closes.
Explore stored procedures in SQL, learn how they encapsulate business logic and support parameters for dynamic queries. Compare them with views and understand creation, execution, and modification.
Learn to convert a table to JSON using for JSON path, including handling null values, root level, and options such as without array wrapper, and contrast with for JSON auto.
Learn to create nested JSON output from SQL queries using master-detail structures, with child fields like product name, price, and dates such as start date, demonstrated on AdventureWorks.
Explore the SQL output clause, using inserted and deleted to return affected rows during insert, delete, update, and merge operations, including table variables and practical examples.
Learn how to implement slowly changing dimensions in sql using merge to handle scd types 1, 2, and 3, including preserving history with start date, end date, and is active.
Explore how indexes improve data retrieval in SQL Server, focusing on clustered indexes and the automatic primary key index, and compare table scans with clustered index scans via execution plans.
Learn how to design and use clustered and non-clustered indexes to speed filtering, sorting, and joining, including covering indexes and uniqueness constraints, with execution plans.
Explore the theory of SQL Server indexes, including clustered and non-clustered indexes, heaps, pages and extents, IAM, partitions, file groups, and the B-tree structure that enables fast data retrieval.
Explore how index seek, index scan, and table scan influence query performance using the Adventureworks customers table, with clustered and non-clustered indexes, and a composite index on names.
Learn how delete and update cascade enforce referential integrity by automatically deleting or updating child records when the parent key changes, using foreign key constraints.
Explore string_agg and string_split in SQL Server: string_agg concatenates rows into a single string with a separator, while string_split returns rows from a delimited string.
Create user-defined functions in SQL Server, including scalar functions, inline table-valued functions, and multi-statement table-valued functions, compare with stored procedures, and learn parameters, returns, and schemas.
Learn how cross-apply and outer-apply enable row-by-row execution of table-valued functions, overcoming join limitations when joining a table to a TVF.
The Codename function returns a Unicode nvarchar wrapped in square brackets to create delimited identifiers and guard against SQL injection in dynamic SQL.
Explore how sp_executesql, a system stored procedure, safely executes dynamic sql with parameters, demonstrates unicode nvarchar handling, and protects against sql injection.
Explore SQL across 100 videos in the Azure data engineering end-to-end course, delivering practical SQL insights for building robust data solutions.
Explore how untrusted input enables SQL injection through dynamic SQL and concatenation, risks like data leakage and data manipulation, and how parameterized queries and sp_executesql safeguard pipelines and stored procedures.
Learn to build a dynamic pivot in SQL by turning distinct product values into pivoted columns with string aggregate and code name, then execute dynamic SQL to pivot multiple measures.
In this SQL scenarios series, learn to find the average of a column and see that null does not change it while a blank lowers it via implicit casting.
Explore inner, left, right, full outer, and cross joins on two tables, and learn how nulls behave, including that two nulls are not the same, and affect row counts.
Identify and delete duplicate rows using the row_number window function, partitioning by columns, and execute via subquery or a common table expression to remove duplicates.
Learn how to perform custom merging with a common table expression using accept to identify changed or new rows, and merge from employees_stage to employees updating and inserting by employee_id.
Identify missing departments in cities by generating all city–department combinations with distinct cities and departments, then left joining to employees to flag nulls.
Learn how to retrieve the second largest salary in SQL using multiple approaches, including subqueries, order by with top, and a dense_rank-based common table expression to handle ties.
Partition data by department and rank salaries to identify the second largest in each department. Filter where the rank equals two to return the second largest salary per department.
Discover how to assign incremental row numbers without disturbing data order using the row_number window function. Use a subquery with a null or constant in over to preserve the order.
Find alternate rows in a table by using the modulo operator to pick odd or even rows, and apply row number with cte or subquery when no numeric column exists.
Apply sql techniques to detect palindrome words using the reverse function and a case expression, flagging each word as yes or no in the palindrome table's names column.
Identify duplicate emails in the contacts table using group by with having count greater than one, and also with a row_number over partition by email to filter duplicates.
Use a case expression to fix gender values in the employees new table, then update the entire table with an update statement and verify results with a select.
Transform student info with unpivot and alternative techniques such as union, CTE, and cross join to align subjects with marks.
Learn to classify defects using a defects table with raised and closed dates, applying case expressions to mark open or closed, and calculating days open with date diff.
Compute daily running totals by summing sales per order date and using a sum over clause in a common table expression to produce day-wise cumulative sales.
Calculate the month-to-date running total (MTD) using a common table expression and over clause, resetting at each month, and compare with QTD and YTD using Adventureworks net sales data.
Learn to compute quarter-to-date totals by partitioning by quarter and year and using sum(total_sales) over to reset totals across calendar quarters in Adventureworks data.
Learn to compute year to date totals in SQL using sum(total_sales) over partition by year order by order_date, with a CTE named YTD and options for daily or monthly views.
Learn to calculate previous year totals and year-on-year growth using a lag window function in SQL, with the Adventureworks data, grouping by year, and using a multi-part CTE.
Learn to compute a three month moving total with a cte and a window frame rows between two preceding and current row on Adventureworks, using sum, average, min, or max.
Learn the fundamentals of data warehousing, including ETL and ELT, data sources, and common tools, with a view to end-to-end data pipelines and BI integration.
Explore data loads in data warehousing, including full load with truncate, upsert with keys, incremental loads using date columns, and slowly changing dimensions types 1–3 in Data Factory and PySpark.
Explore three storage layout models—row store, column store, and hybrid store—and learn how row-wise storage suits OLTP while column store enables fast reads and targeted column extraction.
Robin round distribution allocates rows sequentially to partitions p1 to p4, enabling even data spread and fast loads for staging, but is less efficient for joins due to data shuffling.
Understand hash distribution by applying a hash function and modulo operation to assign rows to partitions based on a key column, boosting performance for large fact tables.
Explore acid properties—atomicity, consistency, isolation, and durability—and learn how transactions either complete fully or roll back, with practical money-transfer examples and delta lake relevance.
Explore data normalization in SQL, from one to third normal form (1NF, 2NF, 3NF) and BCNF, emphasizing reducing redundancy, preserving data integrity, and handling anomalies.
Explore cloud computing basics by learning how internet-based services enable data lakes storage, sql databases, etl processes, and virtual machines for modern data engineering on cloud platforms.
Explore the top cloud providers—AWS, Google Cloud Platform, and Microsoft Azure—and learn how Azure data engineering drives growing demand for Azure engineers and architects.
Explore Azure as a cloud platform, highlighting its no-installation, pay-as-you-go, and fully managed services, with a focus on Data Factory, Databricks, DevOps, and Synapse in a 28-day free trial.
Sign up for a free Azure trial by creating a Microsoft account, verifying via email, and entering payment details to access the Azure portal and $200 credit.
Log in to portal.azure.com, explore the Azure portal, and create a resource group (CDU dev) in East US; learn how resource groups logically containerize resources for dev and production environments.
Create and provision an Azure Data Lake Storage Gen2 account within a resource group, then create containers, upload files, and organize them with directories for scalable data storage.
Learn to provision an Azure SQL server and database in a resource group, configure SQL authentication, choose a compute and storage tier, and enable firewall access for query editors.
Provision Azure Data Factory and design, run, and monitor data pipelines. Learn ETL and ELT basics, set up a v2 data factory, and review billing during the free trial.
Upgrade your Azure free trial to pay-as-you-go in the portal, verify address. Verify pan id for Indian residents and choose the basic plan to avoid costly add-ons.
Explore the basics of ETL and ELT on Azure Data Factory, defining data movement and transformation from diverse sources into a central repository for analytics.
Learn how Azure Data Factory acts as an ETL tool for moving and transforming data through pipelines, activities, and data sets, with 90 plus connectors and secure link services.
Learn how Azure Data Factory activities such as copy data, get metadata, set variable, for each, and lookup power ETL pipelines with serial or parallel execution.
Explore truncate and load, upsert, and incremental load strategies in Azure Data Factory, including when to use each, their trade-offs, and how copy data and pre copy scripts support them.
Learn the fundamental building blocks of Azure Data Factory, including pipelines, activities, data sets, link services, integration runtime, and how they orchestrate data movement, transformation, and control tasks.
Design a data factory pipeline to copy a CSV file from an input to an output folder in a data lake Gen2, using copy data activity, datasets, and link services.
Use the data factory copy data activity to copy an entire folder or specific files from a data lake, applying wildcard patterns and file-type filters.
Copy data from Azure Data Lake to Azure SQL Database and back using Data Factory, with CSV files, autocreate table, and explicit link services and schemas.
Copy data from a data lake to a SQL table, adding a file name column, using wildcard paths and additional columns in Azure Data Factory.
Learn how to create and use set variable activity in Azure Data Factory, including variable types, default values, and serial or parallel execution, with overwriting behavior.
Learn to copy data in Azure Data Factory within a dynamic time frame by filtering files modified in the last hour, 24 hours, or other intervals using expressions and functions.
Explore the get metadata activity in Azure Data Factory, learning to check file existence, list folder contents, and retrieve metadata such as column count, size, and last modified date.
Learn how the for each activity in Azure Data Factory functions as a loop that processes items from arrays or parameters, with sequential or parallel execution and dynamic content.
Use Azure Data Factory to copy each data lake file into a new SQL table using Get metadata and for each, name the table from the file name without extension.
Implement truncate and load in Azure Data Factory's copy activity to avoid duplicates by clearing the destination table before each load using a pre-copy script.
Learn to copy data with upsert in Azure Data Factory, updating existing rows and inserting new ones using a unique key like date. Avoid truncation and enable incremental loads.
Explore how append variable in Azure Data Factory accumulates values in an array type variable, unlike set variable which replaces, with a practical example.
Learn how to use custom SQL queries in Azure Data Factory's copy data activity, including left joins, previewing results, and handling schemas and views.
Explore column mappings in copy data activity to align source and target schemas, import schemas, and map or drop columns when moving data from data lake to Azure SQL database.
Learn how to use the delete activity in Azure Data Factory to remove files across data lake services, configure datasets and logging, and optionally delete recursively after a copy operation.
Learn to use stored procedures with copy activity, import parameters like country, and copy results from Azure SQL Database into Data Lake Gen2.
Execute stored procedures in Azure Data Factory using the stored procedure activity with parameters. Learn why it doesn't return data, compare it to copy activity, and see a truncate-and-load example.
Use the lookup activity to read data from sources such as csv files or database tables and return rows, with options for first row only or all rows.
Learn to use the filter activity in Azure Data Factory to filter an array with expressions and output items greater than three.
Explore the if activity in Azure Data Factory, parameterize expressions, and route to true or false paths. Use dynamic content and equals to debug set variables and SQL-like control flow.
Explore the switch activity in Azure Data Factory, compare it with if and the SQL case expression, use cases to route logic, set variables, handle defaults, and note string inputs.
Learn to use the script activity in Data Factory to run ad hoc SQL statements on a SQL database, performing DDL and DML without returning data.
Learn to use the validation activity in Azure Data Factory to verify dataset existence, configure timeouts and sleep intervals, and understand its impact on pipeline failure versus Get Metadata activity.
Convert a CSV file to JSON in Azure Data Factory using a copy data activity, with a JSON dataset stored in Data Lake Gen2, and understand JSON structure.
Copy json files to SQL database using the copy data activity in Azure Data Factory, covering simple and nested json with mapping to aligned SQL tables.
Learn how the execute pipeline activity calls a child pipeline from a master pipeline, enabling serial or parallel execution and parameter passing, with debugging via run IDs and outputs.
Check file existence with get metadata, run an if condition, and copy data from csv to a database only if the file exists; otherwise fail the pipeline.
Evaluate the difference between parameters and variables in data factory pipelines, noting that parameters stay fixed during execution while variables can be updated, with data type differences and usage guidelines.
Delete blank files in a data factory pipeline by listing files with get metadata, filtering to files, and deleting those under 1 kb.
Learn to copy a headerless CSV to a database using Azure Data Factory, configure a CSV dataset and mapping, handle header absence, adjust ordinal mappings, and troubleshoot type conversion errors.
Explore retry logic for copy data activities in Azure Data Factory pipelines, configure retry attempts and intervals, and understand how initial and subsequent attempts recover from timeouts or missing files.
Explore copy behavior in Azure Data Factory's copy data activity, demonstrating preserve hierarchy, flatten hierarchy, and merge files when moving data between input and output folders in a data lake.
Configure max rows per file in the copy data activity to split large data into multiple files on the sink side when writing to a data lake.
Split data by a criterion in azure data factory using a lookup for countries, then loop with for-each to generate country-specific csv files in data lake named after each country.
Learn to split data by multiple criteria in azure data factory, creating dynamic country and region folders and exporting filtered rows to per-country region csv files.
Consolidate data from multiple csv files into a single table using a data factory pipeline. Dynamically derive country names from file names and use for-each looping with a truncate script.
Consolidate data from multiple folders using Azure Data Factory by listing countries with a master pipeline and regions with a child pipeline, then copy data with country and region columns.
Learn to copy data using custom and dynamic mappings in Azure data factory, starting with a single-table example and expanding to multiple tables later.
Explore copying data with custom mappings across multiple tables in Azure Data Factory, using table mappings, a loop over products, employees, and countries, and dynamic JSON mappings for each table.
Learn to copy data with a pipe delimiter by changing the source dataset column delimiter from comma to pipe, then preview and import schema to fix the error.
Discover how the quote character influences delimited data in a data factory copy activity, and choose double quote, single quote, or no quote to avoid incorrect splits.
Discover how data flows in Azure Data Factory enable data transformations within a pipeline, using sources, data sets, projections, and previews to shape and validate data.
Learn data flow transformations in Azure Data Factory, use the select transformation to choose and rename columns, and preview before publishing to an Azure SQL database sink.
Learn the sort transformation in Data Factory data flows, sorting by country (and sales) with ascending or descending order and default null handling. See previews, debugging, and single-partition behavior.
Master the data flow filter transformation using the expression builder to create boolean conditions, emulating a SQL where clause, preview results, and filter India records.
Learn how to use the derived column transformation to add or update columns, deriving year, month, day, and month name from order dates, and write results to a new table.
Leverage the conditional split transformation in data flows to route rows by expressions into streams like India and France, with dedicated sinks for each output.
Master the cast transformation in Azure Data Factory to change data types after a series of transformations, selecting columns such as date, short, text, and double for accurate casting.
Explore surrogate key transformation in data factory, compare it to identity in T-SQL, and configure start and step values to assign a unique, auto-incremented key to each row.
Learn to use aggregate transformations in Azure Data Factory data flows, combining group by and multiple aggregates (sum, average, min, max), with expression builder for custom calculations and sink output.
Learn pivot transformation in Azure Data Factory to convert rows into columns, producing a product-country two-dimensional summary using group by, pivot key, and pivot columns with sales amounts.
Use the unpivot transformation in a data flow with a sql source to convert columns to rows, preview results, drop nulls, and differentiate unpivot from pivot.
discover how to apply the rank transformation in azure data factory, rank by sales amount, compare rank with dense ranking, and handle partitioning and single-partition considerations.
Apply window transformation to compute running totals of sales by date, using aggregate then window functions with partition by year and month to reset totals per period.
Learn how the union transformation in Data Factory appends rows from multiple sources, using union by name or by position, handling data type alignment, and writing the combined result.
Learn to use a lookup transformation in data flows to add product and customer names to sales using product key and customer key. Explore lookups, left-join behavior, and duplicates handling.
Learn how to perform inner and left outer join transformations in Azure Data Factory, join two or more datasets, manage keys and mappings, and preview and monitor results.
Apply the exist transformation in data factory to check for product key existence across sales and products, returning true or false and distinguishing exists vs doesn't exist.
Flatten json in azure data factory using the flatten transformation to unroll items into multiple rows. Configure json settings, select array of documents, and map to a sql sink.
Learn how to use the parse transformation to split pipe-delimited and JSON data in a data flow, define output columns, and map split results to a sink.
Learn how to use the stringify transformation in a data flow to convert a nested phone field into a string, flattening complex data for a delimited data lake output.
Explore the integration runtime as the compute infrastructure powering data movement in Azure Data Factory and Azure Synapse, with three types: Azure, self-hosted, and Azure Sis.
Learn how to install and register the self-hosted integration runtime for data migration from on-premises to Azure Data Lake or Azure SQL Database, including setup steps and validation.
Copy data from local on premises to Azure Data Lake Gen2 using a self-hosted runtime, configure delimited text dataset, and troubleshoot access with PowerShell to disable local folder path validation.
Learn to copy data from local or on premises to Azure SQL DB, staging in data lake as a landing zone, and push to Azure SQL DB with 1000-row example.
Learn to copy a single table from local SQL database to Azure SQL database using Data Factory, including setting up a local SQL link service and a self-hosted integration runtime.
copy multiple tables from local sql to azure sql using azure data factory, manage dynamic table selection with a mappings table, and audit source and copied row counts.
Discover how to provision Azure Key Vault, store secrets, and grant Data Factory access via access policies; securely retrieve passwords for link services and data transfers.
Create and configure a schedule trigger in Azure Data Factory to run pipelines automatically at designated times, set recurrence and time zone, attach multiple pipelines, then publish and monitor results.
Learn how storage event triggers in Azure Data Factory copy new files as soon as they land in the Data Lake, including parameterized pipelines and Event Grid setup.
Copy files by type in azure data factory using metadata in an excel file to separate csv, json, and parquet into dedicated output folders.
Learn incremental loads from adls to sql db using a staging table and watermarking on the order date, with truncate and load to capture new rows.
Load incremental data from data lake to sql db using data flows in azure data factory, eliminating staging by using max date from a lookup and parameterized filtering.
Load incremental data from Azure Data Lake Storage to sql database for multiple tables using file timestamp, with a mapping table, get metadata, and a max timestamp update workflow.
Learn to perform incremental load without a date or unique key by generating a hash key from concatenated columns using sha-1 in Azure Data Factory data flows.
Build a metadata-driven data integration framework using Azure Data Factory and SQL to process CSV files in Azure Data Lake with auditable file metadata and staging to data warehouse.
Register metadata by setting up a data lake, Azure Data Factory, and Azure SQL Database, and capture file metadata with audit, STG, and DW schemas via a metadata pipeline.
Load dimension tables using scd type 2 with staging and final tables in the dw schema. Implement a metadata-driven adf pipeline for full and incremental loads with hash detection.
Learn Python programming, a high-level language, prized for simplicity and readability, and apply it across web development, artificial intelligence, machine learning, and data engineering.
Install Python and VS Code, download and verify Python on Windows, add Python to PATH, explore Idle and the code window, and set up VS Code for modern Python development.
Learn python basics by setting up VS Code, creating folders, py files, using print to run code; compare py scripts with ipynb notebooks and understand cells with an interpreter.
define variables as names that refer to objects in memory, storing values in Python. learn dynamic typing, the assignment operator, and how to reuse values via multiple variables and swapping.
Learn how to concatenate strings in python using the plus operator, including aligning spaces, variable reuse, and handling type errors by converting numbers to strings.
Explore string interpolation in Python, learning how f-strings embed variables with curly brace placeholders, replacing concatenation and enabling on-the-fly calculations and formatting. This is the industry-standard approach.
Learn how the input function in Python captures user input at runtime, converts to integers when needed, and supports prompts to create interactive, dynamic programs.
Learn how the Python is operator compares memory locations rather than values, using id to reveal object identity and differences between a is b and a == b.
Learn how to use comments in Python to explain code, document important details, and safely comment out code using hash and triple quotes, with tips for readable, collaborative projects.
Explore loops in programming with Python, including for and while loops that execute code repeatedly until a condition is met. Learn how loops reduce repetition and save time.
This lecture introduces Python for loops and range, teaching how to iterate over sequences with start, stop, and step, and apply it to tables, factorials, and string characters.
Discover break and continue in Python for and while loops, using if conditions to terminate or skip iterations and control output.
Master the Python match case statement and its use with an atm-style menu and day-of-week examples, including syntax, case clauses, and handling case sensitivity by lowering inputs.
Develop logical thinking in Python by building and printing star patterns using for loops and nested loops, exploring vertical lines, triangles, and rectangles.
Learn how to write small anonymous Python functions with lambda, compare it to def, and use it with map, filter, reduce, sorting, and closures for on-the-fly calculations.
Trace the evolution of programming from machine language to object oriented programming and learn how objects unite data and behavior through encapsulation, inheritance, polymorphism, and abstraction.
Define a class as a blueprint, instantiate objects from it, and add attributes and methods, using self to reference the current object.
Master the Python init method, a dunder initializer that runs automatically when a new object is created and initializes attributes with self, also known as constructor.
Learn how class attributes are shared by all objects, how they differ from instance attributes, and how defining them outside the init clarifies their behavior.
Explore Python dunder methods, including init, str, repr, eq, hash, call, and delete, using a bank account example to show how objects customize behavior for end users and developers.
Encapsulation bundles data and methods to protect and control access, using public, protected, and private members, getters and setters, and a property decorator for safe, validated access.
Explore encapsulation in Python by refactoring a bank account class with public, protected, and private attributes, implementing getter, setter, and property accessors, plus deposit and withdrawal validation.
Explore inheritance in Python's object-oriented programming by using a parent employee class to share attributes across child classes, and learn how init overriding and super promote code reuse and maintainability.
Learn how the Python super function lets a child class initialize parent attributes from the employee class, while the developer adds language, enabling inheritance and avoiding code duplication.
Explore method overriding in Python inheritance by redefining a child method with the same name, using super() to reuse the parent implementation and avoid repetition.
Explore the five common types of inheritance in Python—from single inheritance to multi-level, hierarchical, multiple, and hybrid forms—with practical notes on their relationships and challenges.
Demonstrates single and multi-level inheritance in Python, showing how a developer inherits from an employee and how multi-level inheritance connects grandparent, parent, and child classes, with method resolution order illustrated.
Explore hierarchical inheritance in Python by building a parent employee class and three child classes—Developer, Manager, and HR—demonstrating constructor initialization, info methods, and MRO concepts.
Delve into multiple inheritance in Python by merging employee and trainer classes into a Techlead, and master method resolution order and cooperative inheritance with super.
Explore hybrid inheritance in Python, understand the diamond problem, and learn how method resolution order and super with keyword arguments prevent duplicate initialization.
Explore polymorphism in Python: use a single interface with many forms, where a common work method behaves differently across developer, manager, HR, and tester objects via dynamic dispatch and MRO.
Master abstraction by using an abstract base class with abstract methods and concrete methods to enforce pay method implementations, exposing only essential payment interfaces while hiding processing details.
This course covers multiple tools and technologies needed to become an azure data engineer.
The best part is there are no pre-requisites!
Anyone can enroll and learn through this course.
Our videos are simple to understand, to the point, short and yet covering everything you need.
The content we are offering in this course is immense and requires full dedication, self -discipline and daily learning to ensure a completion and to make you independent skilled professional.
We have trained 1000's of students and shaped their career and you could be the next one, join us by enrolling in the course and benefit immensely through our course offering.
You will get best of both quality and quantity!
In this course you will learn below tools and technologies from scratch:
SQL : Learn structured query language in Microsoft SQL Server.
Data warehousing : Learn fundamental concepts of data warehousing.
Azure cloud : Learn about cloud computing, benefits and different services.
Azure Data factory: Learn ETL on azure cloud, code free.
Python programming: Learn programming in python with simple way.
Big data fundamentals: Master the big data concepts to build strong foundation.
Databricks: Learn databricks, the leading data platform.
PySpark: Learn big data processing in pyspark on databricks.
Delta lake: Learn the delta lake features.
Spark structured streaming
Azure devops
POWER BI
Fabric
2 End-to-end projects : 1. Data quality project 2. Insurance domain(Rest api) project
GEN AI with langchain and langgraph frameworks to build AI agents and Agentic AI(Under progress)
This course is for everyone from beginner to architect level!
Note: Data bricks and Pyspark content is re-recorded and released with latest interface, updates, in-depth spark working in 2026!