Azure Data engineering end to end 2026 (Hindi)

Explore CDU content 3.0 as part of the Azure data engineering end to end course in Hindi.

Explore the Databricks and PySpark update in CDU content 3.0, featuring a dataframe-first approach, Unity Catalog integration, shorter 8.5-hour content, and end-notebook interview questions with Spark DAG explanations.

Explore SQL basics and its role as a universal query language across SQL Server, MySQL, PostgreSQL, and Oracle, including T-SQL dialects and step-by-step installation concepts.

Explore the history of SQL and its role in data engineering within the Azure data engineering end-to-end course in Hindi.

Install SQL Server 2025 by downloading from the Microsoft site, selecting an edition, and completing the setup with the database engine and SSMS tools.

Install sql server and ssms, connect to the server via object explorer, and create or drop databases with sql queries in the query window.

Discover SQL Server hierarchy within the Azure data engineering end to end 2026 course, delivered in Hindi.

Explore built-in system databases in SQL Server, including master, model, msdb, and tempdb; learn to monitor, create or drop, copy, and rename these databases for operations.

Create and drop tables in SQL Server using SQL queries and design view. Define columns like employee ID and employee name with data types such as integer and text.

Insert data into a table using insert into with employee id, name, date of joining, and salary. Verify results with select queries on the employees table.

Learn to craft select statements in SQL, fetch data with specific columns or star, apply filters, joins, and aggregations, and prepare data for an ETL pipeline and Power BI consumption.

Learn how to install the Adventure Works sample database for Microsoft, perform backup and restore with SSMS or T-SQL, and set up an OLTP data warehouse workflow.

Discover how to retrieve unique values with SQL using distinct, by selecting a column like color from the dim product table in Adventureworks, and learn to remove duplicates in Excel.

Learn how to sort data using sql with order by, including single and multi-column sorting, ascending and descending orders, and sorting by color, product name, and list price.

Discover how to comment and uncomment SQL queries using double-dash line comments and block comments, run queries in the AdventureWorks database, and view results from the dim product.

Learn how to filter data with a where clause and filter operation using comparison and logical operators, combining color and price criteria to produce true or false results.

Explore pattern-based filtering using wildcards with the like operator to query data, including begins with, ends with, contains, and exact matches in SQL where clauses.

Explore aggregate functions in SQL, including sum, average, min, max, and count, and learn how to rename results with aliases and handle null values.

Learn to use the group by clause with aggregate functions to compute total sales per product or customer, selecting from fact internet sales and grouping by product key.

Learn how to copy a table with select into, transferring data to a new or existing table, and using where conditions to filter results like red products. Verify via queries.

Learn how to create a table with a primary key, enforce unique and non-null constraints, and prevent duplicate employee IDs while inserting and querying data in a data warehousing context.

Create tables with the not null constraint, enforce primary keys, and decide which columns allow nulls versus default values during table design.

Learn how a unique constraint enforces row-level uniqueness in a table, how it relates to the primary key, and how duplicates or null values are handled, using an email example.

this lecture shows creating a table with a check constraint to validate salary values, using create table and alter table scripts, and testing inserts for constraint violations.

Configure a default constraint to automatically supply a value when inserting rows, as shown with the city default London and an override when you specify NYC.

Learn how to create a table with an auto increment identity column for the primary key, ensuring not null constraints and automatic number generation for the employee id.

Explore updating rows in a table using the update statement, set clauses, and where conditions, verify results, and safeguard data with backup tables and select into techniques.

Learn how to delete rows from a table, compare delete, truncate, and drop operations, use where clauses, and understand backup strategies and identity implications.

Explore string functions in SQL, including upper and lower case conversion, substring, left and right extraction, split, length, and trim, with examples on transforming column data and hard-coded values.

Explore string functions in the query window, focusing on the replace function, position-based replacement, and operations such as extract, reverse, insert, count, and occurrences.

Explore concat, concat_ws, isnull, and space separators to build full names, then use select into to backup, alter table to add full name, and update it.

Extract first, middle, and last names from a full name using left, right, index, and length functions with nested patterns and dynamic positions.

Explore date and time concepts in Azure data engineering with current date and date time functions, time zone handling, pattern formatting, validation, and precision control.

Discover how to extract year, month, and day, format dates, compute differences, and work with week days and date parts using datetime functions.

Master date and time inputs, intervals, and date part operations. Compute differences, end-of-month dates, and current dates with date from parts and related functions.

Explore integer data types such as tinyint, smallint, int, and bigint in SQL tables, and learn to define, cast, and convert them within storage and range bounds.

Explore numeric data types, focusing on approximate numeric types and exact types such as decimal, float, and money, and learn how precision, scale, and storage range affect tables.

Explore date and time data types, including date, time, and date ranges, along with storage considerations and examples like midnight and date range usage.

Explore string data types such as char, varchar, and nvarchar, and compare fixed versus variable length, Unicode encoding, max length, and truncation impacts for multi-byte characters.

Learn how the unique identifier acts as a new id to guarantee a primary key, generate system ids, and manage insertion in an employees table with 16-byte identifiers.

Explore the bit data type in SQL databases, using 1/0 to represent true or false, and practice insert and update operations to set active or inactive flags.

Explore data type conversion in sql server, covering implicit and explicit conversions, cast and convert functions, and common failures during numeric and string concatenations.

Learn how to join sales and product tables using inner and left joins, including equi join, common product ID keys, and handling nulls for non-matching rows.

Join dim product with fact tables using inner, left, and anti joins to analyze sales by product key and name.

Join the main table, dim product, and dim customer tables using product key and customer key to output product name and customer name with total sales by year.

Use expressions to fix key mismatches in joins by casting strings to numbers and extracting numeric portions, enabling inner joins between dim products and sales data.

Explore outer joins in SQL, comparing inner, left, right, and full outer joins, with practical examples joining products and sales tables and handling non-matching rows.

Learn how to perform self-joins using inner and outer joins on a single employees table to map managers, parents, and salaries.

Explore data warehousing basics by distinguishing fact and dimension tables, mastering entity and master tables, and visualizing star and snowflake schemas with key relationships in AdventureWorks.

Explore the IIF function in T-SQL, compare operators, and nested if and case expressions to build conditional logic for data queries and output.

Explore the case statement expression for conditional execution, including when-then-else syntax and end keywords. Use color and price examples, exact matches, ranges, and comparison operators in data engineering workflows.

Explore iif and case expressions to implement multiple conditions using when and then, range and exact value comparisons, and and/or filters to produce true or false results.

Master azure data engineering: custom sorting in sql using order by case expression to arrange departments such as sales, finance, production, and engineering, with ascending or descending sequences.

Explore union and union all operations in SQL to combine rows from multiple tables, handling duplicates and using select into for output tables.

Learn intersect and union on tables, identify common rows, and apply inner joins with matching rules across multiple columns, ensuring the same column count and data types.

Explore SQL set operators such as union, union all, intersect, and except, and learn join types like inner join, left join, and right join with practical examples.

Learn how foreign key constraints and primary key relationships enforce referential integrity between product master tables, dimension tables, and the sales transaction fact table, preventing orphan records and inconsistent states.

Explore subqueries in sql by using aggregate functions like max, min, average, and sum within inner queries to derive values for outer queries and where clauses.

Learn how to use subqueries with IN and NOT IN to filter a fact table by a dimension table, matching product keys via color and color code.

Use subqueries to update employee base rates by country with joins and where clauses, and maintain a backup table across employee and sales territory dimensions, including Canada.

Learn how subqueries create derived tables and how inner and outer queries work together to compute average and total transactions by customer key in the internet sales data.

Illustrate exists and correlated subqueries, using in and not in operators to filter transactions from the fact internet sales by product key from dim product, showing true/false results.

Calculate year-wise total sales and contribution percentage using master total and grand total, with subqueries and derived tables. Sort by year and format results as a percentage.

Explore how the having clause filters aggregated totals after group by, using total sales and monthly thresholds, and compare its behavior with where and execution order.

Learn how to use the top clause to select top records from a product table—top 100 and top 10 percent—while sorting with order by and leveraging derived table or subquery.

Identify top ten and bottom ten products by total sales using aggregate sums, group by, and order by, mapping product keys to names via the product dimension.

Explore how the row_number window function assigns a unique row number within an over clause, partition by, and order by, using an employee table with city, department, and salary.

Explore window functions like rank and dense_rank, compare them with row_number, and learn how over, order by, and partition by rank employees by salary (descending) across departments or cities.

Learn how window functions lag and lead work with over, partition by city, and order by date to apply offset and handle nulls.

Explore first_value and last_value window functions in Azure data engineering, using partition by city and order by date to extract first or last rows in data orders.

Classify sql into categories: ddl, dml, tcl, and dcl. Explain how ddl creates, alters, and drops definitions, how dml inserts, updates, and deletes, and how tcl handles transactions and permissions.

Explore alter table operations to add, drop, and modify columns, change data types, and manage constraints, while understanding backups, data loss risks, and permissions in SQL DDL.

Learn how to modify tables with alter table constraints, add and drop constraints, define not null and default values, and implement primary and foreign keys in SQL.

Learn how the SQL offset keyword skips rows and returns a data subset. See practical examples, such as top ten queries and fetch first ten rows, with order by requirements.

The lecture explains the coalesce function, returning the first non-null value among inputs and using it to fill null vendor frequency data (yearly, half yearly, quarterly, monthly) for consistent reporting.

Explore how to use the merge statement to synchronize a target employees table with a source table, covering insert, update, and delete scenarios.

Explore database schemas, including the default dbo schema, and how a table schema defines structure, keys, and data types; learn how access, permissions, and role-based boundaries govern tables and departments.

Learn to use group by with rollup to generate year-level subtotals and a grand total in a sales table. Apply isnull and formatting to display year and grand total clearly.

Learn to use group by with rollup and cube to generate subtotals and grand totals across year and month, and compare rollup versus cube outputs.

Master pivot operations to transform rows into columns and create a two-dimensional salary summary by city and department using group by and aggregation, including dynamic pivoting.

Learn how unpivot operations act as the pivot's opposite to transform columns into rows, such as salary and department data into base data including city and sales.

Explore views as virtual tables that reference underlying data in physical tables and learn to create, query, and secure them for end users and tools Excel, Power BI, and Tableau.

Explore the concept of common table expressions (CTEs) as temporary named result sets that substitute subqueries, simplify complex queries, and improve readability and performance in SQL.

Master common table expressions in this multi-part cte series, join sales to product data by product key, and calculate total sales through group by operations.

Master common table expressions and recursion in SQL by using anchor and recursive members with union all, learn to control recursion limits to avoid infinite loops.

Learn variables as basic containers in programming, mastering declare, set, and fetch operations across languages, with data types like integer and concepts such as default values and null.

Explore SQL if-else logic, including the if function, case expression, and control flow statements, with practical examples of declaring variables, conditional selects, and begin-end blocks.

Explore the while loop concept across languages and t-sql, showing how a condition drives iteration and how the loop terminates when the condition becomes false.

Learn how temp tables and temporary objects work, including local and global temp tables, their tempdb scope, and using them for temporary storage and data transformations.

Discover stored procedures and their parameterized, dynamic business logic, and compare them to views. Learn to create, execute, alter, and drop procedures with parameters and security layers for data pipelines.

Convert a table to JSON by applying JSON path and array structures, handling employee id keys and null values, and producing output JSON suitable for SQL Server and MySQL.

Create nested JSON output to build the Adventureworks database, query with SQL joins and JSON path, and map fields like product name, list price, and sales amount.

Explore the output clause in sql, understanding how insert, delete, update, and merge operations affect the products table, output rows, and table-valued variables.

Explore slowly changing dimensions in data warehousing, covering type 1, 2, and 3, merging staging and target tables to preserve history with start date end date and is active.

Learn how indexes improve data retrieval in SQL, including clustered and non-clustered indexes, execution plans, and how proper indexing reduces table scans and speeds queries.

Explore clustered and non-clustered indexes, understand how they shape execution plans, and learn when to use include columns, filtering, and sorting to accelerate joins on large tables.

Explore the theory of clustered and non-clustered indexes, their impact on data access, and how pages, extents, and B-tree structures organize data for faster retrieval.

Explore how clustered and non-clustered indexes optimize SQL queries, demonstrating index seek versus index scan in an execution plan and how filters and key lookups boost performance.

Learn how delete and update cascading works with foreign keys in a master and child schema, including on delete cascade and on update cascade to automatically affect child rows.

Learn string_agg and string_split to convert multiple rows into a single string and split a value into rows, with practical customer and product examples.

Discover how to create user-defined functions (UDF) in SQL Server, including scalar functions, inline TVFs, and multi-statement TVFs, with syntax, returns, parameters, and testing in dbo.

Explore cross-apply and outer-apply in SQL Server, contrasting them with inner and left joins, to join a table with a table-valued function using row-by-row parameter parsing.

Explore the codename function, which wraps inputs as delimited identifiers using square brackets to create valid SQL identifiers, supporting Unicode and dynamic SQL while explaining its limits against SQL injection.

Discover how the built-in system store procedure sp_executesql executes dynamic t-sql with parameter definitions, enabling safe, parameterized queries and reducing sql injection risk.

Explore 100 videos in SQL as part of Azure data engineering end to end 2026 (Hindi).

SQL injection occurs when untrusted input alters a SQL statement, risking data leakage and unauthorized modification; prevent it with parameterized dynamic SQL using SP__EXECUTE SQL, separating query and data.

Master dynamic pivot creation by converting product values into columns, using distinct and string aggregate to build dynamic SQL with sp_executesql, and handle metrics and row totals against SQL injection.

Learn how to calculate averages in SQL scenarios, handling null and zero values, implicit casting, and simple data type conversions in insert and select cases.

Explore joins in sql by comparing inner, left, right, full outer, and cross joins using sample tables a and b, handling nulls and cartesian products.

Identify and delete duplicate rows in SQL using row_number and partition by, with subqueries and CTE options, to keep data clean during large data loads.

Learn custom merging with merge statements in Databricks SQL to upsert records from a source into a target table, updating on match and inserting on not matched.

Discover missing departments per city by comparing city and department pairs with multi-part CTEs, cross joins, and left joins to reveal nulls where data is absent.

Learn how to find the second largest salary in the employees table using a subquery and a common table expression, including dense_rank and ordering by salary descending.

Calculate the second largest salary in each department using the employees table by partitioning by department, ordering salaries descending, and applying dense_rank in a cte to filter rank = 2.

Learn how the row_number window function assigns sequential numbers using an over clause and order by, illustrated with a country list and practical select examples.

Identify alternate rows in a table by using the modulo operator and row_number over, filtering for even or odd rows in the employee data.

Explore SQL techniques to detect palindromes by reversing a name column and comparing it to the original using a case expression, yielding yes or no in practical scenarios.

Identify duplicate emails in a contacts table using SQL techniques like group by, row_number, derived table, and common table expression.

Learn to apply a case expression to correct gender values in an employees table by constructing and running an update statement.

Transform student information with unpivot and pivot techniques to map maths and science marks, using derived tables, case expressions, and CTEs for a complete student-subject-marks view.

Learn how defects are classified as open or closed using defect_id, product_id, and dates, and use SQL case expressions and a common table expression to compute open days.

Learn to compute a daily running total from internet sales data, creating a cumulative sales figure using group by, window functions, and an over clause ordered by order date.

Explore azure data engineering by calculating month-to-date, quarter-to-date, and year-to-date totals using window functions on the Adventureworks fact internet sales data, focusing on order date and total sales.

Master quarter-to-date totals with sql window functions, partition by date, and date part logic for calendar and fiscal quarters in sales data.

Compute year-to-date sales totals from order dates using CTEs and partition by year in SQL Data Studio for Azure data engineering end to end.

Learn to compute year-on-year growth in multi-year data using SQL, including previous year sales, current year sales, lag functions, CTEs, and formatting percent growth.

Compute three month moving totals and moving averages from Adventureworks internet sales data using sql window functions and ctes, grouping by order year and order month to produce running totals.

Apply an alternate row arrangement using row_number and partition by to create a custom country order like India and US, exploring order by active and employee examples.

Explore the fundamentals of data warehousing, including ETL and ELT, data sources and formats, and the role of Azure Data Factory and SSIS in building an enterprise warehouse.

Explore data loads in a data warehousing pipeline using data factory and PySpark, including truncate-and-load and upsert strategies, incremental loads, and slowly changing dimensions types 2 and 3.

Explore storage layout models, including rowstore, columnstore, and hybrid stores, with vertical and horizontal partitioning, OLTP optimization, and compression insights for SQL databases and Parquet formats.

Learn robin round distribution as a partitioning technique that assigns rows to partitions for fast data loading in Azure Synapse Analytics with Spark; explore robin, round robin, and hash partitioning.

Learn how hash distribution uses a hash function on a partition key to assign rows to partitions p0 through p3, using modulo operations to balance data and improve query performance.

Master ACID properties—atomicity, consistency, isolation, and durability—through a banking transaction example that shows rollback and commit ensuring data integrity.

Explore normalization in SQL data warehousing, learn first to third normal forms and Boyce-Codd normal form, reduce redundancy, maintain referential integrity with keys, and prevent anomalies.

Explore the basics of cloud computing with Azure, covering cloud services, data storage with data lakes and SQL databases, and tools like ETL and Azure Data Factory.

Explore cloud computing service providers, comparing Azure, AWS, and Google Cloud Platform, and see how end-to-end data lakes, ETL, and reporting fit into devops with Azure Data Studio.

Explore azure to understand the microsoft cloud platform, the sign-up process, trial period, and pay-as-you-go billing, with services like data lake and ETL.

Visit azure.microsoft.com and click start free to create an azure free trial account. Complete the sign-up with your details, verify via email, and review terms and payment options.

Start with Azure by creating a free trial and signing into the portal, then use resource groups to organize regions like East US and provision storage, Data Factory, and databases.

Create and manage an Azure Data Lake Storage Gen2 account, organize raw data in containers, and configure resource groups, subscriptions, and regions for data lake provisioning.

Provision an Azure SQL Server and database, configure resource group and pricing tier, set SQL authentication with username and password, and adjust firewall rules to allow client IPs.

Explore creating and configuring an Azure Data Factory to provision data services, connect to a data lake, manage resource groups, ETL and ELT pipelines, and monitor costs.

Upgrade your Azure free trial to pay-as-you-go using the upgrade option and verified payment method. Explore basic cost management, including cost alerts and pay-as-you-go data factory pipelines.

Master ETL and ELT in Azure Data Factory by moving data from diverse sources into a central repository and transforming it before loading.

Learn to use Azure Data Factory to move and transform data with pipelines and activities, reusing data sets and linked services to connect 90 plus connectors.

Explore Azure Data Factory activities, including copy data, get metadata, set variable, for each, lookup, and execute pipeline, to configure datasets and scalable ETL pipelines.

Learn truncate and load, upsert, and incremental load strategies in Azure Data Factory for processing large datasets; compare full loads with upserts and explain when to apply delta-based updates.

Explore Azure data factory basics with pipelines, activities, datasets, linked services, and integration runtime for ETL/ELT data movement and transformation.

Set up a copy data activity in Azure Data Factory to move data from input folder to output folder in data lake gen2, using pipelines, datasets, and link services.

Learn to copy an entire folder or specific files in Azure Data Factory, using wildcards to filter CSV and Excel files between data lake folders with copy activity and datasets.

Learn to copy data from Azure Data Lake to Azure SQL Database and back using a Data Factory pipeline and copy data activity, including CSV files and table mappings.

Copy data with additional columns from data lake CSV files to an Azure SQL Database using Data Factory copy activity, with wildcard paths, year-wise folders, and auto-created tables.

Learn to use set variable activity in Azure Data Factory pipelines, creating variables, setting default values, and applying dynamic content for serial or parallel execution.

Copy files within a timeframe using Azure Data Factory pipelines, applying last modified filters and dynamic expressions (UTC time, start/end time) to CSV or JSON data sources.

Learn to use the get metadata activity in Azure Data Factory to obtain file and folder metadata—existence, size, last modified, and column details for data sets in end-to-end pipelines.

Leverage the for each activity to loop over collection in a data factory pipeline, parameterize inputs, set variables, and run a task for each item with sequential or parallel execution.

Copy each file to a new table with a dynamic Azure Data Factory pipeline, to your data lake, using get metadata, for each loop, and data set parameters.

Explore truncate and load in data warehousing by copying data, clearing the destination, and preserving or recreating tables, with practical steps for auto create, script copy, and permissions.

Explore copy data with upsert in Azure Data Factory, performing truncate-and-load, update or insert logic using a primary key, enabling incremental loads and handling slowly changing dimensions.

Learn how to use append variable and set variable to accumulate values, including arrays and various types (string, boolean, integer), with MySQL append and insert operations.

Use SQL queries in copy data activity to move and transform data from source databases to Azure SQL or RDS, apply joins and aggregates, and preview results.

Learn how to configure column mappings in a data factory copy data activity, mapping csv source columns to an autocreated destination table, with validation, errors, and manual or import schemas.

Learn how to configure delete activity in Azure Data Factory to remove files or datasets across Azure Blob, data lake gen2, and Amazon S3, with logging and csv log generation.

Learn how stored procedures drive copy data scenarios in Azure Data Factory, using SQL databases, data lake, and parameters like country to perform aggregations such as product wise sales.

Learn how Azure Data Factory's stored procedure activity executes SQL procedures, handles parameters and outputs, and coordinates with copy activity and data lake for practical data integration.

Learn how the lookup activity reads data from a dataset, supports CSV and SQL sources, and outputs data for downstream use, with typical use cases and limitations.

Apply the filter activity to input arrays in the azure data engineering course, using conditions and operators such as equals and greater than to produce a filtered output.

Learn how to implement conditional logic with the if activity in data pipelines, using expressions, sql-like logic, and true/false branches, with set variable, print, and debug outputs.

Learn how to implement the switch activity in a data pipeline, compare it to if statements, and use expressions, cases, defaults, and dynamic content to route results.

Use script activity in Azure Data Factory to run ad hoc SQL scripts, copy data, and perform DML or DDL statements, including lookup metadata and syntax checks.

Explore how the validation activity in a data pipeline checks file or folder existence, retrieves metadata, handles timeouts, and interprets statuses such as succeeded, failed, or timed out.

Learn how to convert csv to JSON using data factory pipelines and Spark, handling headers, key-value pairs, and nested JSON structures.

Learn to copy nested json data to sql db using a data factory pipeline, mapping json fields to employees table columns, and validating with copy activity and debugging.

Learn to execute pipeline activities by invoking pipelines, generating run ids, and coordinating serial or parallel child pipelines with parameters, variables, and triggers.

Demonstrates executing a data copy pipeline that checks file existence with get metadata, uses conditional logic to copy csv or sql datasets, and handles success or failure scenarios.

Explore the fundamental differences between parameters and variables in Azure data engineering, clarifying how key value changes affect execution time, pipelines, and monitoring, with examples of data types for each.

Learn how to delete blank files in a data factory pipeline by using get metadata, filter by file size, and execute delete file activities.

Learn to copy headerless and header CSV data into a data lake using copy activity, configure schemas, map columns, and troubleshoot type conversion and duplicate column errors.

Implement retry logic in data pipelines to handle failed or delayed csv file to database operations, configure retry intervals, and monitor pipeline status.

Master copy behavior in a data lake Gen2 pipeline, copying data between input and output folders, preserving or flattening hierarchy, and merging files across excel, csv, and binary inputs.

Split data into multiple files using a max rows per file setting, automatically distributing output to an output folder and naming prefixes for CSV or text files.

Explore splitting an orders table by country into per-country output files using distinct country data, split data, and copy data activities with dynamic queries and metadata.

Split data by multiple criteria in an Azure Data Factory pipeline, creating country and region folders, using dynamic queries and lookups to export per-country data from a data lake.

Master consolidating data from multiple csv files into a single sql table using copy activities, dynamic metadata, and data lake gen2 workflows.

Consolidate data from multiple country and region folders using nested loops in Azure Data Factory, dynamically retrieving metadata and executing pipelines to export to an Azure SQL invoice data table.

Explore copy data with custom mappings for a single table, including manual and dynamic column mappings, and configure a data factory pipeline to load into an azure sql database.

Explore copy data with custom mappings across multiple tables (countries, employees, products) using column mappings, stored procedures, and data lake paths in an end-to-end Azure data engineering scenario.

Learn to copy delimited data with delimiters (comma, pipe, semicolon) using Azure Data Factory copy data flow, from csv to Azure Data Lake Gen2 and Azure SQL, with debugging tips.

Copy data from a delimited text dataset to data lake gen2 using csv with comma delimiters and configurable quote characters, including header selection and data preview.

Explore how data flows move data from source to destination through transformations and etl, and visualize these steps with data flow activities and pipelines.

Explore data flow transformations with the select transformation in Azure Data Factory, project and rename columns, select fields, apply SQL-like expressions, and debug the pipeline.

Explore the sort transformation to order data by country and sales amount using ascending or descending options, with preview and partitioning in an Azure data engineering pipeline.

Explore the filter transformation in a data flow, using the expression builder to craft SQL-like where clauses with comparison operators, preview results, and refresh conditions for India.

Derive and update columns in a data flow using the derived column transformation, with the expression builder to create year and month parts and computed fields, then preview and save.

Explore configuring a conditional split transformation in Azure Data Factory dataflows to route records by country using the expression builder. Preview and validate outputs across multiple inputs and streams.

Learn cast transformation for end-to-end data flows, changing data types, projecting results, and casting columns to string or double, with previews and status checks.

Learn surrogate key transformation in data pipelines and warehousing, using SQL Server auto increment, primary key and foreign key concepts, seeds, and T-SQL to uniquely identify records.

Learn to configure aggregate transformations in dataflows, using group by, sum, min, max, count, and average with expression builder, preview results, and country or category wise aggregations.

Explore pivot transformation setups in t-sql, transforming rows into columns to create a two-dimensional sales view by product name and country, with group by, pivot key, and sum aggregations.

Learn how to use unpivot transformations to convert pivoted data back into rows, shaping schema and handling nulls in SQL data flows.

Explore rank transformation in a data flow pipeline, applying rank and dense rank window functions to order sales amount and other fields, with partitioning options.

Master window transformation in data pipelines by using over clauses, partition by, and sort by to compute running totals and monthly cumulative sales with aggregates such as sum and rank.

Learn how the union transformation combines multiple data sets from sources into a single data set, with union by name or by position, handling headers and data types.

Learn how to implement a lookup transformation to join sales and product datasets using a common key. Enrich streams with product and customer information through mapping and left join concepts.

Discover how to use join transformation in a data flow to combine sales and products data with inner, left, right, and full outer joins, using lookup conditions on common keys.

Explore the exists transformation in data flows, using left joins and conditional logic to detect true or false existence across products and sales data.

Use the flatten transformation on orders.json to unroll the items array into a flat table with order_id and order_date, enabling fail-fast validation.

Explore the parse transformation in a data flow pipeline, converting Excel datasets with order details and batch details into structured JSON or delimited formats, with splitting and flattening of arrays.

Explore the stringify transformation to convert nested fields into a flat json, preview the data flow, and map fields for writing to data lake gen2.

Discover how the integration runtime powers Azure Data Factory and Synapse pipelines as the compute infrastructure for data movement and transformation.

Install and configure the self-hosted integration runtime for Azure Data Factory, including on-premises deployment, gateway setup, and connecting to cloud data factory for data migration.

Copy orders.csv from the local downloads to ADLS using an on-premises integration runtime in azure data factory; configure local file system as source, delimited text CSV, and test the connection.

Copy data from local on-premises sources to Azure SQL Database using data lake landing and staging areas, migrating datasets such as MySQL into Azure with simple example workflows and monitoring.

Copy data from a local SQL database to Azure SQL database using Data Factory, setting up link services and self-hosted integration runtime, with source and destination validation.

Copy data from a local SQL database to Azure SQL Database using table mappings, dynamic content, and a data migration pipeline with validation and status checks.

Configure key vault secrets for data factory pipelines, securely store and retrieve SQL and CSV passwords, set access policies, and test connections to SQL server and RDS databases.

Explore how to create and manage schedule triggers in Azure Data Factory pipelines, including trigger types, recurrence options, start/end dates, time zones, and publishing, with debugging and monitoring guidance.

Trigger storage events in Azure Data Factory to automatically start pipelines when new CSV files arrive in a container, configure blob created events, and copy data to a SQL sink.

Copy files by type across csv, json, and parquet extensions using a dynamic switch-based pipeline in data factory that reads metadata, sets variables, and routes outputs to folders.

Execute an end-to-end incremental load from data lakes to a SQL database using staging tables, current timestamps, and max-date comparisons to update the final orders table.

Learn how to implement an incremental load from ADLS to SQL database using Dataflow, with dynamic parameterization, a lookup-based filter on the orders table, and a sink to RDS SQL.

Learn to implement incremental loads from ADLS to SQL DB for multiple tables using a watermark and last modified timestamp, with Data Factory and Spark steps.

Explore incremental load from ADLS to SQL database using hash functions to generate keys, perform upserts, and manage data flows for CSV sales data.

Explore a brief introduction to Databricks, its Apache Spark roots, and its lakehouse platform for data engineering, machine learning, and collaboration tools on Azure, AWS, or GCP.

Sign up for the Databricks free edition to access the Community Edition, explore compute options, practice PySpark and SQL, and use a cost-free, serverless data warehouse on Azure Databricks.

Explore the Databricks platform overview, including APN community edition sign up, managing compute clusters, notebooks, and workspaces, and running Python, Scala, R, or SQL in a cloud-based environment.