Azure Data Engineer Masterclass - The Complete Guide

Master SQL from basics to advanced through practical exercises, covering databases, tables, indexing, stored procedures, complex queries, and joins; install SQL Server Management Studio and practice.

Learn what a database is, the relational and non-relational types, and how SQL interacts with databases, including a practical SQL Server Management Studio demo to create a database.

Demonstrate creating databases and tables in SQL Server, switching databases with the use command, and deleting databases with drop, plus defining table columns with int and varchar types.

Create an employee table with id, name, address, salary, and bonus, then read it with select star from employee and insert values with insert into.

Explore how SQL constraints enforce data integrity, including not null, unique, primary key, foreign key, check, and default rules on signup table fields such as email and age.

Enforce data integrity with not null and unique constraints by creating an employee table with email and phone, where email cannot be null and phone numbers are unique.

Explore implementing check and default constraints in sql: enforce age greater than 18, and auto default country to India when omitted, via creating and inserting into an employee table.

Explore primary key and unique key concepts, showing how a primary key uniquely identifies each row, cannot be null or duplicated, and that a table may have multiple unique keys.

Learn how foreign keys link tables by referencing the related table's primary key, illustrated with a department and student table example to enforce valid department ids and prevent errors.

Learn to filter data with the where clause and to sort results with order by in SQL, using salary thresholds and ascending or descending orders, including locations that are null.

Understand delete, truncate, and drop in SQL, including when to delete specific rows, when to remove all data while preserving the table, and when to drop the table.

Learn to perform sql updates: set location to india where null, set salary for Sam to one lakh, and update null salaries to one lakh using update, set, and where.

Master conditional statements in SQL by using and, or, in, and not in within where clauses to filter employee data by department, salary, and id.

Explore aggregate functions in SQL, including max, min, sum, average, mean, and count, using an employee table to compute salaries and counts.

Learn how to group rows by a column with the group by clause, count employees per location and department, and compute max, min, and sum salaries within each group.

Learn to apply the SQL like operator to filter description fields in a retail data set, using starts with, ends with, and contains patterns with % wildcards.

Learn how the having clause filters aggregates, unlike where, and apply sum or average with group by to filter addresses where total or average salaries exceed thresholds.

Master top and limit in SQL to fetch first records, top n, percent, or bottom results; sort by salary, filter by location, and apply MySQL limit or SQL Server top.

Discover how to extract unique records with SQL using distinct, remove duplicates from an employee table, and return unique combinations across id, name, and other columns.

Learn to use the coalesce function in sql to replace nulls with defaults, select the first non-null email, and compute salary plus bonus with nulls as zero.

Master SQL joins by learning inner, left, right, and full outer joins that merge two or more tables using a common column such as department ID.

Compare union and union all to combine multiple tables on a row basis, noting that union removes duplicates and requires identical schemas.

Use the alter command in SQL to change a table’s structure by adding a new salary column, altering a column’s type, or dropping a column.

Learn how window functions in SQL perform calculations across a related row set using over and partition by, with examples of sum, min, max, and avg by region.

Explore window functions row_number, rank, and dense_rank to rank rows by salary. Understand how row_number assigns unique sequence, dense_rank handles ties without gaps, and rank creates gaps after duplicates.

Master lead and lag window functions in sql to access previous and next row values by transaction date, with practical examples of previous and next amounts.

Learn to use a common table expression to store temporary results and simplify complex sql queries with window functions like row_number, for example third highest and second lowest salaries.

Explore SQL views as virtual tables created by a query that do not store data, enabling reuse of complex queries, simplified access, and data consistency.

Learn how stored procedures in SQL enable code reuse by creating procedures and calling them with execute. Pass parameters to tailor results, and compare stored procedures with views.

Learn how subqueries in SQL use an inner query to compute results and feed them into outer queries, with examples like finding employees above average salary and Chicago departments.

Explore sql triggers and how they run after delete to back up deleted data into a backup table, and learn before, after, and instead of trigger types.

Practice SQL interview questions for data engineers: remove duplicates with distinct, identify duplicates using row_number over partition by, and fetch top three highest and lowest earners by location.

Learn to solve SQL interview questions using window functions, notably row_number with over partition by department or location to identify second or third highest or lowest salaries.

Learn how inner, left, right, and full joins behave with duplicate records and null values, and master calculating row counts with practical table examples.

Create a table with same schema; copy data; filter names starting with A; department IDs start and end with same letter; output xml; show current date, month, and year.

master common sql interview topics, including union vs union all, primary key vs unique key, rank vs dense rank, delete, drop, and truncate, where vs having, and handling null values.

Explore Azure Data Factory fundamentals from beginner to advanced levels through hands-on projects, practical debugging, and real-time exercises to build pipeline skills and prepare for the data engineer role.

Learn to create an azure data factory in the azure portal, configure subscription, resource group, region, and then launch the data factory studio to explore author, monitor, and manage.

Learn Azure Data Factory components—activities, data sets, link services, and integration runtimes—through a pipeline that copies from blob storage to SQL Server, and study triggers and runtime choices (auto resolve, self-hosted, ssis).

Explore how to create an Azure storage account, choose storage types (blob, file, table, queue), select general purpose v2, set access tiers, and upload files via the portal.

Learn to perform a copy activity in Azure Data Factory by copying csv data from a blob storage input container to an output container using datasets and link services.

Learn to configure a delete activity in Azure Data Factory to remove source files from the input blob storage after copies to the output container, including linked services and logging.

Learn to set up a SQL database in Azure, create a server and resource group, configure authentication, and use the query editor with sample data.

Copy data from Azure blob storage CSV to Azure SQL database in a tabular structure using an Azure Data Factory pipeline, with datasets, mapping, and debugging.

Explore how triggers in Azure Data Factory automate pipeline execution based on schedules or events, including schedule, tumbling window, event, and manual triggers.

Learn to configure schedule triggers in Azure Data Factory to automate pipelines. Set start time, time zone, and frequency to run pipelines automatically on a schedule.

Explore how event based triggers in Azure Data Factory automate pipelines in response to blob storage events, such as csv file uploads, triggering a copy from csv to json.

Explore tumbling window triggers in Azure Data Factory, and learn to backfill historical data by running pipelines every 24 hours from SQL to blob storage as CSV.

Learn how data flow transformations in Azure Data Factory enable etl, using filter and sort to extract from Azure SQL and load into Azure Blob Storage.

Use a join transformation in Azure Data Factory to merge employee and department data on employee id, and store the result in the employee department container.

Learn union transformation in Azure Data Factory by row-wise combining Mumbai and Delhi CSV files, and understand when to use union versus join for matching columns in blob storage.

Explore the select transformation in Azure Data Factory to rename, drop, and reorder columns, demonstrated with a csv file in blob storage and a practical data flow.

Learn how the derived transformation in Azure Data Factory creates new columns and updates existing ones, replaces nulls with unknown, and uppercases addresses.

Apply the aggregate transformation in Azure Data Factory to perform sql-like sum, max, min, and count with group by, computing total salary and unique employee count per address.

Learn how parameterization in Azure Data Factory enables dynamic runtime values to copy multiple SQL tables into blob storage using a single dataset and a table name parameter.

Learn to parameterize a single Azure Data Factory link service to connect to multiple databases by passing dynamic server, database, username, and password values at runtime.

Master end-to-end pipeline parameterization in azure data factory by dynamically routing sql tables to corresponding blob storage containers using a single copy activity.

Explore pivot transformation in Azure Data Factory data flows by turning unique country values into columns, grouping by department, and counting employees to produce a department-country matrix.

Save copied data in Azure Data Factory using a date-formatted file name and JSON format, converting UTC to Indian Standard Time for easy analytics.

Use the get metadata activity in Azure Data Factory to retrieve file and folder metadata, including item name, last modified, and existence, to selectively access department data in blob storage.

Learn to use the for each activity in Azure Data Factory to iterate multiple files with Get metadata, and copy data into JSON in a target storage.

demonstrates the if condition activity in Azure Data Factory, using get metadata and for each to route files containing EMP to the employee container and others to the department container.

Create and set variables in Azure Data Factory using the set variable activity, supporting string, boolean, array, and integer types, and build dynamic file names via concatenation.

Learn how to append values to an array variable in Azure Data Factory by using append variable, then verify results with a set variable and output.

Learn to create and call a stored procedure in Azure SQL database via Azure Data Factory, using the stored procedure activity with parameters.

Learn how the lookup activity in Azure Data Factory reads a table or file, then uses foreach to delete resigned employees via a stored procedure.

Discover how the wait activity in Azure Data Factory pauses a pipeline for a specified duration between copy data activities, such as 30 seconds or 30 minutes, before continuing.

Learn how the switch activity in Azure Data Factory routes a data pipeline using a parameter to copy files into employee or department folders, driven by expressions and multiple cases.

discover how to use the validation activity in Azure Data Factory to validate data set references, pass runtime file name parameters, and conditionally copy data or email on failure.

Learn how the execute pipeline activity in Azure Data Factory and Synapse pipelines invokes another pipeline, with options to wait for completion or run tasks in parallel.

Learn to apply the filter activity in Azure Data Factory to select files from blob storage. Use get metadata to read file names, filter by department.csv, and copy data.

Learn how the until activity in azure data factory mimics a do while loop, repeatedly checking a condition such as file existence until it becomes true, with timeout options.

Learn how to use the web activity in Azure Data Factory to call REST APIs, pass JSON bodies, and manage get and post requests with datasets and link services.

Explore using the fail activity in Azure Data Factory to throw errors based on conditions, with a switch-based pipeline that copies employee files and alerts on unknown data.

Explore using the script activity in Azure Data Factory to perform DML and DDL on Azure SQL, Synapse, SQL Server, Oracle, and Snowflake, with parameters and stored procedures.

Explore axis transformation in Azure Data Factory by applying the exist transformation to filter records based on ID matches between employee and department datasets stored in blob storage.

Explore how to use the conditional split transformation in Azure Data Factory to route rows by location, aligning India, US, and UK data into separate containers via a case-statement-like condition.

Explore pivot transformation in Azure Data Factory data flow by turning unique country values into columns and counting employees per department, using group by and aggregate functions.

Learn how to apply pivot and unpivot transformations in Azure data factory data flows to convert rows to columns and vice versa, using practical examples of country and employee counts.

Use surrogate key transformation in Azure Data Factory to add an incremental ID to each row, starting at one and incrementing by one, for dimension tables in a star schema.

Explore windows transformation in azure data factory, using over partition by and order by to compute rank, dense rank, lead, lag, and cumulative sums for location-based salary analysis.

Explore the alter row transformation in Azure Data Factory, using insert, update, delete, and upsert on row-level data within a data flow, with practical country and name examples.

Apply rank transformation in Azure Data Factory to generate descending salary rankings, using rank or dense rank, by configuring a sort condition and creating a new rank column.

Resolve the Azure SQL database link service v2 not supported error in data flows by switching to a legacy version and creating a new link service and dataset.

Set up email alerts in Azure Data Factory to notify when any pipeline fails, configure alert rules and action groups, and monitor multiple pipelines.

Resolve the import schema failed error by setting import schema to none in an Azure SQL database data set within a data flow, then preview data after selecting the table.

Troubleshoot the data flow schema import failed error in azure data factory by configuring the source data set and link service, then import projection and preview data.

Learn to count files in a blob storage account using ADF’s get metadata and child item, then trigger a copy activity when the count exceeds five.

Databricks is a unified analytics platform built on Apache Spark, offering a collaborative workspace for data engineers, analysts, and scientists to manage and analyze data in a data lake house.

Evaluate how Databricks automates cluster management and scales resources versus Spark alone. Leverage lakehouse integration, Unity Catalog, collaborative notebooks, and built-in automation to streamline governance, security, data processing, and analytics.

Learn how to create a Databricks free edition account, log in, verify your identity, and explore features like unity catalog and compute options.

Explore the Databricks free edition interface, create notebooks in Python, SQL, Scala, or R within the workspace, and use serverless compute, dashboards, Genie AI, and the Unity Catalog.

Explore how the Databricks lakehouse unifies data lake and data warehouse in one open-format, delta-layered platform, enabling fast analytics on structured, semi-structured, and unstructured data with ACID.

Unity catalog in Databricks provides a centralized governance and data management layer for discovery, security, auditing, and organization of data across all workspaces, with catalog, schema, and table namespaces.

Create and configure a unity catalog in databricks, name data dev, set storage and access, assign privileges and tagging, and prepare for development and production catalogs.

Understand managed versus external tables in Databricks, including metadata and data storage and drop behavior; learn to create both table types with delta and external locations.

Discover volumes in Databricks and how they provide centralized governance for unstructured data with managed and external volumes. Improve security and discovery with access controls, tags, and lineage.

Create and manage volumes in Databricks, distinguishing between manage and external volumes, upload and organize files, and apply Unity Catalog governance for access control.

Create and configure a Databricks PySpark environment using Unity Catalog, a development catalog and data database, a serverless volume, and upload orders files for data frame creation and PySpark transformations.

learn to create a data frame in PySpark by reading a csv with spark.read.csv using header true, then use spark.read.format with header and inferSchema and display the data frame.

Learn to create dataframes from csv, json, and parquet files using pyspark, enable multiline json handling, read with spark methods, and display results in the development catalog.

Explore the PySpark select transformation to efficiently narrow a dataframe to a subset of columns, rename fields with alias, and display results using df.select and show.

Learn how to use withColumn and withColumnRenamed in PySpark to add or rename columns in a dataframe, including creating derived columns, a default value, and updating existing columns.

Learn how to filter PySpark data frames like sql where clauses, using and/or conditions and string predicates such as starts with, ends with, and like with practical examples.

Learn how to sort data in PySpark using sort and order by, with ascending and descending options on columns like customer name and order id, including handling null values.

Learn how PySpark joins two data frames using a common key, including inner, left, right, full outer, left semi, and left anti joins, with department and employee examples.

Learn to merge two data frames with the same schema using union and union all in PySpark, and remove duplicates with distinct in a Databricks workflow.

Learn how to handle null values in PySpark dataframes by using fill operations to replace nulls with blank, zero, or unknown, including per-column and multiple-column approaches.

Discover how collect retrieves all elements from a data frame in PySpark, best for small datasets after filtering or grouping, with caution about memory when datasets are large.

Define the schema for a PySpark data frame using struct type and struct field, specifying column names, data types (integer, string), and nullable values.

Learn how to perform pivot and unpivot operations in PySpark, turning rows into columns and vice versa, using group by, pivot, and stack to reshape data.

Discover how to create and register a UDF in PySpark, extend Spark SQL, and use it to categorize ages into minor, adult, or senior within data frames.

Learn to create and use temporary views in PySpark, turning data frames into a virtual table, enabling reusable SQL queries within a session for data cleaning and analysis.

Explore window functions in PySpark, including row_number, rank, dense_rank, lag, and lead, to compute top salaries and handle duplicates with partition by and order by salary descending.

Learn how PySpark partition by and repartition split large datasets by region to boost query performance. See disk and memory partitioning, with parquet outputs and controlled file counts.

Learn to convert dates in PySpark using the date format function, applying df.select on a date column to produce custom string formats with various patterns.

Master PySpark date functions for data analysis, including date_add, date_sub, date_diff, year, month, and current_date, to handle rolling periods and month-end calculations.

Explore how the explode function in PySpark converts array data into multiple rows and supports pos explode. Learn to handle maps and dictionaries and extract key value pairs with explode.

Explore the Databricks file section: create and read files like 1.py and common.py, share variables across notebooks with imports, and use with open to read and write text files.

Leverage lakehouse federation to query external data sources like Snowflake, PostgreSQL, MySQL, S3, and Redshift directly from Databricks without copying to a delta table, enabling cross-system analytics and centralized governance.

Learn to establish a lakehouse federation connection to external data sources like MySQL or Snowflake by creating a connection, naming it, and linking it to a catalog for querying.

Databricks SQL provides a SQL-first workspace for business analysts to write queries and build dashboards, connecting to BI tools like Power BI and Tableau with fast, low-latency results.

Learn to use SQL parameters in Databricks to replace hard-coded filters, create and pass values for gender and continent, and manage multiple parameters with add parameter and identifier techniques.

Explore the Databricks SQL workspace, write and run queries across catalogs, databases, and tables, and use the new SQL editor with advanced search, filter, and formatting features.

Discover how to create and reuse query snippets in the sql editor for the sales customer table, including static and dynamic case statements with pass values and dollar syntax.

Learn how to perform joins with aliases in Databricks SQL, use CTE for intermediate results, aggregate totals with group by, and rank top products using ROW_NUMBER.

Schedule an insert query to run daily, weekly, or monthly by creating a named job with time, cron-like timing, time zone, and optional failure notifications.

Master monitoring queries with query history and query profile to filter by user, compute, and duration, identify bottlenecks, and optimize execution details such as memory, reads, and transformations.

Learn how Databricks SQL uses an inbuilt cache to store query results, so repeated queries fetch from the cache, improving performance and reducing cost.

Explore configuring alerts in Databricks SQL, set email, Slack, or Microsoft Teams notifications, test conditions, schedule triggers, and customize alert templates to monitor thresholds.

learn to build visualizations and dashboards in Databricks, creating bar, line, and pie charts, and publish interactive dashboards with data sources, filters, and text boxes.

Explore Genie in Databricks, an AI feature that writes queries from natural language, letting you analyze multi-table data and generate code and results with a few prompts.

Learn how lakehouse jobs in Databricks automate and schedule notebooks and SQL queries as pipelines, enabling ETL-like workflows with monitoring and scalable execution.

Create a lake house job that uses PySpark to read data from a volume, load it into a delta table, and run SQL on aggregates daily.

Create an if-else task in a lake house pipeline to conditionally run a notebook first, then a sql file, based on date or weekday, with dependencies and true/false branches.

Learn to loop the same notebook multiple times using the for each activity, pass an array input, adjust optional concurrency, and conditionally run true or false branches in a pipeline.

Learn to schedule and trigger Databricks notebooks using schedule, file arrival, or continuous triggers; set frequency, time, time zone, and storage location for automated notebook execution.

Explore compute in Databricks by configuring serverless sql warehouse or a dedicated cluster, selecting worker and driver nodes, and choosing the spark version to execute your jobs.

Set up job notifications to alert via email or Microsoft Teams when a Databricks job succeeds, fails, or warns, enabling immediate action after issues like schema changes or cluster failures.

Monitor your databricks jobs using graphs, timelines, and lists to view runs, start times, durations, and statuses, and quickly debug errors such as delta table metadata changes from concurrent updates.

Apply a medallion architecture in Databricks to consolidate CSV and JSON data into bronze, silver, and gold layers, then clean with PySpark for dashboard of revenue and average order value.

Explore Databricks medallion project setup in the free edition by creating a catalog, database, and volume, then establish bronze, silver, and gold folders and load csv and json data.

Explore Databricks medallion project setup and the silver layer by cleaning and transforming bronze data with PySpark, creating Delta tables, and preparing for the gold layer.

Join silver orders and silver customer tables on customer ID to form a gold layer, derive year and month, and compute aggregates total sales and average order value for dashboards.

Master Azure Synapse Analytics as an integrated platform that unifies data integration, data lake and data warehouse, big data analytics, and machine learning, with spark and Power BI support.

Create an Azure Synapse workspace in the Azure portal, link a data lake storage, and explore Synapse Studio with the serverless sql pool for pipelines and notebooks.

Understand Azure Synapse workspace concepts, including link services, sql pool types (serverless and dedicated), spark pools, integrate options, monitor tools, and integration runtime choices for external resources.

Learn how to query a CSV file in Azure Data Lake Storage using serverless SQL pool in Azure Synapse, paying per query and avoiding infrastructure management.

Query json files with a serverless SQL pool by using open row set, converting json to csv with the field terminator, and extracting columns via JSON value in Synapse.

Explain how a dedicated SQL pool in Azure Synapse Analytics differs from serverless by provisioning compute with data warehouse units and paying for fixed capacity, enabling parallel processing for queries.

Learn to create a dedicated SQL pool in Azure Synapse, set DW units, upload a CSV to data lake, and create an external table to query data.

Explore Apache Spark in Azure Synapse as a general purpose in-memory computation engine that processes data across a cluster in memory for fast analytics and multiple workloads.

Learn to create an Apache Spark pool in Azure Synapse, configure node size and auto scale. Run PySpark code and Spark SQL in Synapse Studio notebooks.

Create and run a data pipeline in Azure Synapse to copy a csv from data lake storage and convert to json in the output container, using notebooks and triggers.

Explore Microsoft Fabric, an all-in-one data and analytics platform that unifies data engineering to business intelligence in a SaaS, integrating Power BI, Azure Data Factory, data lake, Spark, and Synapse.

Learn how to create a Microsoft Fabric account and Fabric capacity using an Azure free account with $200 credit, using a work email, register, and pause when not in use.

Learn Microsoft Fabric terminology: tenant, capacity, workspace, and items. A tenant represents your fabric account; capacity provides power for workloads, while workspaces host items like notebooks, SQL warehouses, and pipelines.

Navigate Microsoft Fabric’s workspace, data catalogs, and admin portal to monitor pipelines and real-time analytics. Explore workloads such as data engineering, data factory, data science, and data warehouse.

Create a Microsoft Fabric workspace as a collaborative container for lakehouse, data pipelines, notebooks, warehouse, and Power BI reports, then configure capacity, domain, and access for teams.

Explore the one lake concept built on Azure Data Lake Storage Gen2, unifying structured, semi-structured, and unstructured data into a single source of truth for fabric data and analytics.

Understand how the lake house blends data lake and data warehouse concepts into a single fabric that stores unstructured and structured data and supports delta tables for sql queries.

Create a fabric lakehouse within a workspace, upload files to the lakehouse, and explore the file and table sections, semantic model, and open notebooks for data analysis.

Learn to use the One Lake file explorer to access and upload files from Windows into a Fabric workspace, including the Bronze Lake House, with install and login steps.

Upload a csv to the lakehouse fabric, create a new table named EMP with header-based columns, and load the data into a parquet delta table.

Discover shortcuts in Microsoft Fabric that reference data without copying. Learn about internal and external shortcuts, how they prevent data duplication and support a single source of truth, enabling collaboration.

Create multiple fabric workspaces and lakehouses, upload and organize files and folders, and create shortcuts to reference files and tables across workspaces for collaboration.

Learn to create and use external shortcuts in Fabric to access data stored in Azure Data Lake Storage or other data lakes without copying.

Discover how caching in fabric shortcuts creates a temporary copy of external data, like S3 or Azure Data Lake, to boost read performance and lower costs.

Explore SQL endpoints in Microsoft Fabric to query a delta table in the lake house using T-SQL, enabling analysis and easy sharing with Power BI.

Explore fabric data factory, a Microsoft fabric data orchestration service for ETL and ELT. Build pipelines with data flow gen2, notebook-based PySpark or SQL, and lakehouse integration with Power BI.

Learn to create a data factory in Microsoft fabric by setting up a workspace, selecting fabric capacity, and naming a data pipeline. Explore copy data, templates, validation, and run history.

Copy data activity in Microsoft Fabric Data Factory ingests data from sources like SQL or CSV and copies it to a lake house as parquet, without writing code.

Copy data activity moves data from an Azure SQL database to a lake house, exporting as parquet or json, via a data factory pipeline that validates and runs the job.

Use the delete activity in fabric data factory to clean up outdated data before daily S3 copies, ensuring only the latest files remain in the branch folder.

Learn how to use the get metadata activity in Microsoft Fabric Data Factory to extract file metadata such as names, schemas, and last modified times, and filter files for processing.

Learn how the filter activity in Fabric Data Factory selects csv files from blob storage by using get metadata, then loads them into a lakehouse.

Discover how to use the lookup activity in Microsoft Fabric to retrieve data from tables, files, or queries, and drive subsequent pipeline actions such as conditional deletes or data transfers.

Explore how parameters in fabric data factory make pipelines dynamic and reusable by passing runtime values, such as a file name, to copy specific csv files.

Explore how to use the for each activity in Microsoft Fabric to iterate over ADLs files, filter CSV files with get metadata, and delete them with a delete activity.

Learn how the if condition activity in fabric evaluates file names with get metadata and foreach. Delete files with emp and copy others to lakehouse.

Use the set variable activity in Microsoft Fabric to store values in a pipeline. Read file counts with get metadata and copy data only when the count exceeds five.

Set up notifications for pipeline failures by linking Office 365 Outlook and Teams, sign in to your organization Outlook account, and enable alerts.

Use the invoke pipeline activity to orchestrate data factory pipelines, triggering a second pipeline after the first completes by configuring workspace connection and optional parameters.

Schedule your pipeline in fabric by turning on the schedule option and choosing hourly, daily, weekly, or monthly runs with a start date and time zone, so it runs automatically.

Learn to monitor your data pipelines in fabric by using run history, the monitor option, and filters to focus on data pipelines, view failures, and analyze duration and error details.

Learn how the wait activity in Microsoft Fabric inserts delays between pipeline steps, pausing for minutes to space data copies and support retries after API failures.

Explore dataflow gen2 in Microsoft Fabric, a low-code, visual etl tool that uses Power Query to connect to diverse sources, transform data, and load into lakehouse or warehouse.

Connect sources with dataflow gen2 in fabric to upload files to a lakehouse and apply low‑code transformations like remove column, remove duplicates, sort, and change data types.

Explore string transformation in fabric dataflow gen2 by applying uppercase, lowercase, capitalize, trim, add prefix or suffix, split by delimiter, replace values, and rename columns in one-click operations.

Explore data transformation in fabric by applying mathematical operations in the transform tab, including sum, min, max, average, count of distinct values, and arithmetic like discount, square root, and power.

Explore the diagram view in data flow gen two to visualize all transformations, from change column and remove duplicates to group by city with sum of sales.

Demonstrates joining two data sources in Microsoft Fabric using merge queries as new, selecting left or inner joins on customer ID, and loading the merged result to the final destination.

Load transformed data into the final lakehouse table with fabric data flow, configure mappings, publish, and run to refresh the destination, or schedule via a data factory pipeline.

Spark is a general-purpose in-memory computation engine enabling data cleaning, Spark SQL, and MLlib for analytics.

Explore spark pools in Microsoft Fabric, comparing starter and custom pools, their setup times, performance, and when to use each for development versus production workloads.

Create your first Spark notebook in fabric, write PySpark and Spark SQL code, manage clusters and lake houses, and use markdown cells and percent percent SQL to run code.

Create a data frame in PySpark from a lakehouse CSV, using spark.read with header and infer schema, then display with df.show.

Master PySpark data frame operations by using select to view specific columns, limit to show rows, and drop to remove unwanted columns, with display DF showing the data.

Learn withColumn and withColumnRenamed in pyspark to cast string quantities to integers, update values, create new columns, and rename fields like zip mode.

Explore filtering data in PySpark with df.filter, selecting records where country equals United State or segment equals consumer, and combining conditions with and/or and customer name starts with C.

Sort data in PySpark data frames using df.sort or df.orderBy, by single or multiple columns (for example, customer_name and sales), and control ascending or descending order.

Master how to group by in PySpark to aggregate by segment, leveraging sum, min, max, and count on a data frame.

Learn how to handle null values in pyspark on fabric by reading csv into dataframe, filtering is null and is not null, and using df.na.fill to replace with values.

Convert a data frame to a temporary view, then use Spark SQL to transform data or run SQL in PySpark mode. Convert back to a data frame; views are session-scoped.

Create visualizations directly in a pyspark notebook by converting a data frame to a temporary view, then write SQL queries to build bar, pie, and line charts.

Learn to save a cleaned data frame into a lake house using delta format, with append or overwrite modes, and to create managed or external delta tables.

Explore notebook utils to list files in lakehouse locations, copy and move data between folders, run a different notebook from the current one, and exit executions.

Explore environment in fabric to customize per notebook with a dedicated custom pool, specific spark versions, and libraries, avoiding workspace-wide configuration changes.

Schedule and monitor notebooks in fabric either directly or via a data factory pipeline, set run frequency, and review spark job details and logs.

Explore data warehouses as centralized repositories for analytics, reporting, and bi. Understand etl, star and snowflake schemas, and fact and dimension tables.

learn to create and manage a data warehouse in Fabric, run SQL queries, create tables, insert data, and visualize results with Power BI and Excel.

learn how to load data into a warehouse with data factory pipelines, data flows, copy into, and create table as select, with schema and credentials for adls delta files.

Apply the copy into command to load data from ADLs into the data warehouse, creating a gold schema and employee table, using a csv url with varchar columns.

Learn to load data into a data warehouse using a CTAS. Create a gold.consumer table from destination lakehouse data filtered by segment equals consumer for analytics.

Learn how views in a DWH fabric act as virtual tables that run a SQL query without storing data, enabling reusable, secure joins across customer, product, sales, and subscription data.

Learn to create and parameterize stored procedures in T-SQL to reuse complex queries, pass runtime parameters with execute, and compare them with views for dynamic results.

Master the new visual query in dwh fabric, transforming data on a canvas with filter, replace null with zero, and save the result as a view.

Explore the download SQL database project feature to export and view table schemas and associated queries, revealing dbo and gold schemas and all created views for the data warehouse.

learn to write T-SQL queries in a notebook to access the data warehouse, add the warehouse, and run T-SQL against tables and views using PySpark in lake house.

Power BI is a Microsoft business intelligence tool that connects to diverse data sources, cleans data, and creates interactive dashboards and reports for stakeholder decisions.

Learn how to download and install Power BI Desktop on your PC, including visiting the official site, selecting the 64-bit installer, completing the setup, and opening the app for practice.

Explore Power BI Desktop overview, connect data from sources such as Excel, SQL Server, and CSV via get data, and navigate report, table, model, and DAX views to build visualizations.

Upload an Excel file in Power BI, load four sheets: customer, order, people, and return; convert order date to date time, and map the order–customer relationship in the model.

Learn to design a Power BI desktop dashboard from page setup to header and title creation, add a logo with a web URL action, and style the background.

Create and format card visuals in a dashboard to display total sales, total profit, and counts of customers and distinct orders using the superstore data.

Create a cluster column chart in Power BI to visualize year over year sales from order dates, refine axes, colors, and data labels for an interactive dashboard.

Create a line chart in Power BI to show monthly sales and profit trends. Format the x-axis and y-axis, enable data labels, adjust colors and styling, and observe seasonality.

Create a Superstore dashboard in Power BI that combines a stacked column chart for sales with a line chart for profit by market, enabling interactive market-level and monthly analysis.

Create a quarter-based sales pie chart in Power BI Desktop to analyze quarterly sales and profit, applying legend, formatting, and color options for clear visualization.

Demonstrates using a slicer in a Power BI dashboard to make all visuals—bar, line, and pie charts—change dynamically when selecting segment or category, with single, multiple, or select-all options.

Create a static visual in a Power BI dashboard by editing interactions to prevent cards from changing with selections, ensuring total sales and total profit stay constant.

Add a sales KPI to a dashboard by creating a target measure, visualizing year-by-year sales against a 5 million target, and interpreting the percentage gap.

Learn how to add a gauss chart to a dashboard to visualize progress toward a 5 million profit target, using a key performance indicator style gauge and total profit figures.

Create a single 'clear all slicer' button to reset all slicers and return the dashboard to its default view, using insert, customizing the button, and verifying with control and enter.

Learn to create bubble and field maps in Power BI to visualize country-level profit and sales, with gradient coloring by profit and rules coloring sales below or above one lakh.

Discover how the Q&A chart in Power BI converts natural-language questions into visuals, generating bar, pie, or line charts from sales data.

Discover how tooltips in Power BI reveal extra information on hover, like sum of sales and sum of profit, and even count of order ID.

Explore how to apply filters in Power BI across page, all pages, and visuals, using basic and advanced filtering, contains logic, and top N to reveal country-based sales insights.

Explore the drill concept in Power BI, using drill up and drill down to navigate a date hierarchy (year, quarter, month, day) while analyzing sum of sales across levels.

Explore data modeling in Power BI by defining data types and managing relationships across tables, including one-to-one, one-to-many, many-to-one, and many-to-many cardinalities.

Learn how DAX, a data analysis expression language, powers Power BI to create calculated columns and measures, enabling total sales, revenue analysis, and data modeling.

Learn to apply the DAX format function to format columns, turning numbers and dates into readable values, with examples for sales, currency, and month names.

Learn to differentiate on-time and delayed orders using DAX date diff and if, deriving a delay status from order date and ship date, and visualize results with a pie chart.

Derive day of week from order date with format; use a dax switch to label each date as weekend (saturday or sunday) or weekdays, and create a new column.

Explore using a dax variable to determine weekend versus weekday sales from order dates, replacing multiple derived columns with a single variable for a cleaner, more efficient calculation.

Learn to use the DAX CALCULATE function with FILTER to derive a sum of sales restricted to a date range. Filter the order date to capture April 1–15, 2019.

Kick off a Python playlist from basics to advanced topics. Practice in Anaconda or VS Code with Jupyter Notebook for data analyst, data engineer, data science, and software engineer roles.

Install Anaconda on Windows, verify Python via the Anaconda prompt, and launch Jupyter Notebook through Anaconda Navigator for hands-on practice.

Explore Python, a high level, interpreted, general purpose language created by Guido van Rossum, used for web development, data analysis, artificial intelligence, and machine learning with extensive libraries.

Understand how Python variables act as containers that hold values and explore data types: string, integer, float, complex, list, tuple, set, and dictionary, plus dynamic typing and static typing.

Explore the Python input function for user input with email and password examples, and learn that inputs are strings by default and require type casting to int or float.

Explore Python operators across categories—arithmetic (addition, subtraction, multiplication, division, modulus, exponentiation), comparison, logical, assignment, and membership—with practical examples.

Master Python conditional logic with if, elif, else, and nested ifs using dynamic user input; determine even or odd, pass or fail, and assign grades A, B, C.

Learn to use for loops to iterate over lists, tuples, dictionaries, sets, and strings in Python, print items, and control flow with break, continue, range, and nested loops.

Learn how to use the python while loop to run statements while a condition is true, compare it with the for loop, and apply break and continue, with practical examples.

Master lists in Python, a built-in, ordered, mutable container that stores multiple items of data types. Create lists, access elements by index (including negative indexing), and modify them with append.

Explore the list method and its inbuilt tools such as append, insert, remove, pop, sort, reverse, extend, index, and count to manipulate lists for coding interviews.

Practice solving Python list problems by finding the maximum and minimum elements using built-in functions and explicit manual iteration.

Learn to swap list elements in Python by interchanging the first and last items using L[0] and L[-1], and by swapping two elements via their indices with assignments.

Explore three Python list-reversal techniques: the inbuilt reverse method, a second built-in approach, and a method without inbuilt tools that uses a for loop and range.

Learn to compute the sum and average of a Python list using both the built-in sum and a manual loop, with example list L.

Print even and odd numbers from a list in Python using a for loop and modulo tests.

Learn to remove duplicates from a list by iterating through elements and appending only when a value is not already in the new list.

Explore list comprehension as a concise way to build a new list from an existing one. See how it squares numbers, filters evens, and uppercases strings, with faster, readable code.

Discover how tuples in Python store ordered, immutable collections with duplicates, distinguishable from lists by immutability, indexing, and faster access.

Explore Python sets, an unordered, unindexed, and unchangeable collection that doesn't allow duplicates. Learn creation, iteration, and operations like add, update, remove, discard, pop, clear, delete, and union.

Explore Python dictionaries, a built-in data type used to store data as key-value pairs, with order changeable and duplicates not allowed.

Master Python strings, covering quotes variants, mutability, and key text manipulation methods like upper, lower, strip, replace, and split, and learn concatenation and f-strings with user input.

Explore python string tricks for data engineering tasks: palindrome checks, reversing strings and words, counting word occurrences, and removing duplicates using loops and join-split methods.

Explore how Python functions enable code reuse by defining reusable blocks, using built-in functions like sum and max, and calling custom functions with parameters.

Explore non-keyword arguments and keyword arguments in Python by creating flexible functions with *args and **kwargs, including summing arbitrary inputs and handling dictionary key-value pairs.

Explore how to write a Python lambda function, an anonymous single-expression tool for simple operations, with multi-argument examples like summing two numbers.

Explore Python's map, filter, and reduce to transform and aggregate data in a single line, compare map and filter behavior, and use reduce with lambda for cumulative operations.

Learn object oriented programming in Python by building classes and objects. Define blueprints with properties and actions, create instances, and organize code for real world applications.

Explore how to define a class, create objects, and implement methods in Python, with the __init__ constructor automatically executing on object creation.

Explore inheritance in Python, showing how a child class reuses properties and methods from a parent class, including multi-level and multiple inheritance and how to access methods.

Learn encapsulation in Python by restricting access to variables and methods; use private attributes with underscores, access them via class methods, and distinguish protected versus private access levels.

Explore the difference between iterable and iterator, showing how iterables store elements in memory while iterators generate values on demand, using iter and next to handle stop iteration.

Learn to organize Python code into modules, create custom modules, and import them to manage production scripts with addition, multiplication, and inbuilt math functionality.

Explore how the Python os module enables file and directory manipulation, including creating and deleting directories, changing the current directory, listing contents, and iterating over files.

Learn to work with date time in Python, including getting current and custom dates, formatting with strftime, converting strings to dates, and adding or subtracting days with timedelta.

Master Python error handling with try and except, log errors, and handle edge cases like division by zero to prevent system crashes in production.

This end-to-end Azure data project ingests API data into lake storage. Implement medallion bronze-silver-gold architecture with data factory and databricks to derive revenue, purchases, store performance, and product category sales.

Build an end-to-end retail azure data engineering solution using data factory, data lake, databricks, and power bi, following bronze, silver, and gold medallion architecture.

Execute an end-to-end azure data engineering project using adls, adf, spark, and synapse to implement medallion architecture (bronze to gold) and generate daily purchases and revenue reports.

Learn to migrate on prem SQL data to Azure Storage using a self-hosted integration runtime, Azure Data Factory pipelines, and blob storage, including setup, execution, and validation.

Discover how to design an end-to-end Azure Data Factory pipeline that ingests JSON from a REST API, uses a staging table, and performs upsert into an order table.

Master git and GitHub for data engineers, covering version control, repositories, and branching for data pipelines. Learn to raise a pull request in production with steps from cloning to deployment.