Crack Azure Data Engineering by Interview Preparation Course

Explore Azure storage fundamentals, blob storage concepts, and data storage services, highlighting durability, security, scalability, and the hierarchical file system of gen two.

Learn Azure Files, a cloud file share accessible via SMB or NFS across platforms with shared access and Azure queue storage for millions of messages up to 64 kb.

Explain Azure queue storage does not guarantee ordering; Azure Service Bus can provide strict ordering, while Azure Table Storage is a NoSQL, schema-less database that enables fast retrieval of data.

Explore level two Azure storage interview questions to understand why storage account names must be unique and how endpoints and urls like blob.core.windows.net are formed.

Azure blob storage offers block, append, and page blobs; block blobs use blocks up to 190 TB, and page blobs up to 8 TB for VHDs.

Explore factors affecting storage account costs in Azure, including region, account type, access tier, redundancy, transactions, and data egress. Learn how to optimize by choosing appropriate options based on requirements.

Explore authorization options for storage accounts—account access keys, SAS tokens, and Azure AD roles—server side encryption at rest with Microsoft managed keys or customer managed keys via Azure Key Vault.

Enable Azure Resource Manager logging to protect data in Azure Blob storage, enable soft delete for 7–90 days, enable Blob storage versioning, and take regular snapshots to recover deleted data.

Enable soft delete to keep deleted containers recoverable for a default seven-day retention, and restore them from the Azure portal.

Explore the A-Z Copy Tool, a command line utility to copy data to and from a storage account, including installation, authorization, and using AC copy commands.

Compare Azure blob storage and Azure Data Lake Storage (ADLS), highlighting hierarchical namespace in ADLS versus flat file structure in blob storage, and explain why ADLS suits big data analytics.

Explore why storage accounts use two access keys, key one and key two, to enable safe key rotation, regeneration, and seamless app updates.

Explore level three Azure storage scenario questions and identify Azure Blob Storage as the best fit for storing user images and videos, offering scalability, fast access, durability, and encryption.

Apply a time-bound SAS token to the blob storage to grant seven-day access to a third party, avoiding account keys and risky manual revocation.

Discover how to secure data in an Azure storage account with encryption at rest by default, using a customer managed key and strict access controls.

Assign role-based access permissions to a storage account via the Azure portal, using reader or contributor roles authenticated by Azure Active Directory.

Explore how storage accounts handle concurrent access to a single blob, noting that up to 500 requests per second are supported, and consider using a CDN for higher, static-data access.

Keep the storage account in the same region as the application to reduce latency and improve performance; if latency is not critical, explore cheaper regions for cost optimization.

Learn how to automatically move data between hot, cool, and archival storage using Azure lifecycle rules, reducing costs as data access frequency declines.

Identify and track storage costs using the azure cost management and billing portal by selecting your subscription, filtering by storage service in cost analysis, and viewing the costing and forecast.

Learn to generate a comprehensive inventory report of blob metadata in an Azure storage account by configuring blob inventory rules, selecting fields, and scheduling daily or weekly reports.

Enable the static website feature on an Azure storage account to host a cheat sheet in static html in a blob container, accessible via internet url.

Practice hands-on Azure storage interview questions by creating a storage account in Asia Pacific with zone-redundant storage, enable hierarchical namespace, tag operations, and review before creation.

Create two Azure storage containers named raw and curated, with raw set to private and curated set to public, enabling anonymous access via the curated container URL.

Learn to create a file share account inside a storage account in Azure, name it with your name underscore, and map it to a local drive using the snmp protocol.

Learn to create and deploy a static web page on Azure storage by enabling the static website feature, uploading ABC.html to the dollar web container, and using the endpoint url.

Explore Azure storage account access control by assigning a junior read access and a manager contributor access through the portal's role assignments, demonstrating reader and contributor permissions.

Change the storage account soft delete retention from seven to ninety days in the Azure portal, ensuring deleted blobs and containers can be restored within ninety days.

Install and configure the free Azure Storage Explorer, connect to your storage account via Azure Active Directory, storage keys, or SaaS token, and browse subscriptions and containers.

learn how to move a storage account from one resource group to another in the Azure portal, including selecting source and target groups, validating the move, and finalizing the transfer.

Explore generating a shared access signature with read access to Azure Blob objects for seven days, configuring permissions and scope in the Azure portal.

Change the redundancy level of an Azure storage account after creation via the portal. Open configurations, select application level, and save to update redundancy; you may also adjust account keys.

Deepak Goyal motivates you to keep moving after finishing the first section, toward your dream job. He invites you to review the course and ask questions in the Q&A.

Explore how Azure Data Factory orchestrates data migration and ETL, including copy activities moving data from on-prem to the cloud, data flow transformations, and pipelines as sequences of activities.

Discover data sources in Azure Data Factory, with connectors and data types, and learn how link service establishes authenticated connections to move data between SQL DB and blob storage.

Explore data factory basics: define data sets with linked services, understand integration runtime as the compute heart, explore mapping data flows, triggers, and copy activity for data movement.

Explore the difference between triggers and debug in Azure Data Factory, learning how triggers schedule pipelines while debug runs manually with a debug point to test up to a point.

Explain how variables act as placeholders within an Azure Data Factory pipeline, are scoped to each pipeline, and can be set at runtime with the set variable activity.

Learn how pipeline parameters in Azure Data Factory allow runtime value input and property externalization, enabling dynamic execution with different values per run.

Compare variables with pipeline parameters in a data engineering pipeline: variables can be changed during execution via set variable, while pipeline parameters remain fixed after the pipeline starts.

Define global parameters at the Azure Data Factory account level to share common properties across all pipelines, usable in dynamic content but not modifiable inside pipelines; changes propagate automatically.

Explore four levels of parameterization in Azure Data Factory: linked service, dataset, pipeline, and global account parameters, with practical steps to create and apply each parameter.

Configure Azure Data Factory user properties at the activity level to enrich monitoring and debugging of pipelines, using up to five properties per copy activity for source and destination details.

Add annotations to pipelines, datasets, and link services to tag and filter resources by team or project, enabling targeted views in Azure Data Factory.

Azure Data Factory user properties attach to activities and may be dynamic for execution metrics. Annotations are static tags at pipeline, dataset, linked service, or trigger levels for grouping.

Explore the integration runtime types in Data Factory—Azure integration runtime (auto resolve), self hosted, and Azure exercise integration runtime—and their use cases for on-prem data transfer and lift-and-shift migrations.

Not mandatory to create your own integration runtime; Azure data factory provides default integration runtime for cloud data movement, while self-hosted runtime enables on-premises to cloud migrations and linked services.

Learn how to call a child pipeline from a parent pipeline in azure data factory using the execute pipeline activity, with options to wait for completion or run in parallel.

Explore pulling on-premises data with a copy activity inside the forage in Azure Data Factory, and set the forage to sequential to send one DB request at a time.

Learn to migrate on-prem data to Azure with Azure Data Factory by creating a self-hosted integration runtime, configuring linked services and datasets, and executing a successful copy pipeline.

Configure a dedicated azure integration runtime in the US East region to ensure data copied by azure data factory stays within the region and does not move outside it.

Learn how to handle nested for each loops in Azure Data Factory by using two pipelines and an execute pipeline activity, a practical workaround for interview scenarios.

Integrate your data factory with Git, manage dev and prod branches, raise pull requests, and automate deployments to each environment using Azure DevOps pipelines and AMP templates.

Identify pipeline failures in production by using the monitor tab to view execution history, read error messages, and verify input, output, and path validity, then drill down to fix.

Create an incremental pipeline with a high watermark in Azure Data Factory to move on prem data to Azure SQL, using a lookup to identify execution and copy incremental data.

Design a data factory solution that copies data from blob storage to an Azure SQL database, using a storage event trigger or schedule to auto-run the pipeline.

Design an Azure Data Factory pipeline with a copy activity from source to destination, and trigger an email on failure using a logic app via a web activity.

Design a daily data copy pipeline in Azure Data Factory that builds a dynamic year/month/date folder path using format datetime and UTC now.

Design a daily data pipeline that copies data from a rest api to a destination using a dynamic rest endpoint with a date-based relative url.

Design an Azure Data Factory pipeline using get metadata to list files and for each copy data to table via a parameterized dataset that derives table name from file name.

Link an external self-hosted integration runtime from another data factory to reuse existing infrastructure and reduce costs, using the resource ID in Azure Data Factory Studio.

Enable logging and fault tolerance in the copy activity to skip incompatible rows when loading a flat file into a table, and log failed rows to a separate file.

Increase the data integration unit (DIO) in Azure Data Factory copy activity to speed up data transfer, and enable staging to temporarily store data for faster processing.

Explore when to use copy activity mapping in Azure Data Factory, and compare implicit mapping with explicit mapping for mismatched source and destination schemas, such as price versus cost.

Learn how to rerun an incremental data pipeline from the failed activity instead of restarting from the beginning, using the monitor tab and the 'return from a failed activity' option.

Explore why Spark outruns Hadoop with in-memory processing, offering interactive and batch modes, and built-in machine learning capabilities for a holistic data engineering workflow.

Compare Apache Spark, Databricks, and PySpark to clarify differences, including open‑source Spark, Databricks runtime improvements, and the Python library required to run Spark code.

Differentiate transformation and action in spark by noting that transformations produce new dataframes and are lazily evaluated until an action triggers execution, while actions yield non-dataframe results like writing.

Learn how lazy transformations defer execution until an action triggers evaluation, with Spark building a DAG (direct acyclic graph) and creating an optimized plan to reduce queries.

Narrow transformations run within partitions and can execute independently in parallel, while wide transformations require data shuffling across partitions, increasing data transfer and execution time.

This lecture explains differences between RDDs and dataframes in Spark, showing how data is partitioned across a cluster and how dataframes provide a table-like, optimized layer on top of RDDs.

Explore how partitions split data into chunks stored on worker nodes in a Spark cluster, with partition counts guided by cluster configuration and application needs, using hash or range methods.

Understand shuffling as the redistribution of data across partitions during joins and aggregations, causing data transfer, higher bandwidth use, longer runtimes, and increased cost.

Learn how to read files in spark and databricks using spark.read and spark.read.format, accommodating text, csv, json formats, handling header options, and the load function for multi-line inputs.

Infer schema scans the input file to determine data types for each column; enable inferSchema at read time to enhance typing, or supply the schema to save processing time.

Understand how Spark broadcast variables cache read-only data on worker nodes, enabling faster joins and reduced data transfer across the cluster.

Learn to define a Spark schema when reading data, using programmatic or declarative approaches and avoiding infer schema for data sets, with struct type and types like integer and string.

Explore Spark SQL, a Spark module for structured data processing that provides data frames and a distributed SQL engine to transform data using SQL, making Spark accessible to SQL developers.

Create temporary views from a dataframe using create or replace temp view, then query with spark sql. Note these views are session-scoped and dropped when the session ends.

Learn how to create global temporary views using create or replace global temp view, stored in the global temp schema, and accessed across sessions with spark sql.

Discover how Databricks magic commands let you convert a notebook cell to a specific language, run sql queries, execute shell commands, create markdown cells, and run one notebook from another.

Call one notebook from another using the %run command to share initialization, cleanup tasks, or helper functions across notebooks, as shown with a hello example.

Learn how to create a Databricks cluster using the UI, selecting cluster type (single node or high concurrency standard), runtime version, auto scaling, inactivity termination, and worker and driver configurations.

Explore methods to list files in dbfs: using dbutils.fs.ls, the %fs ls magic command, and the data explorer, plus enabling the dbfs file browser in admin settings.

Databricks offers cluster types: interactive clusters for development and job clusters for automated notebook execution. Interactive clusters require manual start and stop, while job clusters run automatically and cost less.

Databricks cluster modes include standard, high concurrency, and single node; standard is default for a single user, while high concurrency supports multiple users with latency and security through separate processes.

Explore Delta Lake as a storage layer for data lakes, enabling consistency, isolation and durability, time travel, and merge updates and deletes on S3, blob storage, or DFS via Spark.

Understand how data lake and delta lake differ: a data lake is a concept, while delta lake is an open source framework adding transactional capabilities and time travel on top.

Databricks architecture divides into a control plane in the Databricks cloud account and a data plane in the customer cloud account, enabling single sign-on and external data source connections.

Spark includes a hive metastore that stores table metadata. Configure an external metastore such as MySQL or Oracle, and Databricks also uses a hive metastore to persist metadata.

Automate the execution of a Databricks notebook by scheduling a job in the Databricks workflow, or via an Azure Data Factory pipeline using a notebook activity.

Showcase best practices for reusing another team's code in an Azure notebook, either by importing a same-workspace notebook or packaging external code as a jar or Python package.

Decide the cluster size by choosing between interactive and job clusters, then pick high concurrency, single node, or standard options, based on data, joins, and aggregations to balance memory use.

Discover how Git integration in Databricks lets teams work on the same notebook without overwriting each other by using branches, commits, and pull requests to merge into master.

Explains calling a Databricks notebook from Azure Data Factory by using a notebook activity, a Databricks link service with an access token, and a trigger for scheduled runs.

Move data from Databricks to destinations such as Snowflake, MongoDB, or Cosmos DB using either staging via a list location with Data Factory or Snowpipe, or direct Databricks connectors.

Learn when to repartition spark dataframes to boost parallelism by increasing partitions, address data skew, and weigh the cost of repartitioning against performance gains.

Spark sql tables persist data in storage and store the schema in metadata, so after cluster termination you can access them with a new cluster using the same storage.