Microsoft Azure Data Lake Storage Service (Gen1 & Gen2)

Create a free Azure subscription to access resources with 200 credit for 30 days, 25 always-free services, and blob storage 5 gigabytes per month plus a 250 gigabyte database.

Discover the Azure portal overview, learn to create, manage, and monitor resources via the web interface, customize dashboards, and leverage the global search, marketplace, and cloud shell.

Explore Azure's broad service categories—from storage and databases to analytics, compute, machine learning, IoT, and security—and learn how pricing, management, and hybrid solutions shape cloud decisions.

Explore how to organize and manage Azure resources using management groups, subscriptions, and resource groups, with policy enforcement, cost control, budgeting, and access control through identity and access management.

Resource groups are containers that hold resources for your solution, with metadata stored in a region, and can include resources from other locations, but deleting a group deletes its resources.

Learn to use tags as metadata to describe and organize Azure resources, resource groups, and subscriptions, enabling tag-based search, cost center categorization, and improved billing visibility.

Discover how to delete resources and resource groups to prevent unexpected charges, and configure cost management to set budgets and email alerts with a dashboard and forecasts.

Data grows at an astonishing rate, outpacing traditional databases' capacity. We need a hyperscale repository that handles volume, variety, and velocity, storing data in native formats for big data analytics.

A data lake is a large repository for vast volumes and varieties of structured and unstructured data in native format, loading data immediately and enabling later transformations.

Explore how data lake architecture relates to Hadoop, recognizing Hadoop as one component, not the sole data platform. See complementary options like Apache Kafka for real-time data.

Learn how data lake Gen2 blends blob storage with HDFS features to support big data analysis in the cloud, delivering cost-efficient, tiered storage for unstructured data.

Compare blob storage and data lake Gen1/Gen2: built on blob storage, optimized for big data analytics, with Hadoop integration and hierarchical namespace.

Provision an Azure data lake Gen2 account by creating a storage account in the portal and enabling the hierarchical namespace to transform it into data lake storage.

Explore the Azure Data Lake Gen2 storage account overview, covering resource group, location, tags, storage types, metrics, activity log, access control, data transfer, and lifecycle management.

Enable hierarchical namespace in azure data lake storage gen2 creates a true directory tree, enabling fast folder operations and seamless Hadoop integration for improved analytics.

Discover Azure data lake storage Gen2 features, including ABFS driver integration with Hadoop ecosystems, scalable petabyte-scale storage, low-cost blob storage, and fine-grained security for data access.

Discover how to ingest data from multiple sources into Azure Data Lake Storage Gen1 and Gen2 using tools like Data Factory, PowerShell, AzCopy, and Storage Explorer.

Demonstrate ingesting data into a Gentoo storage account via portal and storage explorer, creating containers and directories, and uploading diverse files with hot, cool, and archive tier options.

Learn to copy data from on premises to a data lake account using the easy copy utility, configure account name and key, and run recursive copy commands.

Demo shows copying data from Azure Blob storage to data lake Gen2 using Data Factory, including creating a data factory, configuring a copy pipeline, and verifying transfer to Gentoo storage.

Move structured data from a SQL Server database to Azure Data Lake Storage Gen2 using Data Factory, creating connections, transforming to parquet files, and validating through monitoring.

Move data from Amazon S3 to Azure Data Lake Storage Gen2 using Azure Data Factory, demonstrating cross-cloud data transfer and pipeline orchestration.

Explore data flow around a data lake: ingest unstructured data, then explore, analyze, clean, transform, and load into destinations, then visualize with Power BI or export to data warehouses.

See how data lake design with separate storage and compute enables cost savings and on-demand analytics via transient clusters, keeping data safe while you power down unused resources.

Connect a data lake storage with a database without moving data, mount via a service principal, and process data in Scala, Python, or SQL, then save back.

Provision a Databricks workspace and a premium database service with role-based access control, then create and attach an interactive cluster to a notebook and to a workbook.

Mount the data lake to Databricks DBFS by creating a service principal, granting read, write, and execute access, and configuring the mount with the app client and directory IDs.

Explore taxi data in a data lake, analyze with spark to derive statistics, clean and transform records, then load the processed data back to the data link in parquet format.

Demonstrate a batch data pipeline in a data lake by uploading a file, ingesting with Hive, transforming to an aggregated form, and exporting to SQL Server.

Create a data lake storage gen2 source and a SQL Server destination, configure storage and firewall, and set up a delays table for origin city and delay time.

Explore how managed identities enable secure authentication and authorization between Azure services without credentials, comparing system assigned and user assigned identities for secure storage and compute integrations.

Open the Adjure portal, create a managed identity for the inside cluster, and assign it to the storage and sql accounts as storage blob data contributor and sql contributor.

Deploy an HDInsight interactive query cluster. Configure memory-optimized settings, storage options gen1/gen2, and secure credentials, with two heads for high availability and three zookeeper nodes.

Explore Apache Ambari, a Hadoop management platform for cluster administration, monitoring, and configuration, and its graphical user interface that centralizes node management, service status, alerts, and metrics.

Demonstrates ingesting a CSFI CSV dataset from the US Bureau of Transportation Statistics into data lake storage, performing extract, transform, and load with Hive queries to create a new table.

Learn how to use Hive interactive query to ingest comma-separated value airline delay data into external tables, run MapReduce-backed queries, and preview results before transforming data in Hive.

Transform unstructured dfs data into a tabular view with a hive-like query, overwrite an output directory, and extract cities with weather delays and their average delays.

Export data from Hadoop to SQL Server using Sqoop, authenticate with SSL keys and password, adjust firewall to allow IP, and verify 330 rows in the delays table.

Explore how to integrate an inside Hadoop cluster with data storage using SSL and server authentication, via command line, graphical interfaces, or both.

Explore Azure Data Lake Storage with account keys, shared access signatures, storage access policies, dual Active Directory authentication, RBAC and ACL, encryption in transit and at rest, advanced threat protection.

Learn how storage access keys secure Azure data lake storage, including two keys, connection strings, and rotating keys by shifting apps before regeneration; old methods are not recommended for production.

Learn to grant least privilege using shared access signatures (SAS) to specify permissions and a time range, with IP restrictions, instead of sharing storage account keys.

Leverage Azure Active Directory to grant role-based access to storage accounts across subscriptions, resource groups, or containers, replacing storage account keys and shared signatures with identity-based authentication.

Learn how access control lists manage read, write, and execute permissions for users, groups, and service principals in Azure Data Lake Storage, with inheritance and group-based best practices.

Explore how to lock down Azure storage using firewalls and virtual networks, restricting access to selected IP ranges and virtual networks, with exceptions for trusted Microsoft services and logging access.

Enable encryption in transit to protect moving data by enforcing secure transport, using site-to-site or point-to-site VPNs, and applying blindsight encryption for double encryption within apps and storage.

Demonstrate encryption at rest as a built-in, mandatory feature for all storage accounts, using 256-bit Microsoft managed keys by default, with optional customer-managed keys via key vault.

Enable advanced threat protection to detect unusual access to storage, send alerts with ip address and remediation guidance, and enable it easily via the portal (30-day free trial).

Understand how the adjure activity log records storage account management events, such as rbac changes and key rotations, while excluding data plane activity, and how to export and analyze these logs.

Explore and filter activity logs for Azure data lake storage resources to see operations, status, time, and initiator, then export logs to log analytics via diagnostic settings for analysis.

Explore how metrics track storage capacity, transaction performance, and end-to-end latency in Azure Data Lake Storage, with time-series data, alerts, and near real-time monitoring.

Learn how to monitor Azure Data Lake Storage resources with the unified monitor service, viewing capacity and transaction metrics, latency, and end-to-end performance for storage accounts.

Explore the new insights monitoring tool for Azure Data Lake Storage Gen1 and Gen2, visualizing availability, latency, transactions, and capacity across APIs with customizable workbooks and dashboards.

Explore creating alerts in an Azure storage account, define metric and signal-based conditions, and use action groups to email or automate responses when capacity or security events exceed thresholds.

Explore diagnostic settings and log analytics to collect logs from services, analyze with the custom query language, visualize data, and configure alerts with application insights, storage analytics, and on-premises sources.

Explore how diagnostic settings monitor a storage account with metrics and logs, configure per-service logging for blob, file, queue, and table, and manage retention or move logs to cheaper storage.

Optimize data lake throughput by accelerating ingestion with fast networks, using file sizes from 256 megabytes to 100 gigabytes, and date-based naming to minimize latency and transactions.

Delete resources in the Azure Data Lake Storage course, including resource groups, storage accounts, input data, Data Factory, and SQL Server, to avoid costs and note that deletion is irreversible.