Azure Data Engineer Real World Projects

Build a data pipeline that processes images with Azure Cognitive Service, extracts text via computer vision, and stores results in Cosmos DB using Data Lake Storage and an Azure Function.

Log into the Azure portal, create a resource group in Australia East, deploy a data lake storage with hierarchical namespace, and create an input container for uploading files.

Deploy a function app in the eShop portal by selecting a subscription and resource group, choosing Python 3.9 on Linux with a consumption plan, configuring storage account and application insights.

Deploy Azure computer vision in real world data engineer projects, create a cognitive service, select subscription, resource group, region, and retrieve keys and endpoint for Python code, using free tier.

Deploy Azure Cosmos DB in a resource group, select the Cosmos DB API with serverless capacity, and use Data Explorer to view containers and databases.

Install Python 3.9 and add to the path, install Azure Function Core Tools, set up Visual Studio Code, install Python and Azure Functions extensions, and sign into Azure.

Learn to create a Python function app in Visual Studio Code, configure a blob storage trigger with Cosmos DB outbound binding, set local settings, and deploy locally.

Execute function from Visual Studio Code to trigger a computer vision workflow, upload images to a storage container, and write results to Cosmos DB, with real-time logs and created entries.

learn how to deploy the function code to azure functions and function app, configure subscription key and endpoint, set storage and cosmos settings, redeploy, and monitor execution.

Walk through Azure function code, detailing triggers and bindings, including storage input and Cosmos output. Demonstrates calling the computer vision API, parsing tags and captions, and writing results to Cosmos.

Upload an image to a blob container, watermark it with python-powered Azure functions, and save the transformed file to a separate output container in the same storage account.

Deploy Azure resources by creating a resource group, storage account with input and output containers, and a Python function app on a consumption plan in Australia East.

Develop and validate an Azure function in Visual Studio Code using a blob storage trigger, Python, and input/output bindings. Run and debug locally using a storage account.

Deploy your validated code to an Azure Function App, configure storage and connection settings, restart the app, and verify the watermark output by uploading test files to the input container.

Upload input files to an Azure Data Lake container. Transform them with Databricks Spark notebooks, write results to Azure SQL, and orchestrate with Azure Data Factory.

Deploy a storage account in a resource group with Azure Data Lake Storage Gen 2, enable hierarchical namespace, set local redundancy in a chosen region, and create a demo container.

Deploy an Azure key vault in the project zero four resource group with a unique name in region, enabling standard pricing and soft delete, and manage secrets with access policies.

Deploy Azure Data Factory to orchestrate and schedule jobs, configuring resource group, subscription, and location, selecting version two, then review and create the resource for Data Factory Studio workflows.

Deploy Azure SQL database and server, select serverless configuration to minimize costs, and configure firewall whitelisting to allow connections for managing and querying the database with SQL Server Management Studio.

Deploy Azure Databricks by creating a Databricks workspace in a resource group in Australia East with premium pricing for 14 days, then launch and create notebooks, clusters, and repos.

Learn how to create a spark cluster in Azure Databricks by configuring compute, cluster name, standard mode, runtime version, and auto scaling settings, then deploy and monitor status.

Create a Databricks notebook, mount a storage container, and read a csv to build a dataframe with spark while exploring credential scopes for security.

Create a scope in Azure Databricks and link it to the key vault to securely store and retrieve the storage key using dbutils.secrets.get, avoiding hard-coded credentials.

Import and execute a notebook in Databricks, mount a blob, read a CSV into a dataframe, compute yearly max audience score, and write results to an Azure SQL database.

Create linked services in Azure Data Factory to connect to Key Vault and Databricks. Store the Databricks token securely in Key Vault, test connection, and prepare for pipeline orchestration.

Learn to execute a Databricks notebook through Azure Data Factory by creating Databricks link service, building a pipeline with a notebook activity, and wiring a storage-triggered file name parameter flow.

Execute a Databricks notebook using a job cluster by creating a dedicated link service, triggering a pipeline, and terminating the cluster after job completion to save costs.

Explore building a unified data flow with blob storage, Synapse Analytics, Spark notebooks, and Cosmos DB. Orchestrate end-to-end processing with Synapse pipeline triggered by file uploads.

Log into the Azure portal, create a resource group, then deploy a storage account, Synapse Analytics workspace, and Cosmos DB in the same region with standard redundancy and provisioned throughput.

Explore the Azure Synapse Analytics Workspace Studio to create notebooks, SQL scripts, data flow, pipelines, and Spark jobs, manage linked resources and pools, and monitor activity.

Upload a file to adls gen2 storage, create a container, and access the data from Synapse Analytics using notebooks and SQL to build and view a dataframe.

Create a linked service for blob storage and connect to the demo container to access data; load it into a spark dataframe and create a spark table.

Create a linked service for Cosmos DB in Synapse, set up a database, test connectivity, and write data frames into a Cosmos DB container using notebook code.

Develop and run a notebook to read data from blob storage, transform with PySpark DataFrame operations, and write results to a lake database and Cosmos DB.

Develop and execute a Synapse notebook through a pipeline, using a blob trigger to pass the file name parameter, publish changes, and monitor Spark jobs that update Cosmos DB data.