Terraform for Data Engineering: Hands on End to End Guide

Name: Terraform for Data Engineering: Hands on End to End Guide
Rating: 4.0 (5 reviews)

Use Terraform to Provision Infrastructures on Azure (with Hands-on Labs and Mini project)

Created byGaniyu Yusuf

Last updated 5/2024

English

English [Auto],

What you'll learn

How to setup Terraform on different machine
How to use connect Terraform to Azure
How to use Terraform to automate infrastructure provisioning on Azure
How to create Data Pipelines with Terraform and Azure

Course content

7 sections • 32 lectures • 4h 17m total length

Introduction0:34
Learn to provision Azure resources with Terraform, from resource groups to storage accounts, Key Vault, Databricks, and HDInsight, preparing to deploy 37 resources end to end.
Quick Information on Course Expectations0:32
Set expectations for a practical, hands-on course on provisioning resources on Azure with Terraform. Learn to apply Tensorflow?
Course Structure1:31
Explore the six-section terraform for data engineering course, covering what Terraform is, how it works, installation, Azure resource provisioning, Azure Data Factory automation, and advanced analytics with Synapse and HDInsight.
What is Terraform?0:30
Explore how Terraform enables open source infrastructure as code, letting you define and provision data center infrastructure using the HashiCorp Configuration Language Hql or JSON.
Why Terraform?2:57
Understand how Terraform enables cross-platform compatibility across AWS, GCP, and Azure as infrastructure as code. Automate provisioning, reduce errors, and enable auditable state management for collaboration.
How Does Terraform Works?1:19
Terraform uses cloud provider APIs via providers to create and manage resources; write code, generate a plan, verify, and apply to Datadog, AWS, GCP, Microsoft, and many other providers.
Terraform Key Aspects3:13
Explore how Terraform enables infrastructure as code, versioning, collaboration, and immutable infrastructure, with plan and apply workflows, declarative language, state management, and modular, reusable modules with variables.
Terraform Commands and Workflow4:39
Initialize your Terraform project with init to pull required providers, then plan to preview resource changes. Validate, apply, or destroy resources on Azure, and manage state to track deployments.

Installing IDE on your machine1:03
Install Visual Studio Code on your system by downloading from code.visualstudio.com, choosing Mac OS, Windows, or Linux builds, and double-clicking to install.
Setting up Terraform on your machine (Mac OS)4:00
Install terraform on mac os via brew or the standalone installer, upgrade to the latest version (arm64 on apple silicon), and verify the installation with terraform version.
Setting up Terraform on your machine (Windows OS)0:52
Install and verify Terraform on Windows by downloading the correct binary (32-bit 386 or 64-bit), extracting the zip, and confirming the installation with a version check in the terminal.
Setting up Terraform on your machine (Linux OS)2:24
Install Terraform on Linux by identifying your distro, including Ubuntu, CentOS, Fedora, Amazon Linux or Homebrew variants, copy and paste the installation scripts, then verify with Terraform version.
Setting up Azure CLI on your machine (Windows OS)1:37
Learn how to install Azure CLI on Windows using the MSI installer or winget, with PowerShell options, choosing 32-bit or 64-bit, and adapting the command to the desired version.
Setting up Azure CLI on your machine (Linux OS)2:20
Install Azure CLI on your Linux machine using apt for Ubuntu and Debian, or dnf for Red Hat, and verify the installation with az --version.
Setting up Azure CLI on your machine (Mac OS)1:02
Install Azure CLI on macOS with homebrew, run the provided code in the terminal, and verify it is up to date or reinstall if needed; learn troubleshooting and removal commands.
Linking Azure CLI on your PC to Azure Cloud1:54
Link your Terraform setup and Azure CLI to your cloud infrastructure by typing Azure login, which opens a browser and confirms a successful Azure login.

Provisioning Azure Resource Groups with Terraform9:37
Provision an Azure resource group with Terraform by configuring the Azure ARM provider in main.tf, setting variables for name, location UK South, and tags, and running init, plan, and apply.
Provisioning Azure Storage Accounts and Containers with Terraform12:37
Provision an Azure storage account and containers using Terraform, organize code with modules and variables, and manage resource groups, locations, and access types via init, plan, apply, and destroy workflows.
Provisioning Azure Key Vaults with Terraform18:38
Learn to provision Azure key vaults with Terraform using a key vault module and azurerm_key_vault, with access policies and key, secret, and storage permissions. Validate and apply changes.
Provisioning Azure Databricks with Terraform13:35
Provision an Azure Databricks workspace with Terraform, using a Databricks module, variables for resource group, location, and tags, then run init, plan, apply, and finally destroy to clean up.

Provisioning Azure Compute (Linux Virtual Machines) with Terraform27:19
Terraform guides provisioning an Azure Linux VM via a compute module, building vnet, subnet, nic, and Ubuntu 22.04, with ssh keys, public IP, and security group, using init plan apply.
Provisioning Azure Compute (Windows Virtual Machines) with Terraform10:37
Provision a Windows virtual machine in Azure using Terraform, reusing the existing VNet, subnet, public IP, and security group, with a Windows Server 2022 image and RDP access.
Provisioning IIS Webservers on Windows Virtual Machines with Terraform10:52
Provision a Windows virtual machine with Terraform and enable IIS using a custom script extension to install the web server and expose port 80.

Provisioning data factory module with Terraform on Azure29:45
Provision an Azure data factory with Terraform, linking blob storage to create source and destination datasets, and implement a pipeline to orchestrate data movement.
Working with Azure Data Factory Triggers7:33
Define and attach a five-minute schedule trigger to the Azure Data Factory copy data pipeline using Terraform, then switch to a one-minute cadence, publish, and monitor pipeline runs.
Creating Azure SQL Linked Service14:34
learn how to create an azure data factory linked service to sql database with terraform, extract the sql server connection string, and test the connection to validate the setup.
Creating and Validation Azure Data Factory Pipelines14:11
Learn to create and validate an Azure Data Factory copy data pipeline using a Terraform script, including source and destination data sets, copy activity, and scheduling.

Provisioning Synapse Analytics with Terraform16:59
Provision an Azure Synapse workspace with Terraform, configuring storage data lake Gen2, firewall rules, integration runtime, and a linked service, including Azure AD admin and SQL credentials.
Provisioning Azure HDInsight Spark Cluster with Terraform19:09
Learn to provision an Azure HDInsight Spark cluster with Terraform by building a reusable module, configuring storage and variables, and validating with plan and apply.
Provisioning Azure HDInsight Hadoop Cluster with Terraform11:14
Learn how to provision an Azure HDInsight Hadoop cluster with Terraform by duplicating existing Spark configs, updating variables, validating, and troubleshooting version and VM size issues.
Provisioning Azure HDInsight Kafka Cluster with Terraform9:55

Requirements

No programming experience needed

Description

This is a Terraform-Azure practical hands-on course based on a lot of requests by students.

This course will provision 50+ different resources on Azure.

There is a mini-project for an end-to-end implementation of the application at the end of the course.

This comprehensive course is designed to guide students through the intricacies of using Terraform for cloud infrastructure automation, with a specific focus on Microsoft Azure. From setting up your environment to deploying complex Azure services, this course covers a broad range of topics to equip you with the skills needed to proficiently use Terraform in real-world scenarios.

What You Will Learn:

Introduction to Terraform: Understand what Terraform is, why it's essential, and how it works. Delve into its key aspects, commands, and workflow to get a solid foundation.
Setting Up Your Environment: Step-by-step instructions to install necessary tools like VS Code, Terraform, and Azure CLI on various operating systems (Mac OS, Windows OS, Linux OS). Learn how to link Azure CLI to Azure Cloud for seamless operations.
Azure Resources Provisioning: Gain hands-on experience in provisioning Azure Resource Groups, Storage Accounts, Key Vaults, and Databricks using Terraform.
Working with Azure Compute: Learn to provision both Linux and Windows Virtual Machines in Azure. Understand the specifics of setting up IIS on Azure Windows VMs.
- Work with virtual networks
- Network Security Groups
- Ports and their exposures
- Virtual Machines
Azure Data Factory Pipeline Automation: Master the art of provisioning Azure Data Factory, creating triggers, and setting up data factory modules. Learn to create Azure SQL Linked Services and validate Data Factory Pipelines.
- Create a provision Data Lakes
- Linked Services
- Datasets
- Triggers
- Pipelines
Advanced Topics - Synapse Analytics and HDInsight: Expand your knowledge by provisioning Synapse Analytics and Azure HDInsight clusters (Spark, Hadoop, Kafka) using Terraform.
- Provision Synapse Analytics workspace
- Provision HDInsight Spark Clusters
- Provision HDInsight Hadoop Clusters
- Provision HDInsight Kafka Clusters
Resources and Codes: Access a wealth of resources, code samples to reinforce your learning and help you in practical implementation and mini projects to help you practice.
- Full codes
- Mini Project

Who This Course Is For:

Individuals looking to deepen their understanding of Terraform in the context of Azure.
Cloud professionals aiming to enhance their skills in infrastructure automation and cloud services provisioning.
Students and professionals who prefer a hands-on approach to learning Terraform.

Course Highlights:

Detailed lectures covering every aspect of Terraform and Azure integration.
Practical, real-world examples to enhance learning.
Comprehensive resources and code repository for extended learning.

Who this course is for:

Beginner Data Engineers curious about data engineering
Experienced Data Engineers looking to automate azure pipelines with Terraform

Terraform for Data Engineering: Hands on End to End Guide

What you'll learn

Explore related topics

Course content

Introduction8 lectures • 15min

Installations and Infrastructure Setup8 lectures • 15min

Azure Resources Provisioning4 lectures • 54min

Working with Azure Compute (Virtual Machines)3 lectures • 49min

Azure Data Factory Pipeline Automation4 lectures • 1hr 6min

Provisioning Additional Resources (Synapse Analytics and HDInsight)4 lectures • 57min

Resources and Mini Project1 lecture • 1min

Requirements

Description

Who this course is for: