Teach on Udemy

Turn what you know into an opportunity and reach millions around the world.

Learn More

Your cart is empty.

Keep shopping

Azure Cloud Azure Databricks Apache Spark Machine learning

Name: Azure Cloud Azure Databricks Apache Spark Machine learning
Rating: 2.9 (140 reviews)

Big Data, Spark SQL, Hadoop, Kafka, Data Lake, Transfer Learning, Zeppelin Notebook, Graph, Hortonworks HDP, Cloudbreak

Created byBigdata Engineer

Last updated 5/2020

English

What you'll learn

This course will provide you an in depth knowledge of apache Spark and how to work with spark using Azure Databricks
You will understand the main concepts of Azure Cloud
You'll learn about the basic use of Azure Cloud
You will learn to deploy Apache Spark ecosystems locally
You'll learn how to develop a databricks dependency library through IDEA
You will be able to process continual streams of data with Spark streaming
You will learn how to train a machine learning model
You will understand the use case of graph analysis
Deploy Azure Virtual Networks via the Portal
Deploy Azure Resource Groups
You'll learn how to use Azure Data Factory
You'll learn the historical story of Apache Hadoop, Apache Spark and Graph Analysis
You'll learn how to use Apche Zepplin to develop a hello world example of Spark
Deploy Azure Virtual Networks via the Portal

Course content

11 sections • 74 lectures • 8h 41m total length

Introduction4:25
Course introduction, lecture overview, course objectives
What is databricks?4:43
What is databricks, a preliminary understanding of databricks, which is a company.

Hands-on: How to navigate to the databricks service?1:40
How to find databricks service on azure cloud
Hands-on: How to create a workspace?3:20
Understand how to create a workspace in azure databricks through hands-on
Hands-on: How to create a spark cluster?4:48
Understand how to create a spark cluster in a workspace through hands-on
Hands-on: How to create a notebook?0:43
Understand how to create a notebook in a workspace through hands-on
Hands-on: How to create a table?0:56
Understand how to create a table in the notebook through hands-on
Hands-on: How to delete a spark cluster?0:45
Understand how to delete a spark cluster in a workspace through hands-on
Hands-on: How to delete all resources in Azure Cloud?5:02
Delete resources in time to prevent increasing the cost of your cloud
What is workspace?1:44
Understand why workspace is used in azure databricks
What is Resource Group?2:39
Understanding the importance of using resource groups in azure cloud
What is Databricks Runtime?1:35
Understand how to choose Databricks Runtime correctly
What is Cluster?6:03
Understand the types of clusters

History of Hadoop15:54
History of hadoop can let us know the origin and development
What is Apache Spark?8:14
Understand the advantages of Apache Spark, what scenarios spark is used in.
Hands-on: Download and install virtualbox3:24
You will learn how to install and use virtual machines to pave the way for seting up Apache spark locally
Hands-on: Download and Install putty0:59
You will learn what putty is and lay the groundwork for the following practices
Hands-on: Download and Install winscp2:17
You will learn how to download and use winscp
Hands-on: How to download HDP?2:19
You'll learn what HDP is and how to download it.
Hands-on: How to set up HDP6:21
You will learn how to install HDP
Hands-on: How to SSH into HDP with PuTTY?1:42
You'll learn how to use SSH to connect to HDP and execute shell commands
Hands-on: How to connect to HDP with WinSCP?1:22
You will learn how to operate HDP easily through winscp
Hands-on: How to connect to HDP with Web Shell?1:05
You'll learn what a web shell is and how to use it.
How to use ambari to manage Hadoop?5:34
You'll learn about the start-up and management of Hadoop ecosystems
Hands-on: Apache Spark Quick Start3:52
Develop a spark program and get a preliminary understanding of Apache spark
Apache Spark Architecture6:26
Spark's Architecture and Basic Concepts
What is the ecosystem of Apache Spark?5:11
You'll learn about components in the spark ecosystem

What is Databricks Developer tools?6:23
Developer tools help you develop Azure Databricks applications
Hands-on: Download and install python2:31
Learn how to download and install Python and use databricks cli
Hands-on: How to set up databricks cli?6:32
Installation of databricks cli
Hands-on: How to use databricks cli?10:03
Learn how to use commands in databricks cli to operate workspace and dbfs
Hands-on: How to use Databricks Utilities?2:49
Learn how to use DBUtils to operate dbfs and how to use magic commands
Hands-on: Download and install JDK3:11
Understand JDK download and installation
Hands-on: Download and install IntelliJ IDEA5:34
Understand the download and installation of IDEA development tools
Hands-on: Using Databricks Utilities API Library in IDE16:38
You'll learn how to develop a custom library through IDEA and install it into databricks
Hands-on: How to use databricks in Azure Data Factory18:53
You will learn how to use Azure Data Factory and how to integrate Azure Data Factory with Azure Databricks.
Hands-on: How to debug the notebook in pipeline?6:27
You'll learn how to analyze and debug notebook code in Azure Data Factory
Hands-on: ETL with Azure Databricks26:18
Deployment and use of ETL use cases
Hands-on: How to debug ETL notebook in ETL pipeline?14:40
How to verify and debug ETL

Hands-on: Download and install postman2:42
You'll learn what postman is and how to install it in two ways
Hands-on: Generate a token0:53
Generate a token in databricks for authentication of restful API
Hands-on: Create a spark cluster using REST API2:03
The Spark Clusters can be created either manually or through restful APIs
Hands-on: Delete a spark cluster using REST API2:23
Understand how to delete a spark cluster with restful API
Hands-on: Permanently delete a spark cluster using REST API1:49
How to delete spark cluster completely and release resources.

History of Artificial Intelligence13:12
You'll learn about the history of AI, how neural networks came into being, and the trend of AI.
What is Machine Learning?3:10
Workflow of Machine Learning12:18
You will learn what machine learning is and key concepts in machine learning through animated stories.
Hands-on: Using scikit-learn with Spark on Databricks3:14
How to Use Single-machine scikit-learn in Spark Cluster for Machine Learning
Hands-on: Using Spark to distribute parameter tuning10:57
You'll learn how to tune machine learning models with distributed spark
Hands-on: Using tensorflow with Spark on Databricks14:42
You will learn how to train a neural network model using tensorflow based on GPU
Introduction to Distributed Deep Learning6:08
Origin of Distributed Deep Learning, Architecture of Distributed Deep Learning, Application Trend of GPU
Distributed methods for using TensorFlow5:39
Framework, method, architecture of distributed deployment of tensorflow
Hands-on: Distributed deep learning training using TensorFlow with HorovodRunner11:25
Introducing HorovodRunner for Distributed Deep Learning Training
Hands-on: Using MLflow to track parameters and metrics12:14
you'll learn about how to manage the end-to-end machine learning lifecycle
History of Transfer Learning11:22
What is transfer learning and how does transfer learning come into being?
Hands-on: How to perform Transfer Learning on Databricks?18:20
use GPU machines to practice the transfer learning on databricks

Hands-on: Create a virtual network1:50
You'll learn what a virtual network is and how to create a virtual network for HDInsight
Hands-on: Create an Kafka cluster on HDInsight2:31
How to create a Kafka cluster using HDInsight service in azure
Hands-on: How to connect to kafka using an SSH client3:18
You will learn how to use your local computer to connect to the Kafka cluster in the azure cloud
Hands-on: Configure Kafka for IP advertising3:03
In order to communicate between Kafka and databricks, you need to modify the configuration of Kafka
Hands-on: Peer the Kafka cluster to the Azure Databricks cluster1:37
Kafka and databricks are on different networks, and you'll learn how to make the two networks interoperable
Hands-on: Create an Apache Kafka topic0:39
You will learn how to use commands to create topics for Kafka in HDInsight
Hands-on: Production Structured Streaming with Kafka2:34
You will learn how to send messages to the Kafka cluster in databricks
Hands-on: Consumption Structured Streaming with Kafka0:37
You'll learn how to receive Kafka messages in databricks

History of Graph Analytics8:54
You will understand the story of the birth of graph theory, the application scenarios of graph analysis, and the development of graph computing.
Hands-on: How to install dataframes library?1:49
Spark cluster does not contain dataframes library, you will learn how to install dependency libraries in spark cluster.
Hands-on: How to create a graph?3:13
Understand how to create a graph in the spark cluster through hands-on, you will also learn how to query vertices, edges and degree.

Requirements

Apache Spark basic fundamental knowledge is required
Following browsers on Windows and macOS desktop
Free or paid subscription for Microsoft Azure portal.

Description

Microsoft Azure is the fastest growing cloud platform in the world. No prior Azure experience required.

Azure Databricks is unique collaboration between Microsoft and Databricks, forged to deliver Databricks’ Apache Spark-based analytics offering to the Microsoft Azure cloud. With Azure Databricks, you can be developing your first solution within minutes. Azure Databricks is a fast, easy and collaborative Apache Spark–based analytics service.

Databricks builds on top of Spark and adds:

Highly reliable and performant data pipelines

Productive data science at scale

In this course, you'll have a strong understanding of azure databricks, you will know how to use Spark SQL, Machine Learning, Graph Computing and Structured Streaming Computing in Aziure Databricks.

Why Azure Databricks?

Productive : Launch your new Apache Spark environment in minutes.

Scalable : Globally scale your analytics and machine learning projects.

Trusted : Help protect your data and business with Azure AD integration, role-based controls and enterprise-grade SLAs.

Flexible : Build machine learning and AI solutions with your choice of language and deep learning frameworks.

This course contains both theory lectures ( slides are attached to download for reference) and a significant number of hands-on demos that helps you in gaining hands-on experience. This course help you in laying strong basic foundation in preparation of Microsoft Azure Cloud and Databricks.

In this course, you can not only learn azure databricks, but also learn and practice Machine Learning, Streaming Computing, Graph Analysis, installation and deployment of Open Source Apache spark.

Who this course is for:

Anyone who wants to learn Spark, Machine Learning using Azure Databricks
Beginner Apache Spark Developer, Bigdata Engineers or Developers, Software Developer
Students who are willing to learn Azure from ground zero
Students who would like to make a career switch to Microsoft Azure cloud

Azure Cloud Azure Databricks Apache Spark Machine learning

What you'll learn

Explore related topics

Course content

Introduction2 lectures • 9min

Databricks Quickstart11 lectures • 29min

Apache Spark14 lectures • 1hr 5min

Databricks Developer Tools12 lectures • 2hr

Databricks Notebook2 lectures • 21min

Databricks Delta Lake2 lectures • 34min

Databricks REST API5 lectures • 10min

Databricks Machine Learning12 lectures • 2hr 3min

Structured Streaming8 lectures • 16min

Databricks Graph Analysis3 lectures • 14min

Requirements

Description

Who this course is for: