Azure Cloud Azure Databricks Apache Spark Machine learning
3.3 (99 ratings)
Course Ratings are calculated from individual students’ ratings and a variety of other signals, like age of rating and reliability, to ensure that they reflect course quality fairly and accurately.
525 students enrolled

Azure Cloud Azure Databricks Apache Spark Machine learning

Big Data, Spark SQL, Hadoop, Kafka, Data Lake, Transfer Learning, Zeppelin Notebook, Graph, Hortonworks HDP, Cloudbreak
3.3 (99 ratings)
Course Ratings are calculated from individual students’ ratings and a variety of other signals, like age of rating and reliability, to ensure that they reflect course quality fairly and accurately.
525 students enrolled
Created by Bigdata Engineer
Last updated 5/2020
English
English [Auto]
Current price: $13.99 Original price: $19.99 Discount: 30% off
5 hours left at this price!
30-Day Money-Back Guarantee
This course includes
  • 8.5 hours on-demand video
  • 39 downloadable resources
  • Full lifetime access
  • Access on mobile and TV
  • Certificate of Completion
Training 5 or more people?

Get your team access to 4,000+ top Udemy courses anytime, anywhere.

Try Udemy for Business
What you'll learn
  • This course will provide you an in depth knowledge of apache Spark and how to work with spark using Azure Databricks
  • You will understand the main concepts of Azure Cloud
  • You'll learn about the basic use of Azure Cloud
  • You will learn to deploy Apache Spark ecosystems locally
  • You'll learn how to develop a databricks dependency library through IDEA
  • You will be able to process continual streams of data with Spark streaming
  • You will learn how to train a machine learning model
  • You will understand the use case of graph analysis
  • Deploy Azure Virtual Networks via the Portal
  • Deploy Azure Resource Groups
  • You'll learn how to use Azure Data Factory
  • You'll learn the historical story of Apache Hadoop, Apache Spark and Graph Analysis
  • You'll learn how to use Apche Zepplin to develop a hello world example of Spark
  • Deploy Azure Virtual Networks via the Portal
Course content
Expand all 74 lectures 08:41:45
+ Introduction
2 lectures 09:08

Course introduction, lecture overview, course objectives

Preview 04:25

What is databricks, a preliminary understanding of databricks, which is a company.

Preview 04:43
+ Databricks Quickstart
11 lectures 29:15

How to find databricks service on azure cloud

Preview 01:40

Understand how to create a workspace in azure databricks through hands-on

Hands-on: How to create a workspace?
03:20

Understand how to create a spark cluster in a workspace through hands-on

Hands-on: How to create a spark cluster?
04:48

Understand how to create a notebook in a workspace through hands-on

Hands-on: How to create a notebook?
00:43

Understand how to create a table in the notebook through hands-on

Preview 00:56

Understand how to delete a spark cluster in a workspace through hands-on

Hands-on: How to delete a spark cluster?
00:45

Delete resources in time to prevent increasing the cost of your cloud

Hands-on: How to delete all resources in Azure Cloud?
05:02

Understand why workspace is used in azure databricks

What is workspace?
01:44

Understanding the importance of using resource groups in azure cloud

What is Resource Group?
02:39

Understand how to choose Databricks Runtime correctly

What is Databricks Runtime?
01:35

Understand the types of clusters

What is Cluster?
06:03
+ Apache Spark
14 lectures 01:04:40

History of hadoop can let us know the origin and development

Preview 15:54

Understand the advantages of Apache Spark, what scenarios spark is used in.

What is Apache Spark?
08:14

You will learn how to install and use virtual machines to pave the way for seting up Apache spark locally

Hands-on: Download and install virtualbox
03:24

You will learn what putty is and lay the groundwork for the following practices

Hands-on: Download and Install putty
00:59

You will learn how to download and use winscp

Hands-on: Download and Install winscp
02:17

You'll learn what HDP is and how to download it.

Hands-on: How to download HDP?
02:19

You will learn how to install HDP

Hands-on: How to set up HDP
06:21

You'll learn how to use SSH to connect to HDP and execute shell commands

Hands-on: How to SSH into HDP with PuTTY?
01:42

You will learn how to operate HDP easily through winscp

Hands-on: How to connect to HDP with WinSCP?
01:22

You'll learn what a web shell is and how to use it.

Hands-on: How to connect to HDP with Web Shell?
01:05

You'll learn about the start-up and management of Hadoop ecosystems

How to use ambari to manage Hadoop?
05:34

Develop a spark program and get a preliminary understanding of Apache spark

Hands-on: Apache Spark Quick Start
03:52

Spark's Architecture and Basic Concepts

Apache Spark Architecture
06:26

You'll learn about components in the spark ecosystem

What is the ecosystem of Apache Spark?
05:11
+ Databricks Developer Tools
12 lectures 01:59:59

Developer tools help you develop Azure Databricks applications

What is Databricks Developer tools?
06:23

Learn how to download and install Python and use databricks cli

Hands-on: Download and install python
02:31

Installation of databricks cli

Hands-on: How to set up databricks cli?
06:32

Learn how to use commands in databricks cli to operate workspace and dbfs

Hands-on: How to use databricks cli?
10:03

Learn how to use DBUtils to operate dbfs and how to use magic commands

Hands-on: How to use Databricks Utilities?
02:49

Understand JDK download and installation

Hands-on: Download and install JDK
03:11

Understand the download and installation of IDEA development tools

Hands-on: Download and install IntelliJ IDEA
05:34

You'll learn how to develop a custom library through IDEA and install it into databricks

Hands-on: Using Databricks Utilities API Library in IDE
16:38

You will learn how to use Azure Data Factory and how to integrate Azure Data Factory with Azure Databricks.

Hands-on: How to use databricks in Azure Data Factory
18:53

You'll learn how to analyze and debug notebook code in Azure Data Factory

Hands-on: How to debug the notebook in pipeline?
06:27

Deployment and use of ETL use cases

Hands-on: ETL with Azure Databricks
26:18

How to verify and debug ETL

Hands-on: How to debug ETL notebook in ETL pipeline?
14:40
+ Databricks Notebook
2 lectures 20:41

You will learn about the types and functions of notebook

What is notebook?
05:59

You'll learn about the powerful visualization features of notebook in databricks

Hands-on: Using notebook to visualize data
14:42
+ Databricks Delta Lake
2 lectures 34:18

On the first day of the Spark AI Summit 2019, databricks announced a new open source project called Delta Lake to deliver reliability to data lakes, this can be considered as Data Lake 2.0. This lecture demonstrates how to use delta Lake in Apache spark.

Hands-on: Set up Apache Spark with Delta Lake
14:59

you'll learn about Delta Lake on Databricks

Hands-on: Using python to operate delta lake
19:19
+ Databricks REST API
5 lectures 09:50

You'll learn what postman is and how to install it in two ways

Hands-on: Download and install postman
02:42

Generate a token in databricks for authentication of restful API

Hands-on: Generate a token
00:53

The Spark Clusters can be created either manually or through restful APIs

Hands-on: Create a spark cluster using REST API
02:03

Understand how to delete a spark cluster with restful API

Hands-on: Delete a spark cluster using REST API
02:23

How to delete spark cluster completely and release resources.

Hands-on: Permanently delete a spark cluster using REST API
01:49
+ Databricks Machine Learning
12 lectures 02:02:41

You'll learn about the history of AI, how neural networks came into being, and the trend of AI.

History of Artificial Intelligence
13:12
What is Machine Learning?
03:10

You will learn what machine learning is and key concepts in machine learning through animated stories.

Workflow of Machine Learning
12:18

How to Use Single-machine scikit-learn in Spark Cluster for Machine Learning

Hands-on: Using scikit-learn with Spark on Databricks
03:14

You'll learn how to tune machine learning models with distributed spark

Hands-on: Using Spark to distribute parameter tuning
10:57

You will learn how to train a neural network model using tensorflow based on GPU

Hands-on: Using tensorflow with Spark on Databricks
14:42

Origin of Distributed Deep Learning, Architecture of Distributed Deep Learning, Application Trend of GPU

Introduction to Distributed Deep Learning
06:08

Framework, method, architecture of distributed deployment of tensorflow

Distributed methods for using TensorFlow
05:39

Introducing HorovodRunner for Distributed Deep Learning Training

Hands-on: Distributed deep learning training using TensorFlow with HorovodRunner
11:25

you'll learn about how to manage the end-to-end machine learning lifecycle

Hands-on: Using MLflow to track parameters and metrics
12:14

What is transfer learning and how does transfer learning come into being?

History of Transfer Learning
11:22

use GPU machines to practice the transfer learning on databricks

Hands-on: How to perform Transfer Learning on Databricks?
18:20
+ Structured Streaming
8 lectures 16:09

You'll learn what a virtual network is and how to create a virtual network for HDInsight

Hands-on: Create a virtual network
01:50

How to create a Kafka cluster using HDInsight service in azure

Hands-on: Create an Kafka cluster on HDInsight
02:31

You will learn how to use your local computer to connect to the Kafka cluster in the azure cloud

Hands-on: How to connect to kafka using an SSH client
03:18

In order to communicate between Kafka and databricks, you need to modify the configuration of Kafka

Hands-on: Configure Kafka for IP advertising
03:03

Kafka and databricks are on different networks, and you'll learn how to make the two networks interoperable

Hands-on: Peer the Kafka cluster to the Azure Databricks cluster
01:37

You will learn how to use commands to create topics for Kafka in HDInsight

Hands-on: Create an Apache Kafka topic
00:39

You will learn how to send messages to the Kafka cluster in databricks

Hands-on: Production Structured Streaming with Kafka
02:34

You'll learn how to receive Kafka messages in databricks

Hands-on: Consumption Structured Streaming with Kafka
00:37
+ Databricks Graph Analysis
3 lectures 13:56

You will understand the story of the birth of graph theory, the application scenarios of graph analysis, and the development of graph computing.

History of Graph Analytics
08:54

Spark cluster does not contain dataframes library, you will learn how to install dependency libraries in spark cluster.

Hands-on: How to install dataframes library?
01:49

Understand how to create a graph in the spark cluster through hands-on, you will also learn how to query vertices, edges and degree.

Hands-on: How to create a graph?
03:13
Requirements
  • Apache Spark basic fundamental knowledge is required
  • Following browsers on Windows and macOS desktop
  • Free or paid subscription for Microsoft Azure portal.
Description

Microsoft Azure is the fastest growing cloud platform in the world. No prior Azure experience required.

Azure Databricks is unique collaboration between Microsoft and Databricks, forged to deliver Databricks’ Apache Spark-based analytics offering to the Microsoft Azure cloud. With Azure Databricks, you can be developing your first solution within minutes. Azure Databricks is a fast, easy and collaborative Apache Spark–based analytics service.


Databricks builds on top of Spark and adds:

     Highly reliable and performant data pipelines

     Productive data science at scale


In this course, you'll have a strong understanding of azure databricks, you will know how to use Spark SQL, Machine Learning, Graph Computing and Structured Streaming Computing in Aziure Databricks.


Why Azure Databricks?

Productive : Launch your new Apache Spark environment in minutes.

Scalable : Globally scale your analytics and machine learning projects.

Trusted : Help protect your data and business with Azure AD integration, role-based controls and enterprise-grade SLAs.

Flexible : Build machine learning and AI solutions with your choice of language and deep learning frameworks.


This course contains both theory lectures ( slides are attached to download for reference) and a significant number of hands-on demos that helps you in gaining hands-on experience. This course help you in laying strong basic foundation in preparation of Microsoft Azure Cloud and Databricks.


In this course, you can not only learn azure databricks, but also learn and practice Machine Learning, Streaming Computing, Graph Analysis, installation and deployment of Open Source Apache spark.

Who this course is for:
  • Anyone who wants to learn Spark, Machine Learning using Azure Databricks
  • Beginner Apache Spark Developer, Bigdata Engineers or Developers, Software Developer
  • Students who are willing to learn Azure from ground zero
  • Students who would like to make a career switch to Microsoft Azure cloud