Azure Cloud Azure Databricks Apache Spark Machine learning

525 students enrolled

Azure Cloud Azure Databricks Apache Spark Machine learning

Big Data, Spark SQL, Hadoop, Kafka, Data Lake, Transfer Learning, Zeppelin Notebook, Graph, Hortonworks HDP, Cloudbreak

525 students enrolled

Created by Bigdata Engineer

Last updated 5/2020

English

English [Auto]

Preview this course

Azure Cloud Azure Databricks Apache Spark Machine learning

$13.99 $19.99 30% off

5 hours left at this price!

30-Day Money-Back Guarantee

8.5 hours on-demand video
39 downloadable resources
Full lifetime access
Access on mobile and TV

Certificate of Completion

Training 5 or more people?

Get your team access to 4,000+ top Udemy courses anytime, anywhere.

Try Udemy for Business

What you'll learn

This course will provide you an in depth knowledge of apache Spark and how to work with spark using Azure Databricks
You will understand the main concepts of Azure Cloud
You'll learn about the basic use of Azure Cloud
You will learn to deploy Apache Spark ecosystems locally
You'll learn how to develop a databricks dependency library through IDEA
You will be able to process continual streams of data with Spark streaming
You will learn how to train a machine learning model
You will understand the use case of graph analysis
Deploy Azure Virtual Networks via the Portal
Deploy Azure Resource Groups
You'll learn how to use Azure Data Factory
You'll learn the historical story of Apache Hadoop, Apache Spark and Graph Analysis
You'll learn how to use Apche Zepplin to develop a hello world example of Spark
Deploy Azure Virtual Networks via the Portal

+ – Introduction

2 lectures 09:08

Course introduction, lecture overview, course objectives

Preview 04:25

What is databricks, a preliminary understanding of databricks, which is a company.

What is databricks?

Preview 04:43

+ – Databricks Quickstart

11 lectures 29:15

How to find databricks service on azure cloud

Hands-on: How to navigate to the databricks service?

Preview 01:40

Understand how to create a workspace in azure databricks through hands-on

Hands-on: How to create a workspace?

03:20

Understand how to create a spark cluster in a workspace through hands-on

Hands-on: How to create a spark cluster?

04:48

Understand how to create a notebook in a workspace through hands-on

Hands-on: How to create a notebook?

00:43

Understand how to create a table in the notebook through hands-on

Hands-on: How to create a table?

Preview 00:56

Understand how to delete a spark cluster in a workspace through hands-on

Hands-on: How to delete a spark cluster?

00:45

Delete resources in time to prevent increasing the cost of your cloud

Hands-on: How to delete all resources in Azure Cloud?

05:02

Understand why workspace is used in azure databricks

What is workspace?

01:44

Understanding the importance of using resource groups in azure cloud

What is Resource Group?

02:39

Understand how to choose Databricks Runtime correctly

What is Databricks Runtime?

01:35

Understand the types of clusters

What is Cluster?

06:03

+ – Apache Spark

14 lectures 01:04:40

History of hadoop can let us know the origin and development

History of Hadoop

Preview 15:54

Understand the advantages of Apache Spark, what scenarios spark is used in.

What is Apache Spark?

08:14

You will learn how to install and use virtual machines to pave the way for seting up Apache spark locally

Hands-on: Download and install virtualbox

03:24

You will learn what putty is and lay the groundwork for the following practices

Hands-on: Download and Install putty

00:59

You will learn how to download and use winscp

Hands-on: Download and Install winscp

02:17

You'll learn what HDP is and how to download it.

Hands-on: How to download HDP?

02:19

You will learn how to install HDP

Hands-on: How to set up HDP

06:21

You'll learn how to use SSH to connect to HDP and execute shell commands

Hands-on: How to SSH into HDP with PuTTY?

01:42

You will learn how to operate HDP easily through winscp

Hands-on: How to connect to HDP with WinSCP?

01:22

You'll learn what a web shell is and how to use it.

Hands-on: How to connect to HDP with Web Shell?

01:05

You'll learn about the start-up and management of Hadoop ecosystems

How to use ambari to manage Hadoop?

05:34

Develop a spark program and get a preliminary understanding of Apache spark

Hands-on: Apache Spark Quick Start

03:52

Spark's Architecture and Basic Concepts

Apache Spark Architecture

06:26

You'll learn about components in the spark ecosystem

What is the ecosystem of Apache Spark?

05:11

+ – Databricks Developer Tools

12 lectures 01:59:59

Developer tools help you develop Azure Databricks applications

What is Databricks Developer tools?

06:23

Learn how to download and install Python and use databricks cli

Hands-on: Download and install python

02:31

Installation of databricks cli

Hands-on: How to set up databricks cli?

06:32

Learn how to use commands in databricks cli to operate workspace and dbfs

Hands-on: How to use databricks cli?

10:03

Learn how to use DBUtils to operate dbfs and how to use magic commands

Hands-on: How to use Databricks Utilities?

02:49

Understand JDK download and installation

Hands-on: Download and install JDK

03:11

Understand the download and installation of IDEA development tools

Hands-on: Download and install IntelliJ IDEA

05:34

You'll learn how to develop a custom library through IDEA and install it into databricks

Hands-on: Using Databricks Utilities API Library in IDE

16:38

You will learn how to use Azure Data Factory and how to integrate Azure Data Factory with Azure Databricks.

Hands-on: How to use databricks in Azure Data Factory

18:53

You'll learn how to analyze and debug notebook code in Azure Data Factory

Hands-on: How to debug the notebook in pipeline?

06:27

Deployment and use of ETL use cases

Hands-on: ETL with Azure Databricks

26:18

How to verify and debug ETL

Hands-on: How to debug ETL notebook in ETL pipeline?

14:40

+ – Databricks Notebook

2 lectures 20:41

You will learn about the types and functions of notebook

What is notebook?

05:59

You'll learn about the powerful visualization features of notebook in databricks

Hands-on: Using notebook to visualize data

14:42

+ – Databricks Delta Lake

2 lectures 34:18

On the first day of the Spark AI Summit 2019, databricks announced a new open source project called Delta Lake to deliver reliability to data lakes, this can be considered as Data Lake 2.0. This lecture demonstrates how to use delta Lake in Apache spark.

Hands-on: Set up Apache Spark with Delta Lake

14:59

you'll learn about Delta Lake on Databricks

Hands-on: Using python to operate delta lake

19:19

+ – Databricks REST API

5 lectures 09:50

You'll learn what postman is and how to install it in two ways

Hands-on: Download and install postman

02:42

Generate a token in databricks for authentication of restful API

Hands-on: Generate a token

00:53

The Spark Clusters can be created either manually or through restful APIs

Hands-on: Create a spark cluster using REST API

02:03

Understand how to delete a spark cluster with restful API

Hands-on: Delete a spark cluster using REST API

02:23

How to delete spark cluster completely and release resources.

Hands-on: Permanently delete a spark cluster using REST API

01:49

+ – Databricks Machine Learning

12 lectures 02:02:41

You'll learn about the history of AI, how neural networks came into being, and the trend of AI.

History of Artificial Intelligence

13:12

What is Machine Learning?

03:10

You will learn what machine learning is and key concepts in machine learning through animated stories.

Workflow of Machine Learning

12:18

How to Use Single-machine scikit-learn in Spark Cluster for Machine Learning

Hands-on: Using scikit-learn with Spark on Databricks

03:14

You'll learn how to tune machine learning models with distributed spark

Hands-on: Using Spark to distribute parameter tuning

10:57

You will learn how to train a neural network model using tensorflow based on GPU

Hands-on: Using tensorflow with Spark on Databricks

14:42

Origin of Distributed Deep Learning, Architecture of Distributed Deep Learning, Application Trend of GPU

Introduction to Distributed Deep Learning

06:08

Framework, method, architecture of distributed deployment of tensorflow

Distributed methods for using TensorFlow

05:39

Introducing HorovodRunner for Distributed Deep Learning Training

Hands-on: Distributed deep learning training using TensorFlow with HorovodRunner

11:25

you'll learn about how to manage the end-to-end machine learning lifecycle

Hands-on: Using MLflow to track parameters and metrics

12:14

What is transfer learning and how does transfer learning come into being?

History of Transfer Learning

11:22

use GPU machines to practice the transfer learning on databricks

Hands-on: How to perform Transfer Learning on Databricks?

18:20

+ – Structured Streaming

8 lectures 16:09

You'll learn what a virtual network is and how to create a virtual network for HDInsight

Hands-on: Create a virtual network

01:50

How to create a Kafka cluster using HDInsight service in azure

Hands-on: Create an Kafka cluster on HDInsight

02:31

You will learn how to use your local computer to connect to the Kafka cluster in the azure cloud

Hands-on: How to connect to kafka using an SSH client

03:18

In order to communicate between Kafka and databricks, you need to modify the configuration of Kafka

Hands-on: Configure Kafka for IP advertising

03:03

Kafka and databricks are on different networks, and you'll learn how to make the two networks interoperable

Hands-on: Peer the Kafka cluster to the Azure Databricks cluster

01:37

You will learn how to use commands to create topics for Kafka in HDInsight

Hands-on: Create an Apache Kafka topic

00:39

You will learn how to send messages to the Kafka cluster in databricks

Hands-on: Production Structured Streaming with Kafka

02:34

You'll learn how to receive Kafka messages in databricks

Hands-on: Consumption Structured Streaming with Kafka

00:37

+ – Databricks Graph Analysis

3 lectures 13:56

You will understand the story of the birth of graph theory, the application scenarios of graph analysis, and the development of graph computing.

History of Graph Analytics

08:54

Spark cluster does not contain dataframes library, you will learn how to install dependency libraries in spark cluster.

Hands-on: How to install dataframes library?

01:49

Understand how to create a graph in the spark cluster through hands-on, you will also learn how to query vertices, edges and degree.

Hands-on: How to create a graph?

03:13

Requirements

Apache Spark basic fundamental knowledge is required
Following browsers on Windows and macOS desktop
Free or paid subscription for Microsoft Azure portal.

Description

Microsoft Azure is the fastest growing cloud platform in the world. No prior Azure experience required.

Azure Databricks is unique collaboration between Microsoft and Databricks, forged to deliver Databricks’ Apache Spark-based analytics offering to the Microsoft Azure cloud. With Azure Databricks, you can be developing your first solution within minutes. Azure Databricks is a fast, easy and collaborative Apache Spark–based analytics service.

Databricks builds on top of Spark and adds:

Highly reliable and performant data pipelines

Productive data science at scale

In this course, you'll have a strong understanding of azure databricks, you will know how to use Spark SQL, Machine Learning, Graph Computing and Structured Streaming Computing in Aziure Databricks.

Why Azure Databricks?

Productive : Launch your new Apache Spark environment in minutes.

Scalable : Globally scale your analytics and machine learning projects.

Trusted : Help protect your data and business with Azure AD integration, role-based controls and enterprise-grade SLAs.

Flexible : Build machine learning and AI solutions with your choice of language and deep learning frameworks.

This course contains both theory lectures ( slides are attached to download for reference) and a significant number of hands-on demos that helps you in gaining hands-on experience. This course help you in laying strong basic foundation in preparation of Microsoft Azure Cloud and Databricks.

In this course, you can not only learn azure databricks, but also learn and practice Machine Learning, Streaming Computing, Graph Analysis, installation and deployment of Open Source Apache spark.

Who this course is for:

Anyone who wants to learn Spark, Machine Learning using Azure Databricks
Beginner Apache Spark Developer, Bigdata Engineers or Developers, Software Developer
Students who are willing to learn Azure from ground zero
Students who would like to make a career switch to Microsoft Azure cloud