Udemy
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
Development
Web Development Data Science Mobile Development Programming Languages Game Development Database Design & Development Software Testing Software Engineering Development Tools No-Code Development
Business
Entrepreneurship Communications Management Sales Business Strategy Operations Project Management Business Law Business Analytics & Intelligence Human Resources Industry E-Commerce Media Real Estate Other Business
Finance & Accounting
Accounting & Bookkeeping Compliance Cryptocurrency & Blockchain Economics Finance Finance Cert & Exam Prep Financial Modeling & Analysis Investing & Trading Money Management Tools Taxes Other Finance & Accounting
IT & Software
IT Certification Network & Security Hardware Operating Systems Other IT & Software
Office Productivity
Microsoft Apple Google SAP Oracle Other Office Productivity
Personal Development
Personal Transformation Personal Productivity Leadership Career Development Parenting & Relationships Happiness Esoteric Practices Religion & Spirituality Personal Brand Building Creativity Influence Self Esteem & Confidence Stress Management Memory & Study Skills Motivation Other Personal Development
Design
Web Design Graphic Design & Illustration Design Tools User Experience Design Game Design Design Thinking 3D & Animation Fashion Design Architectural Design Interior Design Other Design
Marketing
Digital Marketing Search Engine Optimization Social Media Marketing Branding Marketing Fundamentals Marketing Analytics & Automation Public Relations Advertising Video & Mobile Marketing Content Marketing Growth Hacking Affiliate Marketing Product Marketing Other Marketing
Lifestyle
Arts & Crafts Beauty & Makeup Esoteric Practices Food & Beverage Gaming Home Improvement Pet Care & Training Travel Other Lifestyle
Photography & Video
Digital Photography Photography Portrait Photography Photography Tools Commercial Photography Video Design Other Photography & Video
Health & Fitness
Fitness General Health Sports Nutrition Yoga Mental Health Dieting Self Defense Safety & First Aid Dance Meditation Other Health & Fitness
Music
Instruments Music Production Music Fundamentals Vocal Music Techniques Music Software Other Music
Teaching & Academics
Engineering Humanities Math Science Online Education Social Science Language Teacher Training Test Prep Other Teaching & Academics
AWS Certification Microsoft Certification AWS Certified Solutions Architect - Associate AWS Certified Cloud Practitioner CompTIA A+ Cisco CCNA Amazon AWS AWS Certified Developer - Associate CompTIA Security+
Photoshop Graphic Design Adobe Illustrator Drawing Digital Painting InDesign Character Design Canva Figure Drawing
Life Coach Training Neuro-Linguistic Programming Mindfulness Personal Development Personal Transformation Meditation Life Purpose Coaching Neuroscience
Web Development JavaScript React CSS Angular PHP Node.Js WordPress Python
Google Flutter Android Development iOS Development Swift React Native Dart Programming Language Mobile Development Kotlin SwiftUI
Digital Marketing Google Ads (Adwords) Social Media Marketing Google Ads (AdWords) Certification Marketing Strategy Internet Marketing YouTube Marketing Email Marketing Retargeting
SQL Microsoft Power BI Tableau Business Analysis Business Intelligence MySQL Data Analysis Data Modeling Big Data
Business Fundamentals Entrepreneurship Fundamentals Business Strategy Online Business Business Plan Startup Freelancing Blogging Home Business
Unity Game Development Fundamentals Unreal Engine C# 3D Game Development C++ 2D Game Development Unreal Engine Blueprints Blender
30-Day Money-Back Guarantee

This course includes:

  • 6 hours on-demand video
  • 2 articles
  • Full lifetime access
  • Access on mobile and TV
Development Data Science Apache Spark

Streaming Big Data with Spark Streaming and Scala - Hands On

Spark Streaming tutorial covering Spark Structured Streaming, Kafka integration, and streaming big data in real-time.
Rating: 4.4 out of 54.4 (2,871 ratings)
21,060 students
Created by Sundog Education by Frank Kane, Frank Kane
Last updated 3/2020
English
English [Auto], Italian [Auto]
30-Day Money-Back Guarantee

What you'll learn

  • Process massive streams of real-time data using Spark Streaming
  • Integrate Spark Streaming with data sources, including Kafka, Flume, and Kinesis
  • Use Spark 2's Structured Streaming API
  • Create Spark applications using the Scala programming language
  • Output transformed real-time data to Cassandra or file systems
  • Integrate Spark Streaming with Spark SQL to query streaming data in real time
  • Train machine learning models with streaming data, and use those models for real-time predictions
  • Ingest Apache access log data and transform streams of it
  • Receive real-time streams of Twitter feeds
  • Maintain stateful data across a continuous stream of input data
  • Query streaming data across sliding windows of time
Curated for the Udemy for Business collection

Course content

9 sections • 38 lectures • 6h 15m total length

  • Tip: Apply for a Twitter Developer Account now!
    00:51
  • Preview17:27
  • Preview14:11
  • Preview02:10

  • [Activity] Scala Basics: Part 1
    11:26
  • [Exercise] Scala Basics: Part 2
    09:41
  • [Exercise] Flow Control in Scala
    07:18
  • [Exercise] Functions in Scala
    08:47
  • [Excercise] Data Structures in Scala
    16:38

  • Introduction to Spark
    07:06
  • The Resilient Distributed Dataset (RDD)
    10:40
  • [Activity] RDD's in action: simple word count application
    08:17
  • Introduction to Spark Streaming
    06:32
  • [Activity] Revisiting the PrintTweets application
    05:10
  • Windowing: Aggregating data over longer time spans
    05:00
  • Fault Tolerance in Spark Streaming
    06:06

  • [Exercise] Saving Tweets to Disk
    13:24
  • [Exercise] Tracking the Average Tweet Length
    08:22
  • Preview14:50

  • Preview13:27
  • [Exercise] Alarming on Log Errors
    11:56
  • [Exercise] Integrating Spark Streaming with Spark SQL
    15:03
  • Intro to Structured Streaming in Spark 2
    08:27
  • [Activity] Analyzing Apache Log files with Structured Streaming
    11:24

  • Preview12:20
  • Integrating with Apache Flume
    08:51
  • Integrating with Amazon Kinesis
    05:29
  • [Activity] Writing Custom Data Receivers
    06:55
  • Integrating with Cassandra
    07:35

  • [Exercise] Stateful Information in Spark Streams
    15:07
  • [Activity] Streaming K-Means Clustering
    15:36
  • [Activity] Streaming Linear Regression
    11:50

  • [Activity] Running with spark-submit
    10:37
  • [Activity] Packaging your code with SBT
    17:17
  • Running on a real Hadoop cluster with EMR
    12:56
  • Troubleshooting and Tuning Spark Jobs
    12:35

  • Learning More
    03:44
  • Bonus Lecture: More courses to explore!
    00:38

Requirements

  • To follow along with the examples, you'll need a personal computer. The course is filmed using Windows 10, but the tools we install are available for Linux and MacOS as well.
  • We'll walk through installing the required software in the first lecture: The Scala IDE, Spark, and a JDK.
  • My "Taming Big Data with Apache Spark - Hands On!" would be a helpful introduction to Spark in general, but it is not required for this course. A quick introduction to Spark is included.
  • The course includes a crash course in the Scala programming language if you're new to it; if you already know Scala, then great.

Description

New! Updated for Spark 3.0.0!

"Big Data" analysis is a hot and highly valuable skill. Thing is, "big data" never stops flowing! Spark Streaming is a new and quickly developing technology for processing massive data sets as they are created - why wait for some nightly analysis to run when you can constantly update your analysis in real time, all the time? Whether it's clickstream data from a big website, sensor data from a massive "Internet of Things" deployment, financial data, or something else - Spark Streaming is a powerful technology for transforming and analyzing that data right when it is created, all the time.

You'll be learning from an ex-engineer and senior manager from Amazon and IMDb.

This course gets your hands on to some real live Twitter data, simulated streams of Apache access logs, and even data used to train machine learning models! You'll write and run real Spark Streaming jobs right at home on your own PC, and toward the end of the course, we'll show you how to take those jobs to a real Hadoop cluster and run them in a production environment too.

Across over 30 lectures and almost 6 hours of video content, you'll:

  • Get a crash course in the Scala programming language

  • Learn how Apache Spark operates on a cluster

  • Set up discretized streams with Spark Streaming and transform them as data is received

  • Use structured streaming to stream into dataframes in real-time

  • Analyze streaming data over sliding windows of time

  • Maintain stateful information across streams of data

  • Connect Spark Streaming with highly scalable sources of data, including Kafka, Flume, and Kinesis

  • Dump streams of data in real-time to NoSQL databases such as Cassandra

  • Run SQL queries on streamed data in real time

  • Train machine learning models in real time with streaming data, and use them to make predictions that keep getting better over time

  • Package, deploy, and run self-contained Spark Streaming code to a real Hadoop cluser using Amazon Elastic MapReduce.

This course is very hands-on, filled with achievable activities and exercises to reinforce your learning. By the end of this course, you'll be confidently creating Spark Streaming scripts in Scala, and be prepared to tackle massive streams of data in a whole new way. You'll be surprised at how easy Spark Streaming makes it!

Who this course is for:

  • Students with some prior programming or scripting ability SHOULD take this course.
  • If you're working for a company with "big data" that is being generated continuously, or hope to work for one, this course is for you.
  • Students with no prior software engineering or programming experience should seek an introductory programming course first.

Instructors

Sundog Education by Frank Kane
Founder, Sundog Education. Machine Learning Pro
Sundog Education by Frank Kane
  • 4.5 Instructor Rating
  • 96,067 Reviews
  • 432,487 Students
  • 22 Courses

Sundog Education's mission is to make highly valuable career skills in big data, data science, and machine learning accessible to everyone in the world. Our consortium of expert instructors shares our knowledge in these emerging fields with you, at prices anyone can afford. 

Sundog Education is led by Frank Kane and owned by Frank's company, Sundog Software LLC. Frank spent 9 years at Amazon and IMDb, developing and managing the technology that automatically delivers product and movie recommendations to hundreds of millions of customers, all the time. Frank holds 17 issued patents in the fields of distributed computing, data mining, and machine learning. In 2012, Frank left to start his own successful company, Sundog Software, which focuses on virtual reality environment technology, and teaching others about big data analysis.

Due to our volume of students we are unable to respond to private messages; please post your questions within the Q&A of your course. Thanks for understanding.

Frank Kane
Founder, Sundog Education
Frank Kane
  • 4.5 Instructor Rating
  • 92,764 Reviews
  • 388,456 Students
  • 14 Courses

Frank spent 9 years at Amazon and IMDb, developing and managing the technology that automatically delivers product and movie recommendations to hundreds of millions of customers, all the time. Frank holds 17 issued patents in the fields of distributed computing, data mining, and machine learning. In 2012, Frank left to start his own successful company, Sundog Software, which focuses on virtual reality environment technology, and teaching others about big data analysis.

Due to our volume of students, I am unable to respond to private messages; please post your questions within the Q&A of your course. Thanks for understanding.

  • Udemy for Business
  • Teach on Udemy
  • Get the app
  • About us
  • Contact us
  • Careers
  • Blog
  • Help and Support
  • Affiliate
  • Terms
  • Privacy policy
  • Cookie settings
  • Sitemap
  • Featured courses
Udemy
© 2021 Udemy, Inc.