Udemy
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
Development
Web Development Data Science Mobile Development Programming Languages Game Development Database Design & Development Software Testing Software Engineering Development Tools No-Code Development
Business
Entrepreneurship Communications Management Sales Business Strategy Operations Project Management Business Law Business Analytics & Intelligence Human Resources Industry E-Commerce Media Real Estate Other Business
Finance & Accounting
Accounting & Bookkeeping Compliance Cryptocurrency & Blockchain Economics Finance Finance Cert & Exam Prep Financial Modeling & Analysis Investing & Trading Money Management Tools Taxes Other Finance & Accounting
IT & Software
IT Certification Network & Security Hardware Operating Systems Other IT & Software
Office Productivity
Microsoft Apple Google SAP Oracle Other Office Productivity
Personal Development
Personal Transformation Personal Productivity Leadership Career Development Parenting & Relationships Happiness Esoteric Practices Religion & Spirituality Personal Brand Building Creativity Influence Self Esteem & Confidence Stress Management Memory & Study Skills Motivation Other Personal Development
Design
Web Design Graphic Design & Illustration Design Tools User Experience Design Game Design Design Thinking 3D & Animation Fashion Design Architectural Design Interior Design Other Design
Marketing
Digital Marketing Search Engine Optimization Social Media Marketing Branding Marketing Fundamentals Marketing Analytics & Automation Public Relations Advertising Video & Mobile Marketing Content Marketing Growth Hacking Affiliate Marketing Product Marketing Other Marketing
Lifestyle
Arts & Crafts Beauty & Makeup Esoteric Practices Food & Beverage Gaming Home Improvement Pet Care & Training Travel Other Lifestyle
Photography & Video
Digital Photography Photography Portrait Photography Photography Tools Commercial Photography Video Design Other Photography & Video
Health & Fitness
Fitness General Health Sports Nutrition Yoga Mental Health Dieting Self Defense Safety & First Aid Dance Meditation Other Health & Fitness
Music
Instruments Music Production Music Fundamentals Vocal Music Techniques Music Software Other Music
Teaching & Academics
Engineering Humanities Math Science Online Education Social Science Language Teacher Training Test Prep Other Teaching & Academics
AWS Certification Microsoft Certification AWS Certified Solutions Architect - Associate AWS Certified Cloud Practitioner CompTIA A+ Cisco CCNA CompTIA Security+ Amazon AWS AWS Certified Developer - Associate
Photoshop Graphic Design Adobe Illustrator Drawing Digital Painting InDesign Character Design Canva Figure Drawing
Life Coach Training Neuro-Linguistic Programming Mindfulness Personal Development Personal Transformation Life Purpose Meditation Emotional Intelligence Neuroscience
Web Development JavaScript React CSS Angular PHP WordPress Node.Js Python
Google Flutter Android Development iOS Development Swift React Native Dart Programming Language Mobile Development Kotlin SwiftUI
Digital Marketing Google Ads (Adwords) Social Media Marketing Google Ads (AdWords) Certification Marketing Strategy Internet Marketing YouTube Marketing Email Marketing Google Analytics
SQL Microsoft Power BI Tableau Business Analysis Business Intelligence MySQL Data Analysis Data Modeling Data Science
Business Fundamentals Entrepreneurship Fundamentals Business Strategy Online Business Business Plan Startup Freelancing Blogging Home Business
Unity Game Development Fundamentals Unreal Engine C# 3D Game Development C++ 2D Game Development Unreal Engine Blueprints Blender
30-Day Money-Back Guarantee
Development Data Science Apache Spark

Apache Spark with Scala - Hands On with Big Data!

Apache Spark tutorial with 20+ hands-on examples of analyzing large data sets, on your desktop or on Hadoop with Scala!
Bestseller
Rating: 4.6 out of 54.6 (12,830 ratings)
67,076 students
Created by Sundog Education by Frank Kane, Frank Kane
Last updated 2/2021
English
English, French [Auto], 
30-Day Money-Back Guarantee

What you'll learn

  • Frame big data analysis problems as Apache Spark scripts
  • Develop distributed code using the Scala programming language
  • Optimize Spark jobs through partitioning, caching, and other techniques
  • Build, deploy, and run Spark scripts on Hadoop clusters
  • Process continual streams of data with Spark Streaming
  • Transform structured data using SparkSQL, DataSets, and DataFrames
  • Traverse and analyze graph structures using GraphX
  • Analyze massive data set with Machine Learning on Spark
Curated for the Udemy for Business collection

Course content

10 sections • 68 lectures • 9h 4m total length

  • Preview02:10
  • Alternate download link for the ml-100k dataset
    00:07
  • Preview15:54
  • Introduction to Apache Spark
    14:26
  • Spark Basics
    5 questions
  • Preview06:48

  • [Activity] Scala Basics
    25:58
  • [Exercise] Flow Control in Scala
    09:28
  • [Exercise] Functions in Scala
    09:08
  • [Exercise] Data Structures in Scala
    22:28

  • The Resilient Distributed Dataset
    11:30
  • Ratings Histogram Example
    11:27
  • Preview01:59
  • Key / Value RDD's, and the Average Friends by Age example
    10:42
  • [Activity] Running the Average Friends by Age Example
    04:51
  • Filtering RDD's, and the Minimum Temperature by Location Example
    05:54
  • [Activity] Running the Minimum Temperature Example, and Modifying it for Maximum
    11:35
  • [Activity] Counting Word Occurrences using Flatmap()
    05:46
  • [Activity] Improving the Word Count Script with Regular Expressions
    03:44
  • Preview06:35
  • [Exercise] Find the Total Amount Spent by Customer
    04:30
  • [Exercise] Check your Results, and Sort Them by Total Amount Spent
    05:09
  • Check Your Results and Implementation Against Mine
    03:00

  • Introduction to SparkSQL
    09:44
  • [Activity] Using SparkSQL
    07:05
  • [Activity] Using DataSets
    08:33
  • [Exercise] Implement the "Friends by Age" example using DataSets
    02:40
  • Exercise Solution: Friends by Age, with Datasets.
    07:22
  • [Activity] Word Count example, using Datasets
    10:37
  • [Activity] Revisiting the Minimum Temperature example, with Datasets
    09:00
  • [Exercise] Implement the "Total Spent by Customer" problem with Datasets
    02:10
  • Exercise Solution: Total Spent by Customer with Datasets
    06:28

  • [Activity] Find the Most Popular Movie
    05:24
  • [Activity] Use Broadcast Variables to Display Movie Names
    11:19
  • [Activity] Find the Most Popular Superhero in a Social Graph
    12:18
  • [Exercise] Find the Most Obscure Superheroes
    05:14
  • Exercise Solution: Find the Most Obscure Superheroes
    06:44
  • Superhero Degrees of Separation: Introducing Breadth-First Search
    07:14
  • Superhero Degrees of Separation: Accumulators, and Implementing BFS in Spark
    07:59
  • [Activity] Superhero Degrees of Separation: Review the code, and run it!
    12:55
  • Item-Based Collaborative Filtering in Spark, cache(), and persist()
    07:59
  • [Activity] Running the Similar Movies Script using Spark's Cluster Manager
    Preview14:48
  • [Exercise] Improve the Quality of Similar Movies
    03:54

  • [Activity] Using spark-submit to run Spark driver scripts
    11:43
  • [Activity] Packaging driver scripts with SBT
    15:06
  • [Exercise] Package a Script with SBT and Run it Locally with spark-submit
    02:04
  • Exercise solution: Using SBT and spark-submit
    09:04
  • Introducing Amazon Elastic MapReduce
    07:11
  • Preview11:33
  • Partitioning
    04:18
  • Best Practices for Running on a Cluster
    06:25
  • Troubleshooting, and Managing Dependencies
    10:59

  • Introducing MLLib
    09:56
  • [Activity] Using MLLib to Produce Movie Recommendations
    12:42
  • Linear Regression with MLLib
    06:58
  • [Activity] Running a Linear Regression with Spark
    Preview07:47
  • [Exercise] Predict Real Estate Values with Decision Trees in Spark
    04:56
  • Exercise Solution: Predicting Real Estate with Decision Trees in Spark
    05:47

  • The DStream API for Spark Streaming
    11:28
  • [Activity] Real-time Monitoring of the Most Popular Hashtags on Twitter
    08:51
  • Structured Streaming
    04:03
  • [Activity] Using Structured Streaming for real-time log analysis
    05:33
  • [Exercise] Windowed Operations with Structured Streaming
    06:04
  • Exercise Solution: Top URL's in a 30-second Window
    05:44

  • Preview06:51
  • Using the Pregel API with Spark GraphX
    04:29
  • [Activity] Superhero Degrees of Separation using GraphX
    07:07

  • Learning More, and Career Tips
    04:15
  • Bonus Lecture: More courses to explore!
    00:56

Requirements

  • Some prior programming or scripting experience is required. A crash course in Scala is included, but you need to know the fundamentals of programming in order to pick it up.
  • You will need a desktop PC and an Internet connection. The course is created with Windows in mind, but users comfortable with MacOS or Linux can use the same tools.
  • The software needed for this course is freely available, and I'll walk you through downloading and installing it.

Description

New! Completely updated and re-recorded for Spark 3, IntelliJ, Structured Streaming, and a stronger focus on the DataSet API.

“Big data" analysis is a hot and highly valuable skill – and this course will teach you the hottest technology in big data: Apache Spark. Employers including Amazon, EBay, NASA JPL, and Yahoo all use Spark to quickly extract meaning from massive data sets across a fault-tolerant Hadoop cluster. You'll learn those same techniques, using your own Windows system right at home. It's easier than you might think, and you'll be learning from an ex-engineer and senior manager from Amazon and IMDb.

Spark works best when using the Scala programming language, and this course includes a crash-course in Scala to get you up to speed quickly. For those more familiar with Python however, a Python version of this class is also available: "Taming Big Data with Apache Spark and Python - Hands On".

Learn and master the art of framing data analysis problems as Spark problems through over 20 hands-on examples, and then scale them up to run on cloud computing services in this course.

  • Learn the concepts of Spark's Resilient Distributed Datasets, DataFrames, and Datasets.

  • Get a crash course in the Scala programming language

  • Develop and run Spark jobs quickly using Scala, IntelliJ, and SBT

  • Translate complex analysis problems into iterative or multi-stage Spark scripts

  • Scale up to larger data sets using Amazon's Elastic MapReduce service

  • Understand how Hadoop YARN distributes Spark across computing clusters

  • Practice using other Spark technologies, like Spark SQL, DataFrames, DataSets, Spark Streaming, Machine Learning, and GraphX

By the end of this course, you'll be running code that analyzes gigabytes worth of information – in the cloud – in a matter of minutes. 

We'll have some fun along the way. You'll get warmed up with some simple examples of using Spark to analyze movie ratings data and text in a book. Once you've got the basics under your belt, we'll move to some more complex and interesting tasks. We'll use a million movie ratings to find movies that are similar to each other, and you might even discover some new movies you might like in the process! We'll analyze a social graph of superheroes, and learn who the most “popular" superhero is – and develop a system to find “degrees of separation" between superheroes. Are all Marvel superheroes within a few degrees of being connected to SpiderMan? You'll find the answer.

This course is very hands-on; you'll spend most of your time following along with the instructor as we write, analyze, and run real code together – both on your own system, and in the cloud using Amazon's Elastic MapReduce service. over 8 hours of video content is included, with over 20 real examples of increasing complexity you can build, run and study yourself. Move through them at your own pace, on your own schedule. The course wraps up with an overview of other Spark-based technologies, including Spark SQL, Spark Streaming, and GraphX.

Enroll now, and enjoy the course!


"I studied Spark for the first time using Frank's course "Apache Spark 2 with Scala - Hands On with Big Data!". It was a great starting point for me,  gaining knowledge in Scala and most importantly practical examples of Spark applications. It gave me an understanding of all the relevant Spark core concepts,  RDDs, Dataframes & Datasets, Spark Streaming, AWS EMR. Within a few months of completion, I used the knowledge gained from the course to propose in my current company to  work primarily on Spark applications. Since then I have continued to work with Spark. I would highly recommend any of Franks courses as he simplifies concepts well and his teaching manner is easy to follow and continue with!  " - Joey Faherty

Who this course is for:

  • Software engineers who want to expand their skills into the world of big data processing on a cluster
  • If you have no previous programming or scripting experience, you'll want to take an introductory programming course first.

Featured review

Leonid Medvedev
Leonid Medvedev
3 courses
2 reviews
Rating: 5.0 out of 58 months ago
Everything ok. I just specify what exactly really nice: - all examples clear and nice described; - nice pronunciation of the mentor; - material collected with right sequence; - guys who support the course react fast and concrete - it gives benefit in comparison with other courses. Suggest to increase salary for them :).

Instructors

Sundog Education by Frank Kane
Founder, Sundog Education. Machine Learning Pro
Sundog Education by Frank Kane
  • 4.5 Instructor Rating
  • 98,291 Reviews
  • 441,578 Students
  • 22 Courses

Sundog Education's mission is to make highly valuable career skills in big data, data science, and machine learning accessible to everyone in the world. Our consortium of expert instructors shares our knowledge in these emerging fields with you, at prices anyone can afford. 

Sundog Education is led by Frank Kane and owned by Frank's company, Sundog Software LLC. Frank spent 9 years at Amazon and IMDb, developing and managing the technology that automatically delivers product and movie recommendations to hundreds of millions of customers, all the time. Frank holds 17 issued patents in the fields of distributed computing, data mining, and machine learning. In 2012, Frank left to start his own successful company, Sundog Software, which focuses on virtual reality environment technology, and teaching others about big data analysis.

Due to our volume of students we are unable to respond to private messages; please post your questions within the Q&A of your course. Thanks for understanding.

Frank Kane
Founder, Sundog Education
Frank Kane
  • 4.5 Instructor Rating
  • 94,838 Reviews
  • 397,241 Students
  • 14 Courses

Frank spent 9 years at Amazon and IMDb, developing and managing the technology that automatically delivers product and movie recommendations to hundreds of millions of customers, all the time. Frank holds 17 issued patents in the fields of distributed computing, data mining, and machine learning. In 2012, Frank left to start his own successful company, Sundog Software, which focuses on virtual reality environment technology, and teaching others about big data analysis.

Due to our volume of students, I am unable to respond to private messages; please post your questions within the Q&A of your course. Thanks for understanding.

  • Udemy for Business
  • Teach on Udemy
  • Get the app
  • About us
  • Contact us
  • Careers
  • Blog
  • Help and Support
  • Affiliate
  • Terms
  • Privacy policy
  • Cookie settings
  • Sitemap
  • Featured courses
Udemy
© 2021 Udemy, Inc.