Udemy
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
Development
Web Development Data Science Mobile Development Programming Languages Game Development Database Design & Development Software Testing Software Engineering Development Tools No-Code Development
Business
Entrepreneurship Communications Management Sales Business Strategy Operations Project Management Business Law Business Analytics & Intelligence Human Resources Industry E-Commerce Media Real Estate Other Business
Finance & Accounting
Accounting & Bookkeeping Compliance Cryptocurrency & Blockchain Economics Finance Finance Cert & Exam Prep Financial Modeling & Analysis Investing & Trading Money Management Tools Taxes Other Finance & Accounting
IT & Software
IT Certification Network & Security Hardware Operating Systems Other IT & Software
Office Productivity
Microsoft Apple Google SAP Oracle Other Office Productivity
Personal Development
Personal Transformation Personal Productivity Leadership Career Development Parenting & Relationships Happiness Esoteric Practices Religion & Spirituality Personal Brand Building Creativity Influence Self Esteem & Confidence Stress Management Memory & Study Skills Motivation Other Personal Development
Design
Web Design Graphic Design & Illustration Design Tools User Experience Design Game Design Design Thinking 3D & Animation Fashion Design Architectural Design Interior Design Other Design
Marketing
Digital Marketing Search Engine Optimization Social Media Marketing Branding Marketing Fundamentals Marketing Analytics & Automation Public Relations Advertising Video & Mobile Marketing Content Marketing Growth Hacking Affiliate Marketing Product Marketing Other Marketing
Lifestyle
Arts & Crafts Beauty & Makeup Esoteric Practices Food & Beverage Gaming Home Improvement Pet Care & Training Travel Other Lifestyle
Photography & Video
Digital Photography Photography Portrait Photography Photography Tools Commercial Photography Video Design Other Photography & Video
Health & Fitness
Fitness General Health Sports Nutrition Yoga Mental Health Dieting Self Defense Safety & First Aid Dance Meditation Other Health & Fitness
Music
Instruments Music Production Music Fundamentals Vocal Music Techniques Music Software Other Music
Teaching & Academics
Engineering Humanities Math Science Online Education Social Science Language Teacher Training Test Prep Other Teaching & Academics
AWS Certification Microsoft Certification AWS Certified Solutions Architect - Associate AWS Certified Cloud Practitioner CompTIA A+ Cisco CCNA Amazon AWS CompTIA Security+ AWS Certified Developer - Associate
Graphic Design Photoshop Adobe Illustrator Drawing Digital Painting InDesign Character Design Canva Figure Drawing
Life Coach Training Neuro-Linguistic Programming Mindfulness Personal Development Meditation Personal Transformation Life Purpose Coaching Neuroscience
Web Development JavaScript React CSS Angular PHP WordPress Node.Js Python
Google Flutter Android Development iOS Development Swift React Native Dart Programming Language Mobile Development Kotlin SwiftUI
Digital Marketing Google Ads (Adwords) Social Media Marketing Google Ads (AdWords) Certification Marketing Strategy Internet Marketing YouTube Marketing Email Marketing Retargeting
SQL Microsoft Power BI Tableau Business Analysis Business Intelligence MySQL Data Analysis Data Modeling Big Data
Business Fundamentals Entrepreneurship Fundamentals Business Strategy Online Business Business Plan Startup Freelancing Blogging Home Business
Unity Game Development Fundamentals Unreal Engine C# 3D Game Development C++ 2D Game Development Unreal Engine Blueprints Blender
30-Day Money-Back Guarantee

This course includes:

  • 4.5 hours on-demand video
  • 1 article
  • Full lifetime access
  • Access on mobile and TV
Development Data Science Apache Spark

Apache Spark 3 - Real-time Stream Processing using Scala

Learn to create Real-time Stream Processing applications using Apache Spark
Rating: 4.6 out of 54.6 (45 ratings)
4,268 students
Created by Prashant Kumar Pandey, Learning Journal
Last updated 1/2021
English
English
30-Day Money-Back Guarantee

What you'll learn

  • Real-time Stream Processing Concepts
  • Spark Structured Streaming APIs and Architecture
  • Working with File Streams
  • Working With Kafka Source and Integrating Spark with Kafka
  • State-less and State-full Streaming Transformations
  • Windowing Aggregates using Spark Stream
  • Watermarking and State Cleanup
  • Streaming Joins and Aggregation
  • Handling Memory Problems with Streaming Joins
  • Creating Arbitrary Streaming Sinks

Course content

7 sections • 32 lectures • 4h 17m total length

  • Preview02:58
  • Preview01:19

  • Preview01:44
  • Preview05:25
  • Preview08:26
  • Setup and test your IDE
    06:44
  • Install and run Apache Kafka
    10:24

  • Introduction to Stream Processing
    09:16
  • Spark Streaming APIs - DStream Vs Structured Streaming
    03:50
  • Creating your first stream processing application
    17:40
  • Stream processing model in Spark
    08:34
  • Working with Files and Directories
    11:43
  • Streaming Sources, Sinks and Output Mode
    13:05
  • Fault Tolerance and Restarts
    06:29

  • Streaming from Kafka Source
    16:19
  • Working with Kafka Sinks
    10:32
  • Multi-query Streams Application
    04:13
  • Kafka Serialization and Deserialization for Spark
    05:00
  • Creating Kafka AVRO Sinks
    04:00
  • Working with Kafka AVRO Source
    04:52

  • Stateless Vs Statefull transformations
    10:18
  • Event time and Windowing
    07:14
  • Tumbling Window aggregate
    14:05
  • Watermarking your windows
    12:37
  • Watermark and output modes
    09:54
  • Sliding Window
    07:38

  • Joining Stream to static source
    13:57
  • Joining Stream to another Stream
    09:57
  • Streaming Watermark
    07:17
  • Streaming Outer Joins
    11:17

  • Final Word
    00:50
  • Bonus Lecture : Get Extra
    00:22

Requirements

  • Spark Fundamentals and exposure to Spark Dataframe APIs
  • Kafka Fundamentals and working knowledge of Apache Kafka
  • Programming Knowledge Using Scala Programming Language
  • A Recent 64-bit Windows/Mac/Linux Machine with 8 GB RAM

Description

About the Course

I am creating Apache Spark 3 - Real-time Stream Processing using the Scala course to help you understand the Real-time Stream processing using Apache Spark and apply that knowledge to build real-time stream processing solutions. This course is example-driven and follows a working session like approach. We will be taking a live coding approach and explain all the needed concepts along the way.

Who should take this Course?

I designed this course for software engineers willing to develop a Real-time Stream Processing Pipeline and application using the Apache Spark. I am also creating this course for data architects and data engineers who are responsible for designing and building the organization’s data-centric infrastructure. Another group of people is the managers and architects who do not directly work with Spark implementation. Still, they work with the people who implement Apache Spark at the ground level.

Spark Version used in the Course

This Course is using the Apache Spark 3.x. I have tested all the source code and examples used in this Course on Apache Spark 3.0.0 open-source distribution.

Who this course is for:

  • Software Engineers and Architects who are willing to design and develop a Bigdata Engineering Projects using Apache Spark
  • Programmers and developers who are aspiring to grow and learn Data Engineering using Apache Spark

Instructors

Prashant Kumar Pandey
Architect, Author, Consultant, Trainer @ Learning Journal
Prashant Kumar Pandey
  • 4.6 Instructor Rating
  • 3,492 Reviews
  • 30,115 Students
  • 7 Courses

Prashant Kumar Pandey is passionate about helping people to learn and grow in their career by bridging the gap between their existing and required skills. In his quest to fulfill this mission, he is authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry.

With over 18 years of experience in IT as a developer, architect, consultant, trainer, and mentor, he has worked with international software services organizations on various data-centric and Bigdata projects.

Prashant is a firm believer in lifelong continuous learning and skill development. To popularize the importance of lifelong continuous learning, he started publishing free training videos on his YouTube channel and conceptualized the idea of creating a Journal of his learning under the banner of Learning Journal.

He is the founder, lead author, and chief editor of the Learning Journal portal that offers various skill development courses, training, and technical articles since the beginning of the year 2018.

Learning Journal
Online Training Company
Learning Journal
  • 4.6 Instructor Rating
  • 3,481 Reviews
  • 30,115 Students
  • 7 Courses

Learning Journal is a small team of people passionate about helping others learn and grow in their careers by bridging the gap between their existing and required skills. In our quest to fulfill this mission, we are authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry.

Together we have over 40+ years of experience in IT as a developer, architect, consultant, trainer, and mentor. We have worked with international software services organizations on various data-centric and Bigdata projects.

Learning Journal is a team of firm believers in lifelong continuous learning and skill development. To popularize the importance of lifelong continuous learning, we started publishing free training videos on our YouTube channel. We conceptualized the notion of continuous learning, creating a journal of our learning under the Learning Journal banner.

We authored various skill development courses, training, and technical articles since the beginning of the year 2018.

  • Udemy for Business
  • Teach on Udemy
  • Get the app
  • About us
  • Contact us
  • Careers
  • Blog
  • Help and Support
  • Affiliate
  • Terms
  • Privacy policy
  • Cookie settings
  • Sitemap
  • Featured courses
Udemy
© 2021 Udemy, Inc.