Udemy
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
Development
Web Development Data Science Mobile Development Programming Languages Game Development Database Design & Development Software Testing Software Engineering Development Tools No-Code Development
Business
Entrepreneurship Communications Management Sales Business Strategy Operations Project Management Business Law Business Analytics & Intelligence Human Resources Industry E-Commerce Media Real Estate Other Business
Finance & Accounting
Accounting & Bookkeeping Compliance Cryptocurrency & Blockchain Economics Finance Finance Cert & Exam Prep Financial Modeling & Analysis Investing & Trading Money Management Tools Taxes Other Finance & Accounting
IT & Software
IT Certification Network & Security Hardware Operating Systems Other IT & Software
Office Productivity
Microsoft Apple Google SAP Oracle Other Office Productivity
Personal Development
Personal Transformation Personal Productivity Leadership Career Development Parenting & Relationships Happiness Esoteric Practices Religion & Spirituality Personal Brand Building Creativity Influence Self Esteem & Confidence Stress Management Memory & Study Skills Motivation Other Personal Development
Design
Web Design Graphic Design & Illustration Design Tools User Experience Design Game Design Design Thinking 3D & Animation Fashion Design Architectural Design Interior Design Other Design
Marketing
Digital Marketing Search Engine Optimization Social Media Marketing Branding Marketing Fundamentals Marketing Analytics & Automation Public Relations Advertising Video & Mobile Marketing Content Marketing Growth Hacking Affiliate Marketing Product Marketing Other Marketing
Lifestyle
Arts & Crafts Beauty & Makeup Esoteric Practices Food & Beverage Gaming Home Improvement Pet Care & Training Travel Other Lifestyle
Photography & Video
Digital Photography Photography Portrait Photography Photography Tools Commercial Photography Video Design Other Photography & Video
Health & Fitness
Fitness General Health Sports Nutrition Yoga Mental Health Dieting Self Defense Safety & First Aid Dance Meditation Other Health & Fitness
Music
Instruments Music Production Music Fundamentals Vocal Music Techniques Music Software Other Music
Teaching & Academics
Engineering Humanities Math Science Online Education Social Science Language Teacher Training Test Prep Other Teaching & Academics
AWS Certification Microsoft Certification AWS Certified Solutions Architect - Associate AWS Certified Cloud Practitioner CompTIA A+ Cisco CCNA CompTIA Security+ Amazon AWS AWS Certified Developer - Associate
Photoshop Graphic Design Adobe Illustrator Drawing Digital Painting InDesign Character Design Canva Figure Drawing
Life Coach Training Neuro-Linguistic Programming Mindfulness Personal Development Personal Transformation Life Purpose Meditation Emotional Intelligence Neuroscience
Web Development JavaScript React CSS Angular PHP WordPress Node.Js Python
Google Flutter Android Development iOS Development Swift React Native Dart Programming Language Mobile Development Kotlin SwiftUI
Digital Marketing Google Ads (Adwords) Social Media Marketing Google Ads (AdWords) Certification Marketing Strategy Internet Marketing YouTube Marketing Email Marketing Google Analytics
SQL Microsoft Power BI Tableau Business Analysis Business Intelligence MySQL Data Analysis Data Modeling Data Science
Business Fundamentals Entrepreneurship Fundamentals Business Strategy Online Business Business Plan Startup Freelancing Blogging Home Business
Unity Game Development Fundamentals Unreal Engine C# 3D Game Development C++ 2D Game Development Unreal Engine Blueprints Blender
30-Day Money-Back Guarantee
Development Data Science Apache Spark

Apache Spark 3 - Spark Programming in Python for Beginners

Data Engineering using Spark Structured API
Rating: 4.6 out of 54.6 (818 ratings)
5,577 students
Created by Prashant Kumar Pandey, Learning Journal
Last updated 2/2021
English
English
30-Day Money-Back Guarantee

What you'll learn

  • Apache Spark Foundation and Spark Architecture
  • Data Engineering and Data Processing in Spark
  • Working with Data Sources and Sinks
  • Working with Data Frames and Spark SQL
  • Using PyCharm IDE for Spark Development and Debugging
  • Unit Testing, Managing Application Logs and Cluster Deployment
Curated for the Udemy for Business collection

Course content

10 sections • 60 lectures • 6h 33m total length

  • Preview05:51
  • Understanding the Data Lake Landscape
    06:42
  • Preview08:48
  • Check your knowledge
    15 questions

  • Preview02:52
  • Mac Users - Apache Spark in Local Mode Command Line REPL
    12:08
  • Windows Users - Apache Spark in Local Mode Command Line REPL
    05:49
  • Did you notice?
    3 questions
  • Mac Users - Apache Spark in the IDE - PyCharm
    07:59
  • Apache Spark in the IDE - PyCharm
    05:55
  • Did you notice?
    3 questions
  • Apache Spark in Cloud - Databricks Community and Notebooks
    04:33
  • Check your knowledge
    3 questions
  • Apache Spark in Anaconda - Jupyter Notebook
    04:32

  • Execution Methods - How to Run Spark Programs?
    05:01
  • Check your knowledge
    4 questions
  • Spark Distributed Processing Model - How your program runs?
    03:11
  • Spark Execution Modes and Cluster Managers
    04:55
  • Check your knowledge
    10 questions
  • Summarizing Spark Execution Models - When to use What?
    02:24
  • Working with PySpark Shell - Demo
    04:31
  • Installing Multi-Node Spark Cluster - Demo
    05:36
  • Working with Notebooks in Cluster - Demo
    06:58
  • Working with Spark Submit - Demo
    02:55
  • Section Summary
    01:42
  • Check your knowledge
    10 questions

  • Creating Spark Project Build Configuration
    06:10
  • Configuring Spark Project Application Logs
    10:50
  • Check your knowledge
    5 questions
  • Creating Spark Session
    08:26
  • Configuring Spark Session
    09:12
  • Check your knowledge
    5 questions
  • Data Frame Introduction
    07:43
  • Data Frame Partitions and Executors
    05:24
  • Spark Transformations and Actions
    11:02
  • Spark Jobs Stages and Task
    08:34
  • Understanding your Execution Plan
    09:33
  • Unit Testing Spark Application
    05:01
  • Rounding off Summary
    05:27

  • Introduction to Spark APIs
    05:11
  • Introduction to Spark RDD API
    13:13
  • Working with Spark SQL
    02:37
  • Spark SQL Engine and Catalyst Optimizer
    02:53
  • Section Summary
    01:18

  • Spark Data Sources and Sinks
    06:44
  • Spark DataFrameReader API
    05:00
  • Reading CSV, JSON and Parquet files
    07:59
  • Creating Spark DataFrame Schema
    06:06
  • Spark DataFrameWriter API
    06:09
  • Writing Your Data and Managing Layout
    12:51
  • Spark Databases and Tables
    05:33
  • Working with Spark SQL Tables
    08:41

  • Introduction to Data Transformation
    02:44
  • Working with Dataframe Rows
    05:02
  • DataFrame Rows and Unit Testing
    04:02
  • Dataframe Rows and Unstructured data
    06:08
  • Working with Dataframe Columns
    10:33
  • Creating and Using UDF
    10:01
  • Misc Transformations
    15:34

  • Aggregating Dataframes
    08:58
  • Grouping Aggregations
    04:25
  • Windowing Aggregations
    05:27

  • Dataframe Joins and column name ambiguity
    07:40
  • Outer Joins in Dataframe
    07:25
  • Internals of Spark Join and shuffle
    08:46
  • Optimizing your joins
    12:17
  • Implementing Bucket Joins
    08:57

  • Final Word
    00:50
  • Bonus Lecture : Get Extra
    00:23

Requirements

  • Programming Knowledge Using Python Programming Language
  • A Recent 64-bit Windows/Mac/Linux Machine with 8 GB RAM

Description

This course does not require any prior knowledge of Apache Spark or Hadoop. We have taken enough care to explain Spark Architecture and fundamental concepts to help you come up to speed and grasp the content of this course.


About the Course

I am creating Apache Spark 3 - Spark Programming in Python for Beginners course to help you understand the Spark programming and apply that knowledge to build data engineering solutions. This course is example-driven and follows a working session like approach. We will be taking a live coding approach and explain all the needed concepts along the way.

Who should take this Course?

I designed this course for software engineers willing to develop a Data Engineering pipeline and application using the Apache Spark. I am also creating this course for data architects and data engineers who are responsible for designing and building the organization’s data-centric infrastructure. Another group of people is the managers and architects who do not directly work with Spark implementation. Still, they work with the people who implement Apache Spark at the ground level.

Spark Version used in the Course

This Course is using the Apache Spark 3.x. I have tested all the source code and examples used in this Course on Apache Spark 3.0.0 open-source distribution.

Who this course is for:

  • Software Engineers and Architects who are willing to design and develop a Bigdata Engineering Projects using Apache Spark
  • Programmers and developers who are aspiring to grow and learn Data Engineering using Apache Spark

Featured review

PyDoo ?
PyDoo ?
368 courses
100 reviews
Rating: 5.0 out of 57 months ago
Amazing course on PySpark (5/5). Highly recommended. My personal request to the instructor to add lectures on "PySpark streaming" (or) to provide it as a separate course. In most of the companies, they use pyspark as opposed to spark-scala, because python is easy to learn. Hence requesting the instructor to come up with spark streaming using python Really thankful to the instructor for this course.

Instructors

Prashant Kumar Pandey
Architect, Author, Consultant, Trainer @ Learning Journal
Prashant Kumar Pandey
  • 4.6 Instructor Rating
  • 3,890 Reviews
  • 31,898 Students
  • 8 Courses

Prashant Kumar Pandey is passionate about helping people to learn and grow in their career by bridging the gap between their existing and required skills. In his quest to fulfill this mission, he is authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry.

With over 18 years of experience in IT as a developer, architect, consultant, trainer, and mentor, he has worked with international software services organizations on various data-centric and Bigdata projects.

Prashant is a firm believer in lifelong continuous learning and skill development. To popularize the importance of lifelong continuous learning, he started publishing free training videos on his YouTube channel and conceptualized the idea of creating a Journal of his learning under the banner of Learning Journal.

He is the founder, lead author, and chief editor of the Learning Journal portal that offers various skill development courses, training, and technical articles since the beginning of the year 2018.

Learning Journal
Online Training Company
Learning Journal
  • 4.6 Instructor Rating
  • 3,890 Reviews
  • 31,898 Students
  • 8 Courses

Learning Journal is a small team of people passionate about helping others learn and grow in their careers by bridging the gap between their existing and required skills. In our quest to fulfill this mission, we are authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry.

Together we have over 40+ years of experience in IT as a developer, architect, consultant, trainer, and mentor. We have worked with international software services organizations on various data-centric and Bigdata projects.

Learning Journal is a team of firm believers in lifelong continuous learning and skill development. To popularize the importance of lifelong continuous learning, we started publishing free training videos on our YouTube channel. We conceptualized the notion of continuous learning, creating a journal of our learning under the Learning Journal banner.

We authored various skill development courses, training, and technical articles since the beginning of the year 2018.

  • Udemy for Business
  • Teach on Udemy
  • Get the app
  • About us
  • Contact us
  • Careers
  • Blog
  • Help and Support
  • Affiliate
  • Terms
  • Privacy policy
  • Cookie settings
  • Sitemap
  • Featured courses
Udemy
© 2021 Udemy, Inc.