Udemy
    •  
    •  
    •  
    •  
    •  
    •  
    •  
    •  
Subscribe
Turn what you know into an opportunity and reach millions around the world.
Learn More
Your cart is empty.
Keep shopping
Pyspark Foundation for Data Engineering | Beginners
Rating: 4.3 out of 5(72 ratings)
815 students

Pyspark Foundation for Data Engineering | Beginners

Data Engineering, PySpark, Coding exercise
Created byAkash Sunil Pawar
Last updated 1/2025
English

What you'll learn

  • Fundamentals of PySpark
  • Hands on experience in PySpark
  • Understanding of data using PySpark
  • Performing various operations on DataFrame

Course content

1 section • 24 lectures • 1h 13m total length

  • Introduction2:20
  • SparkSession and Imports3:12

    Launch your PySpark foundation by creating a SparkSession, importing essential libraries, and using basic functions to read dataframes and add columns.

  • Spark DataFrame and its characteristics2:01

    Explore the spark data frame as a two-dimensional, table-like structure with columns and rows where each column holds values for a variable and each row a record.

  • Syntax and example7:10
  • Print operation0:44

    Explore the print operation in PySpark data engineering by observing how code output appears, how newly created data is printed, and how functional print behavior translates to visible results.

  • Understanding the data0:19
  • Number of records in DataFrame0:27
  • Columns present in DataFrame0:30

    Learn how to inspect a pyspark dataframe's structure by using the columns function to reveal all the columns in your data.

  • Summary of a DataFrame1:03
  • Get schema of a DataFrame0:53
  • Create a new column in a dataframe4:53
  • Arithmetic operations on columns5:27
  • Change column Data Types by casting4:19
  • Create a column with integer constant1:52

    Create and manipulate a PySpark DataFrame by adding a new column with an integer constant (for example, 300), explore arithmetic and string constant columns, and view the results.

  • Application of the learnings1:31
  • Rounding operations using bround3:46

    Use bround to round numeric columns to integers or decimals, applying banker’s rounding to the nearest even number and controlling decimal places in PySpark data frames.

  • Sorting operation5:24

    Explore how to sort a pyspark data frame using ascending and descending order, compare different sort methods and the order by syntax, and understand how to apply sorting on columns.

  • Drop columns of a dataframe4:51

    Drop single or multiple columns in a dataframe to create a new dataframe excluding total score and percentage columns, using the drop operation and handling missing columns.

  • Rename a column3:24
  • Create a column with String constant1:14

    Create a column with constant string values to label performers as excellent, very good, good, or average in a PySpark data frame.

  • Conditional Statements4:46
  • Changing Case of a column1:49

    Learn how to change a column's case in PySpark using the upper and lower functions, with practical examples converting values to lowercase and uppercase.

  • Filter operations3:24
  • Grouping and Aggegrations8:27

    Learn grouping and aggregations in PySpark: group by grade, count students, compute mean and max, handle distinct values, and rename or create columns based on conditions.

Requirements

  • There are no pre-requisites for the course. We will learn and practice together.
  • Basic Python knowledge is a plus

Description

This course will prepare you for a real world Data Engineer role (basics)!

Learn to code PySpark like a real world developer. Here our major focus will be on Practical applications of PySpark and bridge the gap between academic knowledge and practical skill.

In this course we will get to know and apply few of the most essential and basic functions in PySpark, that are used frequently in scripting for any project based on PySpark.


About PySpark:

Learn the latest Big Data Technology - Spark! And learn to use it with one of the most popular programming languages, Python!

One of the most valuable technology skills is the ability to analyze huge data sets, and this course is specifically designed to bring you up to speed on one of the best technologies for this task, Apache Spark! The top technology companies like Google, Facebook, Netflix, Airbnb, Amazon, NASA, and more are all using Spark to solve their big data problems!

Spark can perform up to 100x faster than Hadoop MapReduce, which has caused an explosion in demand for this skill! Because the Spark 2.0 DataFrame framework is so new, you now have the ability to quickly become one of the most knowledgeable people in the job market!


What you will learn :

  • SparkSession and imports

  • Spark DataFrame and its characteristics

  • Syntax and example

  • Print results

  • Understanding the data

  • Number of records

  • Columns in dataFrame

  • Describe a DataFrame

  • Schema of a DataFrame

  • Create a new column

  • Arithmetic operations on Data

  • Change column data type

  • Create a column with integer as constant

  • Apply what we know

  • Rounding of digits

  • Sorting operation

  • Drop columns

  • Rename columns

  • Create a column with string as constant

  • Conditional Statements

  • Changing case of a column

  • Filter operations

  • Grouping and aggregations


Prerequisites :

  • Some basic programming skills (Not Mandatory)

  • Will to implement theoretical knowledge in practical.


Who this course is for:

  • Beginners who want to learn Big Data or experienced people who want to transition to a Big Data role

  • Big data beginners who want to learn how to code in the real world

  • Aspiring candidates for data engineering role

Who this course is for:

  • Anyone with interest in Data engineering

Report abuse