TopDataEngineerVoice

300+ PySpark Scenarios Based Interview Questions & Answers Preparation Practice Test | Freshers to Experienced | Detailed Explanations !!

Welcome to the ultimate PySpark Interview Questions Practice Test course! If you're gearing up for a job interview that demands PySpark knowledge, or if you want to strengthen your understanding of PySpark concepts and build confidence before tackling real interview situations, you're in the right place! This all-inclusive practice test course is crafted to help you master PySpark and excel in your interviews with confidence.

As PySpark continues to rise in popularity within the world of big data processing and analysis, gaining a strong grasp of its concepts is essential for those aiming for roles in data engineering, data science, or analytics. This course is divided into five key sections, each thoughtfully designed to cover a comprehensive range of PySpark topics.

1. PySpark Basic Functions: This section covers the fundamentals of PySpark functions, featuring 100+ functions, each explained with examples and detailed scenarios to help you understand the core concepts of these functions.

2. Advanced PySpark Concepts: Take your PySpark skills to the next level with advanced topics such as UDFs, window functions, broadcast joins, integration.

3. PySpark Medium-Level Questions: This section covers Interview Round 1 questions and answers, explained in detail. These questions are designed to help you build speed and confidence in tackling advanced-level interview questions.

4. PySpark Advanced-Level Questions: This section covers advanced-level Round 2 questions and answers, explained in detail. These questions are designed to help you build speed and confidence in tackling higher-level interview questions.

5. Key Points of Questions: This section provides a breakdown of interview questions collected from various companies in the IT industry. Each question is accompanied by details on when and where it was asked, the year it was posed, and which company asked it during live interviews

Here are demonstration of interview- Question & In details examplanation along with Expected Dataframe results:

Question 1: [Persistent Technology -2025] :

You are given a dataset('sampledata') containing a column with date values in the format 'yyyy-MM-dd HH:mm:ss'.  Your task is to calculate how many days remain from each date until the end of that year (December 31st).

For demonstration - Lets suppose your date input is this '2025-01-01', and you should write a script to calculate ,How many days are left in the year 2025 after January 1st"

#sampledata

sampledata  = [('2025-01-10 12:10:00'),('2025-02-10 12:10:00'),('2025-04-10 12:10:00')]

Solution & Explanation :

# Imports required library :

import pyspark

from pyspark.sql import SparkSession

from pyspark.sql.functions import*

from pyspark.sql.types import*

from pyspark.sql import functions as f

# Initialize Spark Session :

spark = SparkSession.builder.appName('udemy').master('local[1]').getOrCreate()

# Step 1 :  To create a dataframe with given sample data as :

# Define the schema for dataframe as:

schema =  ['dateColumn']

dataframe = spark.createDataFrame(data = sampledata , schema = schema)

# Show the dataframe as loaded.

display(dataframe)

# Step 2 : To convert dateColumn to date format as required fur transformation :

dataframeDate = dataframe.withColumn('dateColumn' , to_date(col('dateColumn'),'yyyy-MM-dd HH:mm:ss'))

# Show the dataframe as loaded.

display(dataframeDate)

# Step 4 : To add a new column with the last month date ('31-12') and extract year for column as :

dataframeAddate = dataframeDate.withColumn('YearOfEnd',to_date(concat(lit('31-12-'), year(col('dateColumn')).cast('string')), 'dd-MM-yyyy'))

# Show the dataframe as loaded.

display(dataframeAddate)

# Step 5 : To applying the datediff function to get total days which are left as:

dataframeFinal = dataframeAdddate.withColumn('DaysLeft', datediff('YearOfEnd' , 'dateColumn'))

# Show final the dataframe as expected as:

display(dataframeFinal)

Question 2 : [LTIMindTree-2025] :

Imagine you are analyzing the monthly sales performance records of retail company across multiple regions. I would like to ask you to perform the task as given in poits :

1. To calcaulte the cumulative sales for each region over months.

2. To generate the rank of each month based on sales within the same region.

#sampledata

sampledata = [ ("East", "March", 400), ("East", "April", 300), ("East", "May", 650), ("West", "June", 900), ("West", "July", 370), ("West", "August", 850)]

Solution & Explanation :

# Imports required library :

import pyspark

from pyspark.sql import SparkSession

from pyspark.sql.functions import*

from pyspark.sql.types import*

from pyspark.sql import functions as f

# Initialize Spark Session :

spark = SparkSession.builder.appName('udemy').master('local[1]').getOrCreate()

# Step 1 :  To create a dataframe with given sample data as :

# Define the schema for dataframe as:

schema = ["Region", "Month", "Sales"]

dataframe = spark.createDataFrame(data = sampledata , schema = schema)

# Show the dataframe as loaded

display(dataframe)

# Step 2 : To applying window function get partiton on columns records as :

windowSp = Window.partitionBy('Region').orderBy('Sales')

# Step 3 :  This window function being applying for Rank of desc ordering as :

WindowRank = Window.partitionBy('Region').orderBy(f.desc('Sales'))

# Step 4 : To calculate the cumulative sum & Rank on columns records as:

dataframeFinal = dataframe.withColumn('CumulativeSales',f.sum('Sales').\

over(windowSp)).withColumn('CumulativeRank',f.rank().over(WindowRank))

# Show final the dataframe as expected as:

display(dataframeFinal)

Question 3 : [PWC -2025] :

Assume that as you are working with Ingestion team , However , how would you validate the data between a source and target dataset using PySpark ? Specifically, how would you handle the comparison of records in terms of matching values, missing data, and identifying any discrepancies between the two datasets?

#sampledata

sampledataSource = [(100,'M'),(200,'N'),(300,'O'),(400,'P'),(500,'Q')]

sampledataTarget = [(100,'M'),(200,'N'),(300,'Z'),(400,'X'),(500,'Y')]

Solution & Explanation :

# Imports required library :

import pyspark

from pyspark.sql import SparkSession

from pyspark.sql.functions import*

from pyspark.sql.types import*

from pyspark.sql import functions as f

# Initialize Spark Session :

spark = SparkSession.builder.appName('udemy').master('local[1]').getOrCreate()

# Step 1 :  Create a dataframe with 'sampledataSource' &  'sampledataTarget' as:

dataframe = spark.createDataFrame(data = sampledataSource , schema = ['SourceId','ProductName'])

display(dataframe)

dataframeTarget = spark.createDataFrame(data = sampledataTarget , schema = ['ProductId','ProductName'])

display(dataframeTarget)

# Step 2 : To Join the both dataframe as like source & Target as :

dataframeJoin = dataframe.alias('Table1').join(dataframeTarget.alias('Table2'),on = col('Table1.SourceId') == col('Table2.ProductId'),how='full')

display(dataframeJoin)

# Step 3 :  To use alias for column naming as readable as:

dataframeAlias (dataframeJoin).select(col('Table1.SourceId').alias('SourceId'),col('Table2.ProductId').alias('TargetId'),col('Table1.ProductName').alias('SourecName'),col('Table2.ProductName').alias('TargetName'))

display(dataframeAlias)

# Step 4 : To Add a new columns as 'Mismatched' and compare each records as :

dataframeMismatch = dataframeAlias.withColumn(

    'Mismatched',

    when((col('SourceId') == col('TargetId')) & (col('SourecName') != col('TargetName')), 'Mismatched')

    .when(col('TargetId').isNull(), 'NewRecords in source')

    .when(col('SourceId').isNull(), 'New records in Target Table')

    .otherwise('No mismatch'))

display(dataframeMismatch)

# Step 5 :  To filter missing(value) in columns as :

dataframeFilter = dataframeMismatch.filter(col('Mismatched').isNotNull())

display(dataframeFilter)