Apache Spark Interview Question and Answer (100 FAQ)

Retrieve the SparkContext application id at runtime, or via a Spark listener, Spark UI, and logs for tracking and debugging.

Compare map and flatMap in Spark: map yields one output per input, while flatMap returns zero, one, or many outputs. Use cases include tokenizing text and flattening nested data.

Explain how the sort by key transformation orders key-value RDDs by keys, with ascending or descending options, and that it triggers a shuffle only on pair RDDs.

Control the number of partitions in an RDD to balance workload and optimize Spark performance by setting partition counts at creation or after with repartition or coalesce.

Explore Spark SQL subqueries, including table-derived and scalar types, with key limitations and practical workarounds for efficient query execution.

Learn to concatenate columns in an Apache Spark dataframe using Scala with concat and concat_ws, handling nulls and separators to create full names or composite keys.

Spark dataframes do not enforce primary keys like RDBMS. Simulate uniqueness with a generated id (zip or monotonically_increasing_id), and verify with distinct counts, or use storage layers offering constraints.

Explore the various levels of persistence in Spark, including memory only, memory and disk, and serialization, to reuse RDDs or DataFrames across multiple actions and reduce recomputation.

Explore the difference between cache and persist in Spark RDDs, learning when to keep data in memory versus using configurable storage levels for memory, disk, and serialization.

Understand shuffling in Apache Spark, the redistribution across partitions needed for group by, reduce by key, join, and sort, and why it costs disk I/O and network transfer.

Evaluate a Spark application by tracking execution time, memory and CPU usage, and shuffle performance, using Spark UI and event logs with tools like Ganglia, Grafana, Prometheus, and CloudWatch.

Identify data skew in Apache Spark joins and apply techniques such as broadcast joins, salting, repartitioning, adaptive skew joins, and map-side joins to improve performance.

Catalyst optimizer, Spark SQL's query optimization engine, analyzes SQL and data frame code, converts it into an optimized execution plan, and applies cost-based rules like predicate pushdown and projection pruning.

Understand what stage skipped means in the Spark web UI and why it can be a good thing, driven by caching, persistence, and reusable shuffle outputs.

Disable Spark info logs by setting the log level to warning in code, editing log4j properties, or using cli options; choose level based on learning versus production needs.

Explore how Spark SQL query execution moves from unresolved logical plan to resolved logical plan, through the Catalyst optimizer, to a physical plan and whole stage code generation, then execution.

Learn how receivers in Spark Streaming collect data from sources like Kafka, Flume, or Socket, serve as the streaming entry point, and enable micro-batch processing.

Explore the dstream concept, core API of Spark streaming, as Spark processes continuous data into micro-batches of RDD for scalable, fault-tolerant processing from sources like Kafka, Flume, and Kinesis.

Empowers data engineers to build scalable pipelines and models with MLlib, Apache Spark’s distributed machine learning library. It covers classification, regression, clustering, and recommendations, plus feature engineering and evaluation tools.