Apache Spark Interview Question and Answer (100 FAQ)

Use client mode for development and interactive analysis to access logs. Use cluster mode for production jobs with the driver inside the cluster for fault tolerance and automatic retries.

Install spark only on the edge node from which you submit, not on every Yarn cluster node, as Yarn distributes libraries to executors.

Learn practical methods to stop a running spark application, including canceling a specific or all jobs via spark context cancelJob, yarn commands, UI, or spark.stop.

Learn how to limit retries for Spark jobs on Yarn to prevent overload and long runtimes. Configure spark.task.maxFailure to cap per-task retries and yarn.resourceManager.maxAttempts to cap application relaunches.

Retrieve the SparkContext application id at runtime, or via a Spark listener, Spark UI, and logs for tracking and debugging.

View spark logs on YARN: they reside in local YARN log directory on each node; aggregation copies them to HDFS for viewing via the YARN UI or yarn log -application_id.

Learn how to prevent Spark executors from getting lost in yarn client mode by tuning network timeout and heartbeat settings, improving reliability, and preferring cluster mode for stability.

Mount points in Databricks bridge workspace and external storage, enabling read and write via simple dbfs paths. They simplify data access, centralize credentials, and standardize paths across notebooks.

Learn to run spark on a cluster by understanding cluster modes, deployment options, and cluster managers like standalone, Yarn, Mesos, and Kubernetes, using spark-submit and monitoring jobs.

Define RDD as a resilient distributed data set, an immutable, fault-tolerant collection. Note its lazy transformations and actions, and creation from external data or parallelized collections with in-memory caching.

Explain transformations and actions in Apache Spark's RDDs, highlighting lazy evaluation, lineage graphs, and common operations like map, filter, and count.

Describe how lazy evaluation defers RDD transformations, builds a DAG of map and filter, and executes only when an action like count is called, enabling optimization and fault tolerance.

When a node hosting an RDD partition fails, Spark automatically recomputes the lost partition using its lineage, or retrieves it from replicated storage if persistence is enabled.

Compare map and flatMap in Spark: map yields one output per input, while flatMap returns zero, one, or many outputs. Use cases include tokenizing text and flattening nested data.

View and print an RDD by using actions like collect, take, or take sample, while noting RDDs are distributed and printing triggers Spark jobs; avoid large prints in production.

Learn how to read multiple text files into a single RDD in Spark using sc.textFile with directory paths or wildcards, and when to use wholeTextFile for file-level metadata.

Explain how the sort by key transformation orders key-value RDDs by keys, with ascending or descending options, and that it triggers a shuffle only on pair RDDs.

Control the number of partitions in an RDD to balance workload and optimize Spark performance by setting partition counts at creation or after with repartition or coalesce.

Discover that a data frame in Apache Spark is a distributed collection across multiple nodes, schema-based with named columns, optimized by the Catalyst Optimizer and evaluated lazily.

Explore how data frames provide a high level abstraction and optimized performance via catalyst and tungsten, with SQL compatibility and multi source, multi language support.

Explore Spark SQL, a unified, in-memory engine for structured data with Catalyst optimizer and Tungsten execution. Compare it with Hive's MapReduce batch processing.

Spark SQL reads from data sources—file-based formats (Parquet, JSON, CSV, Avro, text), JDBC databases, Hive tables, streaming sources, and cloud storage—using the unified DataFrame API to load, query, and write.

Explore Spark SQL subqueries, including table-derived and scalar types, with key limitations and practical workarounds for efficient query execution.

Change a column's data type in Spark SQL data frames using cast or SQL. Create a new data frame for the updated column and verify with print schema.

Learn to replace null values in Spark data frame using na.fill, na.replace, and coalesce, with practical examples for cleaning data before analysis and modeling.

Learn how to add a constant column to a spark dataframe using scala, with column and lit to set a fixed value like country India, and verify results.

Learn how to add an index column to a Spark dataframe using Scala, with two approaches: monotonically_increasing_id for a unique id and zipWithIndex for ordered numbering, including RDD conversion.

Learn to concatenate columns in an Apache Spark dataframe using Scala with concat and concat_ws, handling nulls and separators to create full names or composite keys.

Spark dataframes do not enforce primary keys like RDBMS. Simulate uniqueness with a generated id (zip or monotonically_increasing_id), and verify with distinct counts, or use storage layers offering constraints.

Parquet's columnar storage speeds analytics by reading only the required columns, while advanced compression and pushdown boost performance and reduce storage; it also supports schema evolution across the Hadoop ecosystem.

Spark partitions HDFS files by block boundaries, with one HDFS block roughly equal to one Spark partition, enabling data locality, parallelism, and performance tuning through repartition or directory-based partitioning.

Configure spark to compress output when writing to hdfs in standalone mode, using snappy for parquet and gzip for csv or json, with global settings in spark-default.conf.

Partitioning splits data into directories by column values, enabling partition pruning for low cardinality data. Bucketing distributes data into fixed buckets using hashing to improve joins on high cardinality columns.

Learn to join a 100 GB table with a 1 GB table in Spark by broadcasting the smaller dataset and using bucketing, repartitioning, and skew handling to reduce shuffles.

Boost spark performance with best practices in serialization, data formats, partitioning, caching, tuning, data skew handling, and broadcast joins, including parquet or ORC with column pruning and predicate pushdown.

Explore the various levels of persistence in Spark, including memory only, memory and disk, and serialization, to reuse RDDs or DataFrames across multiple actions and reduce recomputation.

Explore the difference between cache and persist in Spark RDDs, learning when to keep data in memory versus using configurable storage levels for memory, disk, and serialization.

The coalesce transformation in Apache Spark reduces partitions without shuffle, enabling efficient output writes; it cannot increase partitions and may cause uneven partition sizes.

Understand shuffling in Apache Spark, the redistribution across partitions needed for group by, reduce by key, join, and sort, and why it costs disk I/O and network transfer.

Explore spark speculative execution, duplicating slow straggler tasks to finish first and reduce stage completion time; learn when to enable it and essential caveats.

Evaluate a Spark application by tracking execution time, memory and CPU usage, and shuffle performance, using Spark UI and event logs with tools like Ganglia, Grafana, Prometheus, and CloudWatch.

Learn how to diagnose and fix spark java out-of-memory errors during shuffle and wide transformations by tuning executor and driver memory, increasing memory overhead, and repartitioning to prevent data skew.

Identify data skew in Apache Spark joins and apply techniques such as broadcast joins, salting, repartitioning, adaptive skew joins, and map-side joins to improve performance.

Discover the top five Apache Spark performance secrets: cryo serialization, minimizing shuffle, broadcast join, strategic caching, and intelligent repartitioning to run faster, cheaper, and more stable.

Catalyst optimizer, Spark SQL's query optimization engine, analyzes SQL and data frame code, converts it into an optimized execution plan, and applies cost-based rules like predicate pushdown and projection pruning.

Discover how the Tungsten project boosts Apache Spark performance by using off-heap memory, cache-friendly data formats, and whole stage code generation to reduce garbage collection and speed up queries.

Explains how whole stage code generation in Spark SQL fuses filters, projections, hashes, and aggregates into a single runtime Java bytecode loop to boost performance 3–10x and reduce overhead.

Understand the difference between group by key and reduce by key in Spark RDDs, focusing on shuffle behavior, map-side combine, and efficient aggregation strategies.

Minimize data transfers in Spark by using reduce by key, map-side pre-aggregation, and strategic partitioning; broadcast variables and joins reduce shuffles, while cache and avoid wide transformations boost performance.

Discover how broadcasting values across a spark cluster avoids shuffle during joins, speeds execution, and reduces network overhead by caching small lookup data on each executor.

Learn how a broadcast join in Spark sends a small table to every executor, enabling a map-side join and avoiding shuffle, with guidance on when to use it.

Understand how Spark sets default parallelism from cluster cpu cores or local cores, inspect it with SparkContext.defaultParallelism, and adjust it to improve concurrency and performance.

Explore how to monitor and troubleshoot Spark UI, navigating jobs, stages, tasks, and executors to identify bottlenecks such as shuffles and data skew, and apply broadcast joins.

Understand what stage skipped means in the Spark web UI and why it can be a good thing, driven by caching, persistence, and reusable shuffle outputs.

Disable Spark info logs by setting the log level to warning in code, editing log4j properties, or using cli options; choose level based on learning versus production needs.

Explore how Spark SQL query execution moves from unresolved logical plan to resolved logical plan, through the Catalyst optimizer, to a physical plan and whole stage code generation, then execution.

Learn how Apache Spark Streaming enables real-time data processing through micro-batches. Ingest data from Kafka, Flume, Kinesis, and more, using DStream RDD operations for fault-tolerant pipelines.

Discover how Spark streaming processes real-time data through micro batches, using receivers and backpressure for fault-tolerant, stateful computations with update state by key and window transforms.

Learn how receivers in Spark Streaming collect data from sources like Kafka, Flume, or Socket, serve as the streaming entry point, and enable micro-batch processing.

Explore the dstream concept, core API of Spark streaming, as Spark processes continuous data into micro-batches of RDD for scalable, fault-tolerant processing from sources like Kafka, Flume, and Kinesis.

Explore sliding window in Spark streaming to perform time-based aggregations on the last 30 seconds of data every 10 seconds, with overlapping data for real-time analytics.

Enable the write-ahead log in Spark streaming to record incoming data to a durable store before processing, ensuring fault-tolerant recovery. This allows deterministic replay and restart-safe processing.

Discover structured streaming, Spark SQL's high-level engine that processes real-time data like batch queries with data frames and data sets, outperforming DStreams' rdd-based micro-batch approach, with event time processing.

Learn how to implement watermarking in Spark Structured Streaming to handle late data, bound state size, and perform efficient windowed aggregates using with watermark on event time.

Learn how to handle late arriving data in structured streaming using watermarking with windowed aggregations, configure event time lateness, and apply custom allowed lateness logic to balance accuracy and performance.

Spark integrates with Kafka to enable real-time analytics by reading Kafka as a data frame via structured streaming, processing micro-batches with automatic offsets and checkpoints for fault tolerance.

Explain how checkpointing and stateful operations enable fault-tolerant, exactly-once structured streaming by preserving state across batches, supporting windowed computations with watermarking.

Empowers data engineers to build scalable pipelines and models with MLlib, Apache Spark’s distributed machine learning library. It covers classification, regression, clustering, and recommendations, plus feature engineering and evaluation tools.

GraphX, a Spark API for graph parallel computation, represents data as vertices and edges on RDD and runs scalable graph analytics with algorithms like PageRank and connected components.

Explore PageRank in Spark as a distributed, iterative graph algorithm that measures node importance in a web graph using a damping factor of 0.85 and convergence to web ranking scores.