Data Engineering Essentials: Databricks, PySpark, and SQL

This overview introduces the module's 5 videos, covering the big data problem, distributed and parallel computing, Apache Hadoop, Apache Spark, and data formats like CSV, JSON, Avro, Parquet, ORC, and Delta Lake. It previews key terms such as distributed computing and columnar storage before diving into the details.

You'll learn how the explosion of data from the internet, social media, and mobile apps in the 2000s overwhelmed traditional databases, tracing the timeline from ARPANET through the rise of Hadoop, Spark, Databricks, and PySpark as the tools built to handle this scale.

This lecture explains how distributed parallel computing works: breaking a large task into subtasks assigned to nodes in a cluster, coordinated by a master node, then combining partial results into a final solution. It covers the 5-step process from problem submission to result aggregation.

You'll learn how Apache Hadoop, introduced in 2006, combined HDFS for distributed storage with MapReduce for distributed processing, including how files are chunked, replicated, and processed by name nodes and worker nodes. The lecture also covers Hadoop's limitations, such as its reliance on disk I/O and its steep MapReduce learning curve, which motivated the creation of Spark.

This lecture traces Spark's history from its 2010 open-sourcing through PySpark and Databricks, covering Spark Core, RDDs, transformations and actions, and lazy evaluation. It also introduces DataFrames, datasets, Spark SQL, MLlib, and Spark Streaming as the components built on top of Spark Core.

You'll learn the strengths and limitations of major data formats used in big data: CSV, XML, JSON, Avro, Protocol Buffers, ORC, Parquet, and Delta Lake. The lecture explains schema enforcement, compression, columnar storage, and schema evolution, showing why formats like Parquet and Delta are preferred for Spark analytics workloads.

This module overview previews 8 videos explaining what Spark actually is (a processing engine, not a database), and introduces the driver, executor, and cluster manager as the three key players in every Spark job, along with resource allocation and the spark-submit command.

Using a cooking analogy, this lecture explains that Spark provides only processing software, not storage or compute, and describes how driver code, executors, and cluster managers like YARN or Kubernetes work together, and how data formats like Parquet and Delta affect execution speed.

You'll learn how the cluster manager coordinates a master-worker architecture, allocating resources when a Spark application is submitted, launching executors on worker nodes, and covering options including Spark standalone, YARN, Mesos, and Kubernetes.

This short lecture explains the driver as the main JVM process launched by spark-submit, responsible for creating the SparkContext, converting user code into a logical execution plan, and requesting resources from the cluster manager to initiate executors.

This brief lecture covers the executor's role: executing tasks assigned by the driver, reporting progress, reading and writing data, and managing partitions in memory or on disk for caching, running for the duration of the application on worker nodes.

You'll walk through a full Spark deployment example, from spark-submit on a local machine to the cluster manager master allocating executors, and learn the difference between client mode and cluster mode for where the driver runs.

This lecture covers the spark-submit script in detail, including the --py-files argument for Python dependencies, client versus cluster mode tradeoffs, and alternative deployment methods like the Spark REST API, Apache Livy, notebooks, and Airflow integration.

You'll learn how executor memory is split into execution and storage portions using the default 60/40 ratio, how CPU cores determine task parallelism, and why the driver delegates execution management to executors for scalability and fault tolerance.

This lecture defines a Spark application as a package combining Spark Core engine libraries with user-written driver code, explaining how the driver code is converted through logical and physical plans, and how PySpark uses Py4J to bridge Python and Java objects.

This module overview introduces Spark Core as the foundation underlying SQL, DataFrames, and streaming, previewing SparkContext, RDDs, partitions, shared variables like broadcast variables and accumulators, and SparkSession using a FIFA World Cup data example.

You'll learn that SparkContext is the main entry point for Spark Core, responsible for creating RDDs and connecting to a cluster, and how it differs from SparkSession, which is the entry point for DataFrames via Spark SQL. The lecture demonstrates creating RDDs with range, parallelize, and collect.

Using a FIFA World Cup dataset, this hands-on lecture demonstrates RDD transformations (map, filter, reduceByKey) and actions (take, first, reduce, collect) to find the country with the most World Cup wins, explaining the rule that transformations return RDDs while actions trigger jobs.

This lecture explains RDDs as logical collections of data pointers distributed across a cluster, how the DAG and lazy evaluation defer computation until an action, and how controlling partition counts affects parallelism and resource utilization using the glom method as an example.

You'll learn about Spark's two shared variable types: broadcast variables, which cache a read-only copy on each worker to avoid repeatedly shipping data with every task, and accumulators, which support associative operations like counters and sums, demonstrated with a countries-of-interest filtering example.

This lecture explains SparkSession as a wrapper over SparkContext and the entry point for DataFrames APIs, covering the builder pattern, getOrCreate, and how only one SparkContext can be active per driver, using a PySpark shell demo to show session lifecycle management.

This module overview shows how the FIFA World Cup problem solved with a dozen RDD transformations can be solved in about 5 lines using DataFrames, previewing schema, strongly-typed columns, DataFrame creation from multiple sources, API operations, and SQL.

You'll learn how to create DataFrames from SparkSession, using methods like read.csv, and perform operations like select, groupBy, count, and orderBy, re-solving the FIFA World Cup winner problem more simply than with RDDs, plus how to register DataFrames as temporary views for SQL queries.

This lecture covers defining and inferring DataFrame schemas using printSchema, StructType, and StructField, and explains how to handle malformed records using permissive, dropMalformed, and failFast read modes, including Databricks' bad-records-location shortcut.

You'll learn PySpark's generic load and write APIs for reading and writing DataFrames across formats like Parquet, JSON, CSV, and JDBC, including configuration options like ignoreCorruptFiles, save modes (errorIfExists, append, overwrite, ignore), and writing to tables in the Hive metastore.

This lecture covers the Column and Row classes in Spark, explaining how columns define schema and support transformations like filtering and aggregation, how rows represent immutable records, and the difference between withColumnRenamed and alias.

You'll learn DataFrame API operations grouped into schema transformations (withColumn, drop, withColumnRenamed, cast) and content transformations (filter, union, dropDuplicates, orderBy, join, groupBy), plus key actions like show, collect, take, count, explain, summary, and foreach.

This lecture mirrors the previous DataFrame API operations using Spark SQL syntax instead, covering column renaming with AS, casting, filtering with WHERE, UNION versus UNION ALL, DISTINCT, joins, GROUP BY aggregation, and switching between SQL views and DataFrames using spark.table.

This module overview introduces Spark's 3-layer optimization stack: automatic optimization via the Catalyst optimizer and adaptive query execution, manual tuning of shuffle partitions and caching, and advanced techniques like skew handling, window functions, and UDFs.

You'll learn how DataFrames build on RDDs with lazy evaluation and the Catalyst optimizer, and how to use the explain command to see how Spark applies column pruning, predicate pushdown, hash partitioning, and shuffle reduction regardless of the order code is written in.

This lecture details Catalyst's three phases—logical plan creation, rule-based and cost-based optimization, and physical plan generation—covering specific rules like predicate pushdown, constant folding, column pruning, and join reordering, demonstrated with the explain(extended) command.

You'll learn how Spark's cost-based optimizer automatically selects join strategies—broadcast join, sort-merge join, and shuffle hash join—based on table size and statistics, including when each strategy applies and how analyze table commands help Spark make better decisions.

This lecture explains AQE, enabled by default since Spark 3.2, which adjusts execution plans at runtime based on actual metrics: splitting skewed partitions, switching join strategies mid-execution, and coalescing shuffle partitions to reduce unnecessary overhead.

You'll learn about Tungsten, Spark's low-level execution engine introduced in version 1.5, covering off-heap memory management, runtime bytecode generation, and cache-aware computation, which together improve CPU and memory efficiency for compute-intensive operations.

This lecture covers vectorized query execution, which processes data in column batches rather than row by row, uses SIMD instructions for parallel operations, and reduces serialization overhead—especially effective with columnar formats like Parquet and ORC.

You'll learn why the default 200 shuffle partitions rarely fit real workloads, with guidance to target 100-200MB per partition and 2-4x more partitions than available cores, plus how AQE can dynamically coalesce shuffle partitions at runtime.

This lecture distinguishes table partitioning (physical organization on disk) from shuffle partitioning, demonstrating writes partitioned by year and city, and explains partition pruning and how to choose low-cardinality, frequently-filtered partition columns.

You'll learn the difference between coalesce, which reduces partitions without a full shuffle, and repartition, which can increase or decrease partitions via a full shuffle for even distribution, including the pattern of repartitioning by key before coalescing.

This lecture covers caching DataFrames in memory or disk using cache and persist, explains storage level tradeoffs (memory-only, memory-and-disk, off-heap), and demonstrates performance gains from caching along with the importance of calling unpersist when done.

You'll learn about the broadcast join size threshold (default 10MB), the tradeoffs of raising it—memory pressure and network saturation—and how to use the explicit broadcast function to force broadcasting for critical workloads.

This lecture explains bucketing as a way to pre-shuffle high-cardinality join columns to avoid costly shuffles at query time, covering bucket pruning, combining sorting with bucketing for merge joins, and best practices like matching bucket counts across joined tables.

You'll learn how to detect data skew from symptoms like long-running tasks and uneven executor memory, how AQE automatically splits skewed partitions, and the manual salting technique for cases AQE cannot resolve, using a heavily skewed customer transaction example.

This lecture covers window functions for rank, dense_rank, row_number, running totals, and lead/lag comparisons, explaining rowsBetween versus rangeBetween, and how window functions can replace complex self-joins for significant performance gains.

You'll learn how to register and use Python UDFs in Spark SQL, along with their performance costs from serialization overhead, and optimization tips including preferring Scala/Java UDFs, using vectorized pandas UDFs, and favoring built-in SQL expressions when possible.

This module overview introduces the Spark UI as the tool for diagnosing performance, previewing the jobs-stages-tasks hierarchy, executor monitoring for memory and load balance, and storage monitoring to verify caching effectiveness.

You'll learn how to access the Spark UI (typically port 4040) and tour its main tabs—Jobs, Stages, Storage, Environment, Executors, SQL, and Structured Streaming—understanding what each tab reveals about performance, resource utilization, and data skew.

This lecture explains the three-level Spark execution hierarchy—jobs triggered by actions, stages created by shuffles, and tasks per partition—and how to use custom job descriptions, shuffle read/write metrics, and task-level logs to diagnose disk spills and stragglers.

You'll learn to read the Executors tab for memory categories, disk usage, core allocation, and task distribution, including how to spot imbalances, failed tasks, and high garbage collection time, and how to adjust spark.executor.memory and related configs.

Using a football events dataset, this lecture demonstrates caching DataFrames and monitoring the Storage tab to see memory versus disk spill, showing how exceeding roughly 50% of executor memory for caching causes performance to degrade rather than improve.

This module overview introduces reading data from Amazon S3 and relational databases via JDBC, covering IAM roles for authentication, columnar formats for performance, and the balance between Spark parallelism and database connection limits.

You'll learn how to connect Spark to Amazon S3 using the Hadoop AWS and AWS Java SDK libraries, authentication options from environment variables to IAM roles, and S3 performance tuning through connection settings, columnar formats, and partitioning.

This lecture covers connecting Spark to relational databases via JDBC, including driver management, connection pooling, partitioning strategies using lower/upper bounds, query pushdown, and write strategies like batch size tuning and save modes, demonstrated with DuckDB.

This module overview introduces structured streaming as processing an endlessly growing table via microbatches, previewing output sinks, output modes, checkpointing, triggers, Kafka integration, and stateful operations like deduplication with watermarks.

You'll learn the fundamental difference between batch processing (cold, stored data) and stream processing (hot, continuously arriving data), and why use cases like fraud detection, IoT monitoring, and real-time recommendations require stream processing instead of batch.

This lecture clarifies Spark's two streaming APIs: the legacy RDD-based DStreams, now in maintenance mode, and Structured Streaming, built on DataFrames and Spark SQL, which treats incoming data as an unbounded table processed via microbatches with the Catalyst optimizer.

You'll learn the core structured streaming APIs, readStream and writeStream, including the requirement for explicit schemas, and the five key write configurations: output sink, output mode, checkpoint location, trigger, and query options.

This lecture explains when to choose Spark structured streaming over alternatives like Apache Flink, Kafka Streams, or Kinesis, emphasizing Spark's unified batch/streaming API and analytical power, while noting its microbatch architecture trades off against sub-second latency needs.

You'll learn to configure the five key writeStream settings in depth: output sinks (file, Kafka, console, memory), output modes (append, complete, update), checkpointing for exactly-once fault tolerance, and trigger types including fixed interval, once, and availableNow.

This hands-on lecture builds a file-based streaming pipeline monitoring an S3 directory for CSV, JSON, and Parquet files, covering manual and inferred schemas, handling multiline JSON, checkpointing for durable sinks, and chaining sources into a basic ETL pipeline.

You'll learn to integrate Spark structured streaming with Kafka, covering bootstrap servers, topic subscription, starting offsets, the binary key/value schema, deserializing JSON payloads, writing back to Kafka, and production options like maxOffsetsPerTrigger and failOnDataLoss.