Data Engineering Vol2 AWS : Data Processing - Spark & Kafka

Explore distributed storage and compute for big data with HDFS, MapReduce, YARN, and zookeeper, and understand Parquet and Avro formats alongside Spark and Kafka.

Learn map and reduce tasks, how map processes records into partitions and how reduce aggregates by key ranges, with Hadoop, HDFS, YARN, Hive, Pig, Mahout, and Spark.

Explore how MapReduce works with a Python mrjob library, creating key-value pairs with mapper and reducer tasks, while illustrating io bottlenecks from temporary files and why Spark overtook MapReduce.

Log in as the Hadoop user on EMR to run HDFS and Spark commands, manage HDFS, local, and S3 files, and copy data from S3 to HDFS using distcp.

Understand how yarn acts as a resource negotiator in a cluster, allocating containers via the resource and node managers for MapReduce and Spark workloads across hdfs or cloud storage.

Discover how zookeeper handles configuration management and service discovery in a distributed cluster, and examine parquet as columnar storage and avro as row oriented storage for analytics.

Discover Apache Spark as the heart of the data engineering pipeline, covering batch and stream processing, Spark ecosystem, and how Spark fits into ingestion, processing, and data storage with PySpark.

Explore the spark ecosystem—from storage layers to data frame APIs and structured streaming. Include ML libraries, Kafka and Kinesis, and the driver, executors, and resource management.

Set up spark on a JVM-based stack by installing the JDK, Scala (if using Scala), and Spark, then configure JAVA_HOME and SPARK_HOME for PySpark with history server.

Explore deploying a Spark cluster on AWS EMR, including primary, core, and task nodes, Spark Submit, and data stored in HDFS or S3.

Deploy spark in standalone mode on a laptop or EC2, set up master and workers, and explore Spark UI and data frames with Spark Session.

Explore spark submit deployment modes, cluster and client, and understand where the driver runs on EMR clusters or local machines, with notes on notebook usage.

Data frames are Spark's in-memory, immutable tables with schemas, created from files or databases and transformed through a directed acyclic graph, leveraging PySpark and Spark SQL.

explains how to create and configure a spark session in pySpark, using builder, conf, master, app name, and getOrCreate or create, plus read stream for batch and streaming with Kafka.

Explore how Spark batch processing uses data frame reader and writer to load csv, parquet, avro, and json into data frames and write results with options.

Define a data frame schema for CSV and JSON files using StructType and StructField in PySpark, or use header and inferSchema to detect types; note Parquet and Avro embed schema.

Explore how Spark partitions input data into blocks, map blocks to tasks and vcpus within executors, and observe the CSV to Parquet workflow in the Spark UI history server.

Learn how spark partitions files into 128 MB maximum blocks with a 4 MB minimum. Note default parallelism from CPUs and how data frames read via spark session.

Explore basic spark transformations and actions on data frames, including lazy evaluation, the load method, and the role of shuffle, executors, and jobs in practical data processing.

Create a spark session, load a csv with a header and inferred schema, and transform data using select and where (filter) and distinct on data frames, previewing results with df.show.

Create sequential data frame transformations from a single input df underscore cust, producing df3 and df4 using select, distinct, and filter. Show results and discuss memory considerations and spark submit.

Explore chain transformation in Spark data processing by chaining select and filter operations to extract visa payments with order totals over 35,000, showing benefits and readability compared to step-by-step transformations.

Explore Spark's basic transformations—including describe, limit, sort, drop, and dropDuplicates—to fetch column statistics, limit results, order by multiple keys, drop fields or duplicates, and handle nulls in data frames.

Discover how actions trigger dag processing and lazy evaluation in spark, transforming input data through transformations into a final data frame and writing it to output.

Explore Spark action and lazy loading by building data frames, applying drop, sort, and filter transformations, and writing parquet outputs for education, vehicle, and housing loan data.

Explore the spark ui hands-on, including history server and application server 4040, mapping an application's actions to jobs and examining environment variables, driver, and executors in a standalone cluster.

Analyze how Spark load behaves as a transformation or action across csv, json, parquet, and avro inputs with header and infer schema choices.

Compare CSV and Parquet file formats using Spark read and write operations, with and without transformations, to illustrate Parquet's analytics advantages and when CSV may suit.

Engage in a hands-on Spark project that builds five dataframes from finance data and uses transformations and actions to produce ten reports, including billion-dollar balances and France customer details.

Explore spark aggregations and transformations, including agg, group by, select expr, and with column, plus join, order by, subtract, and union for complex ETL analytics.

Explore PySpark data transformations to aggregate finance data: use agg with sum, avg, max, min, alias results, and group by transaction status to compute totals.

Learn to join two or more data frames in spark with the join API, using inner, left outer, and right outer joins to produce a combined result set.

Learn to use the column function (col) in PySpark SQL for aliasing, between, contains, starts with, ends with, like, substring, casting, and DataFrame transformations with hands-on examples.

Apply the order by transformation with column expressions to sort by start date, principal amount, and end date, then use select expr for sql-like functions and business loan labeling.

Explore the subtract transformation in Spark dataframes, which removes common rows and shows nonmatching rows from the first dataframe, with notes on select for column control.

Apply spark transformations on data frames: union, union all, intersect, and subtract, where unions keep duplicates and require column counts; use with column to add fields via select and drop.

Explore wide transformations in spark—group by, order by, sort, distinct, drop duplicates, and join—and learn how the shuffle and external shuffle service optimize cross-node data movement, partitions, and stage execution.

Demonstrate the shuffle process in a group by with Spark, using the Spark UI to analyze jobs, stages, and dag visuals, while exploring shuffle partitions and disk writes.

Learn how to set Spark SQL shuffle partitions based on post shuffle partition size (100–200 MB) and core count, using Spark UI to tune performance in joins.

Master spark basics—from transformations and actions to Spark UI—for diagnosing jobs and stages. Implement a ledger–account–customer join, export to parquet, and benchmark shuffle performance.

Explore spark input partitions and how spark.sql.files.maxPartitionBytes controls partition size and task count. The section demonstrates experiments with parquet and csv data, showing limited performance gains from larger partition sizes.

Learn how repartition and coalesce manage Spark output partitions, comparing full versus partial shuffles, and using column-based repartition (hash by column) to control output files.

Understand how partition by arranges data into folders by distinct values without changing output partitions, and how repartition or coalesce can alter partitions, with examples of partition-changing operations.

Explore spark dataframe writer output modes—overwrite, append, ignore, and error—and see how partitioning and partition by affect parquet file outputs on HDFS or S3.

Explore Spark memory and cpu usage for production deployments and whether new data frames keep occupying memory. Examine memory management, narrow vs wide transformations, and dag memory and adaptive execution.

Calculate Spark cluster memory by analyzing input and output data frames, including pre- and post-shuffle stages. Tune CPU needs through maximum parallelism and tasks to meet SLA.

Calculate a Spark prod cluster size using UAT testing and SLA targets, leveraging parallelism and input/output data, and verify with Spark UI insights to decide CPU and memory needs.

Explore spark ui end-to-end, linking dag visualization to whole stage code gen, map partitions, exchanges, and skipped stages, with sql dataframe tab insights.

Explore spark memory management, including reserved memory, spark memory, and user memory. Understand execution and storage memory inside spark memory, and configure memory with spark.conf or spark-submit.

Explore memory spilling in Spark: how execution and storage memory spills to disk when RAM runs short, and how unified memory management enables dynamic sharing and off-heap options.

Cache and persist save a data frame in memory or disk to avoid shuffles; use memory only, memory and disk, disk only, and off heap; clear with spark.catalog.clearcache or df.unpersist.

Explore how to detect memory spills and optimize caching and persist strategies in Spark UI, including on-heap and off-heap memory, storage metrics, and GC impact.

Explore how the Java virtual machine garbage collection affects Spark performance, and how to tune GC, executor and driver memory, and Kryo serialization to optimize Spark workloads.

Learn how the dataframe collect() API gathers records into a Python list and why it risks driver memory bottlenecks, guiding you to limit results and adjust spark driver settings.

Learn how for each and for each partition apply a custom Python function to data frame rows or partitions, enabling business logic beyond spark transformations for databases and apis.

Master Spark dataframe APIs such as replace, expression, when and otherwise, and df.fill, and compare collect with two local iterator to understand memory and driver streaming across partitions.

Explore Spark functions lit, expr, and concat with hands-on exercises, mastering constants, variables, and sql-like expressions on data frames.

Learn how to use Spark date functions such as extract, date_add, date_sub, date_diff, and date_format to manipulate and format date and timestamp data in pipelines.

Learn to use when and otherwise for conditional logic in PySpark SQL. See how lower and upper convert case, use bracketed when conditions for multiple checks, and apply otherwise.

Explore spark window functions and their sql analogs to partition and order data frames, enabling over clauses and functions like dense_rank and rank.

Use window functions in PySpark to find earliest passengers by travel date and top five per country. Apply dense_rank with partition by country and travel date, and drop duplicates.

Explore how to compute the top revenue generating frequent flyers per country using Spark window functions, partition by country, and rank by total cost, with parquet files.

Examine layover analysis, travel activity, and environmental impact on flight data using Spark, covering data cleansing, duplicates removal, same-day travel, layover calculations, and carbon emissions reporting.

Explore spark window functions like lead and lag. Learn nth_value, first_value, last_value, and percent_rank with over partitions by country and order by price.

Explore PySpark window functions, including lead, lag, nth_value, first_value, and last_value, to compare engine performance by passenger country, partitioned and ordered by turbulence.

Master Spark Catalyst Optimizer and Tungsten Execution Engine, understand internal joins and physical plan operators, and explore adaptive query execution for data skew and performance tuning.

Explore spark internal joins, focusing on sort merge join as the default for inner, left, right, outer, semi, and anti joins, driven by shuffle and join keys.

Explore how hash tables power Spark internal joins and how broadcast hash join broadcasts the smaller data frame to all executors, enabling efficient joins with a larger data frame.

Explore Spark's broadcast nested loop join, where the smaller data frame is broadcast to all executors and iterated for each row of the larger data frame, noting performance caveats.

Spark uses shuffled hash join to shuffle both data frames by the join key, build a hash table from the smaller frame, and match rows in large data sets.

Explore Spark internal joins, focusing on the shuffled nested loop join, its two phases, non-equality conditions, and how join hints optimize performance.

Explore Spark DAG and physical plan operators, including hash aggregate, sort aggregate, and broadcast exchange, and learn how in-memory hash tables, memory spills, and shuffle affect performance.

Explain how Spark's broadcast hash join broadcasts the smaller data frame, builds a hash table on each executor, and performs parallel joins, versus the less efficient nested loop broadcast join.

Explore SparkUI operators for data processing, including shuffle exchange, shuffled hash join, shuffled nested loop join, limit, collect limit, window, and reused exchange, and learn when to avoid cartesian products.

Hands-on with spark operators, exploring group by, select, join, and window functions; reveal hash aggregates, exchanges, and limits in spark ui dag and the execution plan.

Learn how Spark explain reveals logical and physical plans, catalyst optimization, and code generation, and map explain output to Spark UI Dag visualizations for performance analysis.

Learn about data skewness in Spark, its partition distribution, and salting as a remedy. See how skew affects joins and group by, and identify it via Spark UI stages.

Analyze data skewness in Spark UI and code by generating skewed flight data for USA, Spain, and France and examining join versus groupby performance.

Learn to mitigate data skew and skewed joins in Spark using salting, by distributing skewed keys across multiple salted partitions and applying salt to both data frames before joining.

A hands-on guide to implementing salting in Spark to fix data skewness, creating salted data frames, and performing inner joins to dramatically reduce processing time.

Explore adaptive query execution in Spark, dynamically re-optimizing execution plans with runtime statistics to reduce data skew, switch join strategies, and coalesce shuffle partitions.

Discover how disabling and enabling adaptive query execution and coalescing shuffle partitions optimize joins on parquet flight and passenger data, using initial partitions and a 400 mb advisory partition size.

Explore how adaptive query execution dynamically rewrites Spark plans to fix data skew and optimize skew joins, reducing shuffle partitions and improving run time.

spark dynamically prunes partitions based on filter columns, skipping irrelevant partitions to speed up queries; partitioning by country enables significant scans and faster execution.

Explore dynamic resource allocation in Spark, which adjusts executors at runtime to optimize compute and memory, guided by the driver and resource negotiator. Understand enabling parameters and backlog timeouts.

Explore catalyst optimizer and tungsten execution, read explain plans, and master Spark's data skewness handling, adaptive query execution, and memory management across Spark deployments.

Explore how to run PySpark applications on cloud hosting platforms, focusing on AWS EMR architecture, primary/core/task nodes, VPC subnets, and EMR software settings and steps.

Configure an EMR cluster with EMR service role and EC2 instance profile to let Spark access S3, RDS, Redshift, and other AWS services.

Set spark defaults and spark env on EMR via software settings, configure yarn parameters, and submit spark apps using EMR steps or spark-submit to S3 or HDFS.

Prepare an EMR environment to run Spark applications by configuring Yarn and executor resources, exploring S3 versus HDFS data access, and running jobs with Spark submit or EMR steps.

Submit spark apps to an EMR cluster from the primary node or laptop, tune memory and cores, and run EMR steps via AWS CLI for S3 and HDFS data.

Use EMR steps to run multiple Spark applications sequentially on one EMR cluster, with code in S3, executing flight efficiency, layover analysis, and read/write json files against S3.

Learn to use Jupyter notebook on EMR for interactive PySpark development, submitting code via Spark Submit or EMR CLI, with EMR Studio and IAM roles for development and testing.

Configure the Spark VPC security groups, create the EMR Jupiter security group, set outbound internet access, and define IAM roles, policies, trust relationships, and an EMR serverless application.

Learn to create an EMR studio and a serverless Spark application, attach a Jupyter notebook, and run PySpark interactively with S3 notebook storage, Glue metastore, and proper VPC networking.

Learn how to read from and write to relational databases using Spark via JDBC, using MySQL as an example, with options like URL, table, user, password, driver, and pushdown settings.

Shows how to write data from S3 to an Aurora MySQL RDBMS using Spark on EMR, including setting up the database, JDBC driver, and write options like num partitions.

Read from a MySQL RDBMS with Spark JDBC, using proper partitioning (partition column, lower and upper bounds, num partitions) and write the result to S3 as parquet.

Explore how Spark processes data from OLTP and NoSQL sources into data warehouses like Redshift. Learn about data marts, data mesh, and CDC with Kafka in AWS environments.

Learn how to read from and write to a redshift data warehouse using Spark on AWS EMR, with S3 temp, IAM roles, and Spark transformations for country revenue.