Apache Spark 3 for Data Engineering & Analytics with Python

Learn spark foundations and build a blog app to compute the most orders per region and country, while exploring spark transformations, actions, and directed acyclic graph in spark web ui.

Explore the Spark web UI and DAG visualisation to see how stages and jobs execute in a lazy transformation-to-action workflow, with Python and Java APIs and data frames.

Remove duplicates with the distinct transformation on an RDD, verify the reduced count, and then filter words with a lambda to keep those that start with s.

Explore map and flatMap transformations in Spark, applying complex operations while preserving record counts; create squared number pairs and flatten words into letters.

Learn how to sort by key using Spark's sort by key transformation on an RDD of country-ranking tuples, including descending sort with map and collect.

Explore the reduce action in RDDs by aggregating values with a lambda to a single result, summing lists, finding the longest word, and deriving max and min values.

Use spark 3 to tackle the XYZ research challenge with the attached data, applying union and subtract transformations to count initiated projects and year-one and year-two completions.

Import libraries, create a spark session, and define a sales schema; load csv data from data folder with a header, then print the schema and show the first 10 records.

Manage performance errors in Spark by stopping and restarting sessions, cleaning up temporary space, and restarting notebooks to recover from block manager errors and file not found exceptions.

Learn to filter data with filter and where functions to select salary 3000 or less, combine conditions with and, and use contains to find Land of the Lost.

Learn how to create individual row items, assemble them into a data frame, and merge data frames using Spark's union transformation for tasks in data engineering and analytics with Python.

Learn to clean sales data with Spark, remove bad records, extract city and state from address, convert data types, add year and month, write output partitioned by report and month.

Extract the city and state from the purchase address using Spark's split function to create new city and state columns in the sales dataframe.

Load the flight summary data into a Spark data frame, infer the schema, and count route usage (origin to destination airports) to set up aggregations.

Group by origin airport, apply count and max aggregations, and order by results to reveal airport counts; extend to group by state and city with California filters.

Discover how to register for the Databricks community edition, complete sign-up, verify your email, and land on the data platform's front page for future Spark data engineering lessons.

Learn to create a Databricks cluster to run Spark code, starting with a single-node community edition, Python 3 support, and understanding idle termination and availability zone settings.

Load sales data files into a Spark dataframe by defining a schema and a data path, then read csv files with headers. Attach the cluster and verify the initial records.

Insert records from a sales raw data frame into the sales table using Spark SQL and a temporary view. Verify compatibility with describe and understand the insert process with DML.

Insert transformed sales records with a spark Eskil statement, convert order date to timestamp, and extract year and month for final analytics and POCKY storage.

Explore the best ad timing by extracting the hour from order dates, measuring purchasing power per unique order id, and visualizing peak moments with a line chart.

Identify the most common product pairs sold together in New York using Apache Spark 3's data frame API, grouping by order IDs and visualizing results with a pie chart.

Dive into core data fundamentals with DDL-based database and table creation, SQL-like select statements, and DML operations, then explore clusters, notebooks, and dashboards.