Apache Spark 3 for Data Engineering & Analytics with Python

Learn spark foundations and build a blog app to compute the most orders per region and country, while exploring spark transformations, actions, and directed acyclic graph in spark web ui.

Create a new project directory, set up a Spark session with the Spok library in a Jupyter notebook, import core functions, and prepare to count country and region sales data.

Create a spark session, load a csv with header and inferred schema into a dataframe, select region and country, and compute total orders per region and country in descending order.

Explore narrow and wide Spark transformations and how single input partitions avoid shuffles. Learn that group by and order trigger shuffles, after which transformations produce new data frames.

Explore the Spark web UI and DAG visualisation to see how stages and jobs execute in a lazy transformation-to-action workflow, with Python and Java APIs and data frames.

Explore resilient distributed datasets (RDDs) as immutable, partitioned collections with parallel transformations and actions, and compare the low-level API to high-level Spark APIs like data frames and data sets.

Set up an Apache Spark session in Python, parallelize a word list, and apply lazy transformations before triggering actions to print results.

Remove duplicates with the distinct transformation on an RDD, verify the reduced count, and then filter words with a lambda to keep those that start with s.

Explore map and flatMap transformations in Spark, applying complex operations while preserving record counts; create squared number pairs and flatten words into letters.

Learn how to sort by key using Spark's sort by key transformation on an RDD of country-ranking tuples, including descending sort with map and collect.

Explore the reduce action in RDDs by aggregating values with a lambda to a single result, summing lists, finding the longest word, and deriving max and min values.

Learn to convert Fahrenheit to centigrade using Spark transformations with map and filter, building a Spark session, and applying a Fahrenheit-to-centigrade function.

Use spark 3 to tackle the XYZ research challenge with the attached data, applying union and subtract transformations to count initiated projects and year-one and year-two completions.

Explore creating rdds for three years of research data, unioning them, deduplicating with distinct, and counting to reveal 12 research projects initiated over the three-year period.

Compute projects completed in year one and across the first two years by applying set operations: subtract, union, and distinct, using 2001 and 2002 data.

Explore why structured APIs in Spark offer readable code and data integrity, comparing the data frame and data set APIs with the RTD API.

Set up a Spark project folder, create dataset directories, initialize and activate a Python virtual environment, then install Spark, Seabourne, and Jupiter Lab for development.

Build a Spark DataFrame by defining a schema with StructType and StructField, using string and integer data types, then show the schema and the data for a person dataset.

Read csv, json, and avro data with header and inferred schema using the dataframe reader; project incident number, date, and city; write to parquet output.

Configure a Spark session, define a string schema, and load raw sales data into a dataframe; then prepare a skimmer with named columns and display the first 10 records.

Import libraries, create a spark session, and define a sales schema; load csv data from data folder with a header, then print the schema and show the first 10 records.

Explore operations on api data to shape data frames using column selection, expressions, filtering, deduplication, sorting, and reshaping with renamed, added, or dropped columns; create python user-defined functions for spark.

Manage performance errors in Spark by stopping and restarting sessions, cleaning up temporary space, and restarting notebooks to recover from block manager errors and file not found exceptions.

Learn to read a json file with Spark, handle nested fields like favorite movies as an array, and define a schema to load and inspect data.

Explore Spark data frame columns and expressions, using call and expression functions like concat_ws to compose full names and compute salary increases with an alias.

Learn to filter data with filter and where functions to select salary 3000 or less, combine conditions with and, and use contains to find Land of the Lost.

Learn to create a unique set of rows with the distinct function, drop duplicated values, and order by year and first name in a Spark DataFrame workflow with Python.

Learn how to create individual row items, assemble them into a data frame, and merge data frames using Spark's union transformation for tasks in data engineering and analytics with Python.

Learn to add, rename, and drop columns in a dataframe, create a salary increase column, rename favorite movies to movies, and round values to two decimals.

Learn to handle missing and bad data in Spark by creating a bad movies data frame, dropping rows with any or all nulls, and describing numeric and string columns.

Create and use Spark user defined functions to compute grades from student scores. Build a Python function, convert to a Spark udf, and apply it to a data frame.

Learn to clean sales data with Spark, remove bad records, extract city and state from address, convert data types, add year and month, write output partitioned by report and month.

Remove null rows and bad records from the sales data frame, drop null values, eliminate duplicates, and filter anomalies, then validate results with describe checks.

Extract the city and state from the purchase address using Spark's split function to create new city and state columns in the sales dataframe.

Rename columns and cast types for the sales data, converting order id to integer, quantity to integer, price to float, and order date to timestamp; add report year and month.

Arrange the final data frame in the proper column order, then write a partitioned output by reporter and month; explain how partitioning boosts performance.

Master how aggregations in Spark group data and compute summaries such as sum and max, illustrated with car prices by year to reveal stock value trends.

Load the flight summary data into a Spark data frame, infer the schema, and count route usage (origin to destination airports) to set up aggregations.

Explore Apache Spark 3 aggregations with Python, focusing on count and count distinct, and learn how count excludes nulls when counting records like destination airports.

Explore Apache Spark 3 aggregations, using min, max, sum, sumDistinct, and avg to compute flight counts and student data, and verify results via two methods.

Group by origin airport, apply count and max aggregations, and order by results to reveal airport counts; extend to group by state and city with California filters.

Read 2019 sales data into a Spark data frame from the party files directory and tackle questions on top month, city, ad timing, and often bought together products in New York.

Rename the sales analytics file to sales dataprep, then create a Spark session and load the 2019 data from the partitioned directory for prep.

Leverage pandas dataframes to calculate monthly sales by grouping by month and summing price times quantity, then visualize with matplotlib and seaborn to identify December as the best sales month.

Identify the city that sold the most products by grouping by city, summing quantity, and ordering results in descending order, then visualize total products per city.

Analyze 2019 sales data to identify peak hours for advertising, showing total orders peak around 7–8 pm and 11 am–12 am, guiding the board's advertising decisions.

Identify frequently bought together products in New York using Apache Spark by aggregating order product lists, filtering two or more item orders, and counting co-purchase frequencies.

Explore how Databricks provides a managed, cloud-scale platform with pay-as-you-go pricing, a minimal collaborative notebook interface, built-in job scheduling, and easy connections to analytics tools.

Learn Spark SQL to create databases and tables, run queries, and manage records using select, insert, update, delete, and where in a relational data context.

Discover how to register for the Databricks community edition, complete sign-up, verify your email, and land on the data platform's front page for future Spark data engineering lessons.

Learn to create a Databricks cluster to run Spark code, starting with a single-node community edition, Python 3 support, and understanding idle termination and availability zone settings.

Create two Databricks notebooks, explore saving and organizing notebooks in the workspace, and initialize a Spark session for reading data in Dataprep.

Load sales data files into a Spark dataframe by defining a schema and a data path, then read csv files with headers. Attach the cluster and verify the initial records.

Create a sales database and a sales table with fields like order_id, product, quantity_ordered, price_each, order_date, and customer_address, using delta format and managed versus unmanaged tables.

Insert records from a sales raw data frame into the sales table using Spark SQL and a temporary view. Verify compatibility with describe and understand the insert process with DML.

Expose bad data and remove junk records to ensure data quality with spark. Query the sales data using select and from patterns to identify and remove problematic records.

Exclude bad records with a query and verify results using a temporary table in a with clause. Eskil enables extracting state and city within one statement and confirms removal.

Extract city from the second element and the two-letter state from the third element of the split purchase address using built-in split and substring functions.

Insert transformed sales records with a spark Eskil statement, convert order date to timestamp, and extract year and month for final analytics and POCKY storage.

Analyze sales data with Apache Spark in an Ezekial Analytics notebook, computing monthly sales from price times quantity and visualizing results to identify the best month.

Group data by city, sum quantities to compute total products sold per city, and sort by total descending to reveal San Francisco as the top city with a bar chart.

Explore the best ad timing by extracting the hour from order dates, measuring purchasing power per unique order id, and visualizing peak moments with a line chart.

Identify the most common product pairs sold together in New York using Apache Spark 3's data frame API, grouping by order IDs and visualizing results with a pie chart.

Create a dashboard by adding visuals, naming it sales analytics, and arranging visuals for alignment. Explore dashboard settings and navigate back to the sales analytics dashboard.

Dive into core data fundamentals with DDL-based database and table creation, SQL-like select statements, and DML operations, then explore clusters, notebooks, and dashboards.