PySpark: Python, Spark and Hadoop Coding Framework & Testing

Learn to set up PySpark development in PyCharm on Mac, installing Java 11 and Python, installing PySpark, configuring conda environments, running Spark sessions and data frames, enabling Hive for tables.

Install Python, PyCharm, and JDK on Windows, configure Spark and winutils for Hadoop and Hive, then build and query data with PySpark in PyCharm.

Discover how spark works, from the driver program to workers, enable hive support, create a spark session, and use spark sql and data frames to query data.

Create and reuse a spark session at the start of the application by initializing it in a class and passing it to the ingest process.

Build a PySpark data pipeline by ingesting a CSV file into a spark dataframe, perform dataframe transformations, and explore operations and debugging with breakpoints in PyCharm.

Organize a PySpark data pipeline by separating ingestion, transformation, and persistence into ingest.py, transform.py, and persist.py, reading a csv, dropping nulls, and persisting to csv with headers.

Learn to use Python's logging library to control log levels from debug to error, configure output, and apply log statements across classes for debugging and application monitoring.

Learn to manage log levels in a Python PySpark project with a configuration file by creating resources/configs, defining a console handler and formatter, and controlling root and handler levels.

Learn Python error handling with try-except blocks, logging errors, and converting exceptions to strings. See how to raise and propagate errors to a data pipeline and exit nonzero.

Explore log4p and log4python as alternatives to the Python logging library, configure with JSON and Python config files, install with pip, and compare debug and error outputs in PySpark.

Ingest data from Hive using Spark, apply transformations, and store the results in PostgreSQL, detailing schema creation, null handling, and dummy data setup.

Transform ingested data from a hive table by replacing null author names with unknown and null reviews with zero, creating df1 and df2, and store result in a postgres table.

Download and install PostgreSQL on Windows, open PgAdmin to localhost, then create a Futurex schema and a Futurex course catalog table with id, name, author, and date fields.

Persist the transformed data frame to PostgreSQL by creating a table with the same columns and appending via JDBC, demonstrating end-to-end read from Hive to PostgreSQL and back.

Read configurations from a properties file using the Configparser library, define a db configs section with a target table, load it, and log the value.

Learn to structure and run unit tests for a PySpark application using the Python unittest framework. Create a test directory, write test classes and methods, and run tests with unittest.main.

Learn to unit test errors in PySpark by asserting the expected exception types, such as attribute error, when passing None to a transform, using a spark session and assertRaises.

Package and deploy Python code to a Spark cluster using spark-submit, verify pandas availability, and run the data pipeline py entry point against Hive tables in the Spark Warehouse.

Explore how Hadoop scales from a single machine to thousands of machines, delivering distributed storage and parallel processing via HDFS and MapReduce, with a master-slave architecture.

Explore hdfs, the distributed file system in the hadoop framework, and learn how 128 MB blocks replicate across three data nodes for fault tolerance, with configurable replication factors.

Utilize Google Cloud Dataproc to deploy a production-like Hadoop and Spark cluster, learn per-second billing, $300 free credits, and cost for a 3-node setup with up to 100 days free.

Sign up for Google Cloud free tier to receive $300 in credits for data engineering and machine learning, then monitor credits and trial end date in the billing console.

Create a production grade big data cluster on Google Cloud Dataproc with a three-node setup (one master, two workers), enable the API, and access via SSH, then delete when finished.

Push local files to HDFS with Hadoop fs put and create a user data directory. Use Hadoop fs ls, get, and rm to manage files and replication on Dataproc cluster.

Explore mapreduce and yarn in the Hadoop ecosystem, detailing map and reduce tasks, HDFS storage, and how Yarn enables Spark and other tools to process data at scale.

Explore querying hdfs data with Hive on a Dataproc cluster, creating a futurex database and a csv-based table to run mapreduce-backed queries.

Analyze a billion records with hive on hdfs, create hive tables, perform aggregations, and compare with spark for large-scale data processing.

Revise big data and Hadoop concepts, including HDFS, MapReduce, and NameNode data nodes. Review five V's: volume, velocity, variety, veracity, value—and discuss replication, fault tolerance, and Hive partitioning and bucketing.

Set up a 1 master, 2 worker Dataproc cluster and use PySpark to load a CSV from HDFS, with header, and query via Spark SQL on a Hive table.

Learn how Spark ingests data from multiple sources into dataframes, performs transformations in a distributed pipeline, and outputs enriched datasets for downstream apps, highlighting its speed and use in production.

Load data from hive tables, CSV files, Parquet, JSON, relational databases, or cloud storage into Spark session data frames, then transform with filters, aggregates, joins, and derived features.

Understand the resilient distributed dataset as the foundational Spark data structure and how RDDs enable in-memory, partitioned data and immutable transformations behind DataFrames.

Master basic Python concepts essential for PySpark, including data types, loops, lists, strings, and reading from and writing to files, in a Colab notebook.

Learn to create rdds in spark by initializing a spark context from a spark session, then build rdds from collections or text files and examine data with collect and take.

Define and call Python functions with def, and return multiple values in a Colab notebook. Use lambda expressions for concise anonymous functions, and apply map and reduce for one-line operations.

Master RDD transformations and actions in PySpark, including map, filter, flatMap, union, and reduce, leveraging immutability and lazy evaluation to build and execute data pipelines.

Drive insights with Spark transformations on a retail orders dataset, mastering filter, select, groupby, and window functions to compute revenue, totals, trends, and data quality in PySpark.

Register a dataframe as a temporary view, then use spark sql to run familiar sql queries on dataframes, and compare sql with the dataframe api.