PySpark: Python, Spark and Hadoop Coding Framework & Testing

Learn to set up PySpark development in PyCharm on Mac, installing Java 11 and Python, installing PySpark, configuring conda environments, running Spark sessions and data frames, enabling Hive for tables.

Discover how spark works, from the driver program to workers, enable hive support, create a spark session, and use spark sql and data frames to query data.

Create and reuse a spark session at the start of the application by initializing it in a class and passing it to the ingest process.

Build a PySpark data pipeline by ingesting a CSV file into a spark dataframe, perform dataframe transformations, and explore operations and debugging with breakpoints in PyCharm.

Organize a PySpark data pipeline by separating ingestion, transformation, and persistence into ingest.py, transform.py, and persist.py, reading a csv, dropping nulls, and persisting to csv with headers.

Learn Python error handling with try-except blocks, logging errors, and converting exceptions to strings. See how to raise and propagate errors to a data pipeline and exit nonzero.

Explore log4p and log4python as alternatives to the Python logging library, configure with JSON and Python config files, install with pip, and compare debug and error outputs in PySpark.

Ingest data from Hive using Spark, apply transformations, and store the results in PostgreSQL, detailing schema creation, null handling, and dummy data setup.

Transform ingested data from a hive table by replacing null author names with unknown and null reviews with zero, creating df1 and df2, and store result in a postgres table.

Learn to structure and run unit tests for a PySpark application using the Python unittest framework. Create a test directory, write test classes and methods, and run tests with unittest.main.

Learn to unit test errors in PySpark by asserting the expected exception types, such as attribute error, when passing None to a transform, using a spark session and assertRaises.

Package and deploy Python code to a Spark cluster using spark-submit, verify pandas availability, and run the data pipeline py entry point against Hive tables in the Spark Warehouse.

Explore how Hadoop scales from a single machine to thousands of machines, delivering distributed storage and parallel processing via HDFS and MapReduce, with a master-slave architecture.

Explore hdfs, the distributed file system in the hadoop framework, and learn how 128 MB blocks replicate across three data nodes for fault tolerance, with configurable replication factors.

Utilize Google Cloud Dataproc to deploy a production-like Hadoop and Spark cluster, learn per-second billing, $300 free credits, and cost for a 3-node setup with up to 100 days free.

Create a production grade big data cluster on Google Cloud Dataproc with a three-node setup (one master, two workers), enable the API, and access via SSH, then delete when finished.

Explore mapreduce and yarn in the Hadoop ecosystem, detailing map and reduce tasks, HDFS storage, and how Yarn enables Spark and other tools to process data at scale.

Explore querying hdfs data with Hive on a Dataproc cluster, creating a futurex database and a csv-based table to run mapreduce-backed queries.

Analyze a billion records with hive on hdfs, create hive tables, perform aggregations, and compare with spark for large-scale data processing.

Learn how Spark ingests data from multiple sources into dataframes, performs transformations in a distributed pipeline, and outputs enriched datasets for downstream apps, highlighting its speed and use in production.

Master basic Python concepts essential for PySpark, including data types, loops, lists, strings, and reading from and writing to files, in a Colab notebook.

Learn to create rdds in spark by initializing a spark context from a spark session, then build rdds from collections or text files and examine data with collect and take.

Define and call Python functions with def, and return multiple values in a Colab notebook. Use lambda expressions for concise anonymous functions, and apply map and reduce for one-line operations.

Register a dataframe as a temporary view, then use spark sql to run familiar sql queries on dataframes, and compare sql with the dataframe api.