Data Engineering Masterclass for Beginners

Understand what big data is, why storing and processing massive data matters, and how the five V's drive insights. Discover distributed storage and computing and Hadoop’s role.

Explore how Hadoop enables distributed storage and parallel processing across a cluster, with HDFS for storage and MapReduce for computation, scalable from a single machine to thousands.

Explore how HDFS distributes large data across a cluster using 128 MB blocks, threefold replication on data nodes for fault tolerance, and how to set up on Google Cloud.

Set up a production-like big data cluster with Google Cloud Dataproc, a fully managed Hadoop and Spark service. Create, use, and terminate clusters with free trial credits.

Sign up for a Google Cloud free tier account to receive $300 in free credits for data engineering and machine learning; monitor credits and trial end date in billing console.

Create a production-grade data cluster on Google Cloud Dataproc with one master and two workers, enable the Dataproc API, and access the master via SSH.

Push local files to hdfs using hadoop fs commands, create user home directories, and manage files with put, ls, mkdir, get, and rm on a single master with two workers.

Explore Hadoop's MapReduce for parallel processing on HDFS data, using map and reduce tasks with Yarn managing resources across the cluster.

Explore hive, a sql-like tool for processing structured data in hdfs; queries translate to mapreduce behind the scenes, enabling hive ql to create tables that provide a logical data view.

Load a retail csv into hdfs on a Dataproc cluster, create a Hive table mapped to the data, and run queries. Explore internal and external tables and MapReduce with Yarn.

Analyze a billion records in hdfs using hive to transform and aggregate data, create hive tables, and compare mapreduce performance to spark.

Explore how Hive partitioning improves query performance on large data by dividing data into country-based folders, enabling targeted scans and faster results.

Explore how hive bucketing speeds queries by splitting data into files via a hashing algorithm. Compare bucketing with partitioning, using a 20-bucket table on customer id to boost performance.

Revise big data concepts and Hadoop basics, including the five V's, HDFS, MapReduce, and Hive, and prepare for interview questions on NameNode, replication, and partitioning vs bucketing.

Explore how Apache Spark speeds big data processing by keeping intermediate results in memory, enabling real-time streaming and batch analytics, and replacing MapReduce in Hadoop ecosystems.

Spark hello world on Dataproc, wire a master–worker cluster, load an HDFS CSV from GitHub, explore the PySpark shell, and query with Spark SQL and Hive integration.

Learn to run PySpark on Google Colab by installing Java, configuring environment variables, and installing Spark and PySpark, then create a Spark session and work with GitHub and Dataproc.

Learn how Spark ingests data from csv, parquet, hive, databases, and S3 into dataframes, applies transformations, and builds scalable pipelines for large-scale data processing.

Load data from various sources into Spark DataFrames, then apply transformations such as filtering, aggregating, handling null values, joins, and derive features on distributed datasets via the Spark session.

Explore the fundamentals of RDDs, resilient distributed datasets, the core building blocks of spark, and how partitions, in-memory processing, and fault-tolerant lineage enable dataframes built on top of them.

Explore spark programming with RDDs, the core building blocks across old and new APIs. Create RDDs from collections and files via a spark context, and use collect, first, and take.

Discover how to define Python functions with def, call them, and return values, including multiple outputs, and use lambda expressions for concise, anonymous functions with map and reduce.

Explore RDD transformations and actions in Spark, learn how map, filter, and flatMap produce new RDDs from immutable data, while collect and reduce trigger lazy evaluation.

Explore Spark transformations on an e-commerce orders dataset using PySpark to filter, group, and compute revenue and time based insights for production pipelines.

Learn to write Spark programs in Scala, practice in the Spark shell, and explore immutable values, var, arrays, lists, maps, and loops for scalable distributed processing.

Convert PySpark transformations to Scala on Google Cloud Dataproc using ChatGPT, run on the e-commerce dataset, and compare PySpark vs Scala syntax for practical Spark proficiency.

Apply distributed data processing to cleanse bank marketing prospects by replacing missing values with column averages and removing unknowns, using Hadoop storage and Spark processing via PySpark and Spark Scala.

Set up a colab notebook with spark and load bank prospects csv. Cleansing: filter unknowns, convert age to integer, salary to float, and fill nulls with mean values, then export.

Execute PySpark transformations on a Dataproc cluster using Spark Submit, loading a bank prospects file from GitHub into HDFS, handling null values and missing records, and creating a Hive table.

Demonstrates a Spark Scala workflow on DataProc to transform a bank prospects project from Git to HDFS, compute average age and salary, and load a Hive table for reporting.

Explore Spark SQL and temporary views to run SQL queries on data frames, register temporary views, and perform joins, aggregations, subqueries, and CTEs using both SQL and the DataFrame API.

Practice converting Spark SQL to Scala and run on Google Cloud Dataproc, using Hive-compatible SQL, DataFrames, and translating business questions into fast, scalable data pipelines.

Develop a data pipeline with PySpark and ChatGPT, reading and transforming a CSV, applying filters and group by, and saving the results to CSV.

Learn to tune spark submit for performance by using ChatGPT to generate dataframe-based Python code, adjust executor memory, cores, and driver memory, and compare run times.

Explore Databricks and PySpark development with PyCharm, applying industry-standard error handling, logging, and code organization to build cloud data pipelines on AWS and automate workflows with Apache NiFi.

Explore how Databricks, built on Apache Spark, enables data engineering, analytics, and machine learning with a collaborative lakehouse platform and Delta tables for scalable data pipelines.

Discover Databricks free edition, a serverless unified data and AI platform built on Apache Spark. Sign up to access notebooks, serverless compute, and Python-based data pipelines, dashboards, and SQL support.

Explore the data workflow in Databricks by ingesting, reading, transforming, and writing data into delta tables with Unity Catalog, including schema evolution, time travel, and simple PySpark transformations.

Explore the Databricks file system dbfs and dbutils to access files programmatically, use sample datasets for practice, and read data into a spark dataframe from dbfs.

Unity Catalog volumes provide governance with access control, audit, and lineage across the catalog's three-level namespace; learn to create volumes, upload files, and read or write data via notebooks.

Load the Titanic dataset from dbfs into a dataframe, filter it, and save as a delta table using the Databricks JNI feature, with AI-generated code from prompts.

Explore no-code data engineering with Databricks generative AI for rapid data engineering, generating and running code, inspecting schemas, counting records, and storing results in delta tables via Unity Catalog.

Compare the Databricks community edition interface with the free edition; notebooks, code, and datasets work the same, while serverless versus cluster setups and Scala availability are highlighted.

Apply PySpark transformations on the Databricks diamonds dataset, including filter, group by, and order by, while learning Spark core and Spark SQL and the lazy, action-driven pipeline.

Explore sample transformations on Databricks using Spark with Scala and Python in a notebook, performing groupBy and avg on a DataFrame, with explicit val and var declarations and camel casing.

Explore Spark user defined functions (UDF) to create and apply custom Python functions to data frames or via SQL, using Databricks, UDFs, concat, lit, and lambda expressions.

Learn to join data from two delta tables using Spark DataFrame APIs and Spark SQL, join on customer ID, and aggregate by country to analyze spending.

Explore various Spark join operations, including inner, left, right, full outer, left semi, and left anti joins, using customer and transaction data frames loaded from delta tables.

Contrast transactional databases with data warehouses, then examine data lakes and lakehouses for storing structured to unstructured data, real-time analytics, governance, and scalable processing.

Discover how the Databricks lakehouse unifies Apache Spark and Delta Lake to store data as Delta tables on Parquet, with acid transactions and cloud storage integration.

Discover delta tables as core of delta lake, storing parquet data with versioning, time travel, and acid transactions, accessible via sql or spark and compatible with s3 or azure blob.

Learn to store data in a Delta table with Databricks SQL, perform time travel to view and restore previous versions, and query with Spark while applying transformations.

Compare spark sql and Databricks sql while creating a delta table from the diamonds data, and perform transformations with both interfaces using describe commands and the optimized engine.

Enable delta table caching in a Databricks cluster to speed up read-heavy queries by caching delta data in memory within the lakehouse.

Discover delta table partitioning that stores data in partitions by a column to speed queries, demonstrated on the New York taxi yellow data set with partitioning by payment type.

Explore g ordering, a delta table optimization that sorts data on disk by specified columns to speed up range queries, demonstrated on payment type with the optimize command.

Learn to write production ready Python Spark code outside notebook environments by organizing code, adding robust logging, and implementing error handling.

Install python, pycharm, and jdk on Windows, then configure spark with winutils for Hadoop and Hive. Connect PyCharm to PySpark and Hive, create dataframes, and query tables.

Follow along with a PyCharm project on Windows or Mac as the project structure remains identical, with differences only in where you add new libraries, per the PyCharm installation video.

Learn to structure a PySpark data pipeline with classes and methods, orchestrating ingest, transform, and persist stages in separate files and a central data pipeline module.

Create and reuse a SparkSession across a PySpark data pipeline by initializing it in a class, storing it as self.spark, and passing it to ingest and pipeline components.

Read a retail store CSV into a Spark dataframe, apply transformations, and inspect results with describe, using breakpoints in PyCharm to run and observe outputs.

Learn to separate ingestion, transformation, and persistence in a Python and Spark data pipeline, removing nulls, and persisting transformed data to CSV with controlled partitioning.

Learn how to implement Python logging using the built-in logging library, control log levels (debug, info, warning, error), and annotate method boundaries for clearer debugging.

Explore configuring log levels through a configuration file by creating a resources/configs directory for the log config, then use a root logger and console handler with a formatter in PySpark.

Create separate loggers for ingest, transform, and persist with a shared console handler; set per-file levels and disable propagation to avoid duplicate logs, enabling targeted production debugging.

Learn Python error handling with try and except, log errors, raise and propagate exceptions, and exit on failure while managing a data pipeline and persistence errors.

Explore using log4p and log4python for Python logging in a PySpark workflow, including installation, configuration, and switching from the core Python logger to file-based logs.

Ingest data from a Hive table using Spark, apply transformations, and prepare results for storage in PostgreSQL, building a pipeline with dummy Hive data and null handling.

Apply transformations to ingested hive data by replacing null author names with unknown and null review counts with zero, using transform.py and df.show to verify before storing in Postgres.

Install PostgreSQL on Windows and configure pgAdmin to support a big data Hadoop and Spark project for absolute beginners, including creating a schema, a table, and inserting records via SQL.

Learn to interact with PostgreSQL from PySpark using the Psycopg2 adapter, read data via pandas, convert to a Spark DataFrame, and insert new records.

Connect to PostgreSQL via a JDBC driver by downloading the jar, configuring the spark session, and reading data with spark.read.jdbc from the future X course catalog table.

Build a Spark data pipeline that reads from Hive, replaces nulls, and appends transformed data to a Postgres table via JDBC, illustrating scalable data transformations.

Organize code by creating a pipeline folder and moving ingest, transform, and persist into it, update imports and resources, and fix logging config and relative jar and data paths.

Read configurations from a properties file using Configparser to load the target table from the db configs section via config.get and log the value from pipeline/resources/pipeline.ini.