Data Engineering Master Course: Spark/Hadoop/Kafka/MongoDB

Sign in to Google Cloud, start the free trial with $300 credit, and create a new project in the cloud console to spawn your big data cluster.

Upload data files to a Google Cloud storage bucket, create the spark-data-frame bucket, and mount it on a virtual machine with GCS Fuse to access files locally.

Set up a MySQL server on the Dataproc cluster for Sqoop, create a scoop user and retail_db with full privileges, and import data to or from MySQL and HDFs.

Clarifies that the course can run on Cloudera or Google Cloud environments with identical content, and begins with an introduction to Sqoop.

Explore how Yarn cluster manages resources for mass produced jobs across a multi-node cluster. See how resource manager, node manager, and application master coordinate virtual machines and data locality.

Sqoop enables data migration between relational databases and the Hadoop ecosystem, importing from databases like MySQL to HDFS and Hive, and exporting back with the Sqoop export command.

Learn to change the warehouse directory and target directory for sqoop imports, creating and using hdfs paths so mapreduce jobs write part files to a specified location.

Apply gzip, snappy, bzip2, and lz4 compression when ingesting data into hdfs. Compare splittability, speed, and storage to choose the right codec for mapreduce and sqoop workflows.

Learn to perform selective Sqoop imports with where clauses, import only specific columns, and run free-form queries, including the required split by column parameter.

Explore how split by and boundary queries determine data distribution across mappers in Sqoop, using default primary keys or custom min/max boundaries, including scenarios without a primary key.

Learn how to customize field delimiters in data import with scoop, using a pipe delimiter and handling nulls differently for strings and non-strings.

Learn to perform incremental appends with Sqoop by fetching only new rows from a MySQL orders table into HDFS using a boundary query and last value, with Scoop jobs.

Copy and paste the beeline command to connect to the hive shell, using your username and cluster name, then execute hive queries on google cloud.

learn to import data from MySQL to Hive using Sqoop, create Hive tables, and verify with describe formatted, while adjusting field delimiter, mappers, destination directory, and compression.

Explore how Sqoop lists databases, tables, and schemas directly from the command line, using scoop list databases, scoop list tables, and scoop eval describe to inspect MySQL structures.

Import complete orders from the orders table in localhost MySQL database using Sqoop, writing a parquet file with tab-delimited fields and snappy compression; represent nulls as minus one and n.

Connects to a MySQL database with Sqoop to import customers with plaza in street name, selecting id, first name, last name, and street, and saves as star-delimited text in HDFS.

Explore sqoop export, moving data from hdfs to a relational database, exporting to mysql table customer_exported, using three mappers and handling nulls as empty or zero.

Export data from a hive table to a mysql table using sqoop export, demonstrating the full command, verification, and data transfer into retail_db.product_exported.

Learn Apache Airflow, an orchestration engine for scheduling and monitoring data pipelines with Python-based dags, and run Sqoop jobs via a Sqoop operator or Bash operator with cron expressions.

Download and untar the flume tar, then set environment variables in your bashrc. Source the bashrc to apply changes and run flume -nc to verify the 1.6.0 installation.

Learn how Apache Flume moves data reliably, scalably, and distributively from multiple sources to destinations like Hdfs, Kafka, HBase, or Hive, using agents, events, and versatile flow types.

Explore using the execution source to run a Unix command at startup and stream its output as logger events through a memory channel, with logger sync for testing.

Learn how to ingest data from Twitter and dump it into HDFS using Apache Flume, including Twitter source configuration, HDFS sink settings, and file rolling parameters.

Connect to a netcat source, configure a Flume agent with a file channel and an hdfs sink, and stream each netcat line into hdfs log files with rollover.

Explore flume interceptors that extend data ingestion with on-the-fly modifications: timestamp, host, static headers, remove header, uuid, morphine, search and replace, and regex based transformations.

Learn how to attach Flume interceptors to a netcat source, using regex, host, timestamp, and static interceptors, and observe the interceptors' execution order with a logger sink.

learn how to implement multi-agent flow in flume by connecting a netcat avro source to an avro sink through a memory or file channel, then dump data into Hdfs.

Enable data consolidation with Apache Flume by ingesting from multiple sources—netcat, spool directory, and X source—and dumping to a single Avro sink connected to HDFS.

Connect to the hive shell on google cloud using the beeline command, enter your username and cluster name to open the shell, and run hive queries.

Explore how Hive extends HDFS to run SQL-like queries with the Hive query language, offering schema on read and MapReduce execution on Hadoop.

Learn to create Hive databases and tables, including managed and external tables, and describe database extended with location and properties. Also learn to insert data and alter schemas, with cascade.

Create external tables by specifying a data location, allowing Hive to read text files from that path without loading data; dropping the table preserves the data, unlike managed tables.

Learn how to load data into Hive tables from delimited files and insert rows. Create and populate an external table by selecting data from an existing table, with MapReduce-backed inserts.

Explore hive analytics with sql-like queries using group by, order by, and aggregates on city data, powered by MapReduce, and apply having, case statements, and min, max, sum, avg.

Learn to create a hive table stored as parquet from an existing table, load data with a select statement, and verify parquet format, schema, and HDFS location using describe format.

Enable gzip or snappy parquet compression in hive by enabling compress output property and setting parquet compression, then create a parquet table and verify with parquet tools and describe formatted.

Read fixed-width data with a regular expression, load it into an external table using a regex SerDe, and map segments to employee_id, name, and age.

Master altering tables with the alter command, adding new columns, renaming columns and tables, setting properties, changing file formats, and updating table locations to reflect new schemas.

Master hive date functions to extract month, year, day, hour, minute, and second; compute date differences, add or subtract days and months, and work with current dates and unix timestamps.

Use hive partitioning to optimize queries by distributing orders data by status for faster results. Create an external partitioned table with dynamic partitioning and populate partitions with insert overwrite.

Learn how Hive bucketing uses a hash function on a chosen column to create ten buckets and bucketed tables, enabling efficient reads and data organization.

Learn about Apache Spark, a distributed framework that enables in-memory, parallel data processing. Focus on core features like Spark SQL and DataFrames, skipping ML libraries, graph processing, and streaming.

Learn how a cluster manager allocates resources, tracks node health, and schedules tasks across a master-slave cluster, enabling Spark apps on Yarn with HDFS.

Explore distributed storage basics, including how data splits into 128 chunks that form HDFS blocks, distributed across multiple machines and managed by name nodes and data nodes.

Submit a Spark application on yarn/hdfs and learn how the driver, application master, and resource manager allocate executors to read blocks managed by the name node and process data.

Understand Spark deploy modes: client mode, where the driver runs in the client, and cluster mode, where the driver runs inside the application master, with cluster mode recommended for production.

Learn how to run Spark jobs on Google Cloud Platform using Spark shell and Zeppelin notebook, including accessing the cluster via SSH and starting Spark context and Spark session.

Mount your Google Cloud bucket into a VM, create an HDFS directory for the Spark tutorial, and move your Spark dataset into HDFS with dfs -mkdir and dfs -put.

explain how the driver process analyzes code for syntax, builds logical and physical plans, and assigns tasks to executors, enabling four parallel tracks.

Explore Spark data structures, contrasting RDDs with data frames and data sets, learn to read JSON files into a dataframe using a Spark session, and compare performance and abstraction levels.

Explore how transformations create new data frames from existing ones and how actions trigger execution in Spark, highlighting immutable dataframes, lazy evaluation, and filter examples.

Compare narrow transformations like filter with wide transformations that require data shuffling and create stages. The lecture demonstrates group by and count by year, data shuffling, lineage, and writing output.

Discover how actions trigger jobs and split into stages and tasks, shaped by wide transformations, shuffles, and input partitions, and how the SD execution plan guides Spark UI execution.

Explore how Spark driver builds and optimizes execution plans—from deep based logical plans to analyzed, optimized, and physical plans—driving stage-by-stage execution of json reads and group by aggregations.