Data Engineering using Kafka and Spark Structured Streaming

Master Kafka, a distributed event streaming platform for publishing and subscribing durable event streams. Explore topics, partitions, fault-tolerant, and deployment options from on premises to cloud.

Manage kafka topics via kafka-topics.sh to list, drop, and create topics using login user prefixes on multi-node clusters, and describe metadata to inspect partitions, replication, and leaders.

Demonstrates creating a demo topic and producing and consuming messages with kafka-console-producer.sh and kafka-console-consumer.sh, including single-node and multi-node CLI setups and from the beginning.

Validate the gen_logs web server log generator in a single-node cluster by starting the generator, writing to access.log, and tailing updates to verify streaming.

Create a minimal web server with netcat and redirect Logstash output from tail logs into the server, simulating website traffic for Kafka and Spark structured streaming.

Produce retail logs to a Kafka topic using the console-producer.sh and tail_logs.sh, validate with a console-consumer, and understand when to use cli tools vs production applications.

Consume retail logs from a kafka topic using the kafka console consumer script, with from-beginning options and single-node or multi-node cluster demonstrations for learning or validation.

Clean up Kafka CLI sessions by terminating the console producer and consumer with ctrl-c, ensuring a clean exit before exploring Kafka Connect for producing and consuming messages.

Prepare Kafka Connect properties to stream data from log files into a Kafka topic using standalone mode. Set string converters and offset storage to ensure reliable data flow.

Configure and run kafka connect to produce messages from log files into the itversity_retail topic, updating properties and then validate with a consumer.

Explore Kafka Connect, a Kafka ecosystem component that ingests data from sources into Kafka topics, moves data between topics, or writes to targets such as HDFS and databases, without coding.

Configure Kafka Connect in standalone mode to publish access.log data to the itversity_retail topic, using the retail_log_standalone.properties and retail_log_files_source.properties files, with bootstrap servers, offsets, and rest port.

Start and validate the Kafka Connect process using standalone properties, review topic creation and offsets, and confirm message production with the Kafka console consumer.

Clean up the Kafka connect process to stop producing messages from the file to the Kafka topic, release resources, and verify no active processes writing to the topic.

Set up the HDFS sink plugin for Kafka Connect, configure a standalone properties file with the source topic and HDFS target, start Kafka Connect process, and validate files in HDFS.

Install the hdfs3 sink connector plugin for Kafka connect via Confluent Hub and deploy it to the server, enabling writing from Kafka topics to HDFS.

Explore Kafka consumer groups that enable scalable, mutually exclusive data processing from a topic using Kafka Connect or Spark Structured Streaming, with offsets managed at the group level.

Configure HDFS 3 sink properties to stream data from Kafka to HDFS using the kafka-connect-hdfs plugin. Learn to set up standalone properties, brokers, topics, and the HDFS sink configuration.

Start a Kafka Connect standalone process with the HDFS sink, and validate that messages flow from the Kafka topic to HDFS, inspecting topics and consumer groups.

Terminate the Kafka connect process, stop producer and consumer sessions, delete the topic and consumer groups, and remove the HDFS sink data folder to complete cleanup.

Understand the streaming context in Spark, compare it with the batch Spark context, and see how keeping the Spark context alive between triggers handles overhead for frequent streaming jobs.

Validate a high-quality log dataset for data engineering using Kafka and Spark Structured Streaming by using the gen_logs generator and confirming access.log updates under opt/gen_logs/logs, with troubleshooting as needed.

Push log messages to a Netcat web server, validate live generation with tail_log.sh on the access.log, and prepare the stream for consumption by Spark Structured Streaming.

Explore built-in input sources in spark structured streaming, including file source, kafka source, socket source, and rate source, to power batch and streaming pipelines from files or kafka.

Explore reading web server logs with Spark structured streaming, using a socket source for learning with netcat web server logs. Then write results to the console with a five-second trigger.

Explore Spark structured streaming output modes: append, complete, and update—and learn when to use each mode after reading from source and writing to the sink, including aggregations.

Learn how append as output mode works in Spark structured streaming, demoing console output with filter and select, and why append fails with streaming aggregations without a watermark.

Explore using complete as the output mode for streaming aggregations in Spark Structured Streaming, comparing it with append and highlighting how to build department counts from log messages.

Explore triggers in Spark structured streaming, including unspecified micro-batch, fixed interval micro-batches, one-time micro-batch, and continuous with fixed checkpoint interval (experimental). See how processing time governs batch scheduling.

Explore built-in output sinks in spark structured streaming, including console, memory, file, Kafka, and Foreach sinks, and learn their append, complete, and update modes, debugging use, and production pipelines.

Create and manage a Kafka topic named itversity_retail by deleting and recreating it, then inspect partitions, replication, and leadership to support streaming with Spark structured streaming.

Read data from a Kafka topic using Spark Structured Streaming, configuring bootstrap servers and topic subscription, and inspect the streaming data frame schema (key, value, topic, partition, offset, timestamp).

Preview kafka data with spark structured streaming by creating a streaming dataframe and displaying key and value as strings on the console every five seconds.

Preview streaming data from a Kafka topic using Spark Structured Streaming memory by registering a query name and running Spark SQL queries to view records and counts.

Transform streaming web log data with Spark APIs to derive year, month, and day from timestamps, using Spark Structured Streaming to read from Kafka and partition output by date.

Learn to write streaming data from Kafka to HDFS using Spark Structured Streaming, partitioning by year, month, and day, with CSV output and HDFS checkpointing.

Validate streaming data written to HDFS via Spark structured streaming by inspecting data and checkpoint folders, review offsets and sources, and note partitioning by year, month, and day.

Write streaming data from a Kafka topic to HDFS in CSV format with header and tab separator using Spark Structured Streaming, partitioned by year, month, and day.

Stop the Kafka Connect process, clean up the HDFS CSV data partitioned by year, month, and day, and delete the offset file and checkpoints to release resources.

Discover how to process data incrementally with Spark Structured Streaming by using triggers: continuous (experimental), processing time, and trigger once, plus fixed interval and one-time micro-batches.

Read JSON data with readStream and infer the schema. Incrementally write to parquet on HDFS using triggers (once or processing time), stage locally, and partition by year, month, day.

Create a working directory in hdfs to support streaming pipelines, including landing and bronze folders, and prepare for Spark Structured Streaming that partitions data by year, month, and day.

Upload gharchive files from web to hdfs by downloading with requests, writing to local storage, and using hdfs put to organize by year, month, day for streaming.

Download gharchive files to local storage, create hdfs directories, and upload via hdfs dfs -put, then ingest 24 hourly files with a reusable function for spark structured streaming.

Upload gharchive files to hdfs by downloading from an external web service, staging locally, and moving to hdfs with a reusable function, enabling incremental loads via Spark structured streaming.

Learn to read json data using spark structured streaming with spark.readStream, format options, and common controls such as path, maxFilesPerTrigger, latestFirst, maxFileAge, clean source, and schema inference.

Implement spark structured streaming to read from a source and write parquet files to hdfs, partitioned by created year, month, and day, using trigger once and source delete with checkpointing.

Validate gharchive parquet data with spark by reading partitioned files, creating a dataframe, and validating counts and schema. Use spark sql to analyze newly created repositories and counts.

Demonstrates incremental data loading with Spark Structured Streaming by adding 2021 January 14th files, copying to hdfs, and validating incremental processing via the streaming APIs.

Load data incrementally into a target table using spark structured streaming, partition by date, and write parquet files with trigger once and clean source delete for automatic processing.

Validate incremental load using Spark structured streaming by reading parquet data and confirming counts across two days, with created year, month, and day fields aligning by date.

Process gharchive data for 2021-01-13 to 2021-01-15 by uploading files to hdfs and running an incremental load with spark structured streaming, as you explore additional options.

Leverage maxFilesPerTrigger and latest first in Spark structured streaming to limit processed files, prioritize latest data, and manage baseline and spike loads with processing time triggers.

Validate incremental load using spark structured streaming by counting daily records and verifying created year, month, and day fields across 2021-01-13 to 2021-01-15, then plan archiving for 2021-01-16.

Process and archive new gharchive files with spark structured streaming, downloading and uploading 2021-01-16 data to hdfs, then validate incremental processing workflows.

Demonstrates incremental load with spark structured streaming by archiving processed files via sourceArchiveDir, configuring maxFilesPerTrigger and writing parquet with created year, created month, and created day for validation.

Validate incremental load by inspecting the target for 2021 January 13–16, review counts and schema, and confirm read/write of data using spark structured streaming from files to files.

Learn to set up a cloud-based development environment with AWS Cloud9, access a browser-based IDE to write, run, and debug code, and create a Cloud9 workspace in the AWS console.

Create a Cloud9 environment in AWS by selecting an environment type and t3.medium on ubuntu server 18.04, with cost-saving settings.

Explore the Cloud9 IDE interface, access and customize the terminal, create and run Python files, and leverage GitHub, AWS explorer, and Docker for data engineering practice.

Explore how EC2 powers Cloud9, the IDE on an EC2 instance, with automated idle shutdown to save costs. Navigate the AWS console to view and manage the Cloud9 EC2 instance.

Open the Apache2 web server on a Cloud9 EC2 by configuring security group inbound rules for http (port 80), using the DNS or IP, and enabling an elastic IP later.

Learn how to associate an Elastic IP with a Cloud9 EC2 instance to maintain a static public IP and DNS for web services, even after restarts.

Increase the Cloud9 EC2 attached EBS volume from 10 GB to 32GB by modifying the volume in the EC2 console, then reboot the instance to apply the change.

Set up Jupyter Lab on a Cloud9 instance by creating a Python 3 virtual environment, installing Jupyter Lab with pip, and running it on port 8888 with security group access.

Sign up for Google Cloud Platform to access a $300 credit for one year and practice with Linux, Python, and SQL; sign in with Gmail and activate the free trial.

Navigate the GCP web console to provision, stop/start, and manage virtual machines, review billing credits, and configure firewall rules for Compute Engine VM instances.

Provision an Ubuntu 18.04 VM on Google Cloud, configure a static external IP, and access it via the web console to start, stop, and manage costs.

Set up Docker on Ubuntu 18.04 across cloud VMs or bare metal, including apt updates, Docker CE installation, and validation with a hello-world container.

Validate python on ubuntu 18.04 servers by confirming python3, installing pip with apt, and testing library installs like config parser; enable venv creation and jupyter setup for data engineering workflows.

Set up a Jupyter lab on an Ubuntu VM in google cloud by creating a Python virtual environment, installing Jupyter lab, and binding to 0.0.0.0 via firewall port 8888.

Set up JupyterLab on a Mac by creating a Python 3 virtual environment, installing JupyterLab with pip, and launching a browser-based notebook experience from the terminal.

Learn to set up a single node Hadoop cluster on Ubuntu 18.04, prerequisites, passwordless login, Hadoop installation, and configuring HDFS and YARN, with cloud options on gcp, aws, or azure.

Set up a single-node hadoop and spark cluster on Ubuntu 18.04 in gcp by installing wget and jdk, validating jre and javac, and accessing the VM via terminal or browser.

Set up password less login on the host by generating ssh keys and updating authorised keys, enabling seamless login to start Hadoop components for HDFS and YARN.

Download and install Hadoop 3.3.0 on a GCP single-node cluster, unpack the tarball, move to /opt, set ownership to the itversity user, and configure HDFS and YARN.

Configure hdfs in a single-node cluster by editing core-site.xml and hdfs-site.xml, setting dfs.replication to one, updating profile and java home, then format and start hdfs for validation.

Update path for hadoop bin and sbin, enable passwordless login, start hdfs with start-dfs.sh, validate with jps and hdfs commands; create /user/$user, copy files, view core-site.xml, then prepare yarn.

Configure yarn as part of a single-node Hadoop cluster by updating yarn-site.xml and mapred-site.xml, then start and validate yarn components.

Start yarn with start-yarn.sh and validate components via jps, confirming ResourceManager and NodeManager alongside NameNode, SecondaryNameNode, and data node on this single-node cluster, with PATH and passwordless login configured.

Learn to manage a single node Hadoop on Ubuntu 18.04 in GCP, shutting down Yarn first, then HDFS, to prevent corruption and save costs, and starting HDFS before Yarn.

Download and install Hive on Hadoop, configure metastore, and start and validate Hive in a single-node cluster.

Set up a postgres database in docker for hive metastore, expose port 6432, create metastore and hive user, grant permissions, install the postgres client, and configure hive with environment variables.

Configure and set up hive metastore with postgres, creating hive-site.xml, initializing the schema, and validating metastore tables to enable hive and spark sql metadata management.

Launch the hive cli, create retail_db, a four-column orders table, load data from /data/retail_db/orders into hdfs, and validate with a mapreduce-backed query showing the first 10 records and count 68883.

Learn to manage a single-node data engineering cluster by creating start and stop scripts for Hdfs, Yarn, and the Postgres container, then validate with jps and Hdfs commands.

Download Spark 2.4.7 on a node Hadoop cluster with wget and tar, move to /opt, and create a Spark 2 symlink, since it’s pre-built for Hadoop 2.7 and not production.

Configure spark 2 on a Hadoop cluster by setting yarn mode and Hadoop paths. Integrate Hive metastore with a hive-site.xml symlink and update the Postgres JDBC driver.

Validate Spark 2 using CLI, Pyspark, and Spark SQL to confirm Hive metastore integration and run queries on retail_db.orders.

Activate the virtual environment and set up Jupyter to integrate Spark 2 with Python in pyspark 2, enabling interactive Spark sessions in Jupyter Lab.

Download and install Spark 3.1.1 for Hadoop 3.2 in a single-node cluster, using wget to untar and unzip, move to /opt, create a symlink, and validate via the CLI.

Configure spark 3 on a single-node cluster by integrating with hadoop, enabling yarn mode, updating spark-env.sh and spark-defaults.conf, and disabling hive metastore schema verification in hive-site.xml.

Set up spark3 on a single node cluster, validate spark 3.1.2 with Scala, Python, and SQL CLIs, and verify Hive metastore integration by querying databases and tables.

Integrate Spark 3 with Jupyter Lab to run PySpark3 in a virtual environment, create a new kernel, and verify Spark SQL queries on a single-node cluster.