Real-World Data Engineering: Streaming & Cloud Projects

Develop a real-time delay detection pipeline with Kafka and Spark that combines delivery events with weather and traffic data to compute driver fault percentage and actionable delay insights.

Organize streaming data project by sourcing schemas, setting up VS Code and Kafka, and ingesting delivery, traffic, and weather data to compute weighted driver score and store results in parquet.

Set up a VS Code project: create folders for data, checkpoints, and outputs, initialize git, configure a virtual environment, and install Kafka and pandas, plus a producer script.

Develop a Kafka producer to ingest csv data by converting it to a pandas dataframe and json, then stream each row to the deliveries topic. Validate delivery with a consumer.

Understand how a Spark pipeline consumes Kafka messages, deserializes with a predefined schema, and computes a driver fault percentage from delay, traffic, and weather scores.

Read from Kafka deliveries using Spark structured streaming, deserialize JSON payloads into a delivery schema, and filter records with non-null pickup and expected times.

Cast all time fields to strings, convert to timestamps, load weather and traffic data, and enrich deliveries with spark broadcast joins on city, truncating times to the hour for streaming.

Extend a Spark enriched data frame by truncating timestamps, joining weather and traffic, and adding metrics: is late, delay minutes, delay category, and driver fault percentage; then write to parquet.

Run the end-to-end data pipeline by starting ZooKeeper and Kafka, creating the deliveries topic, streaming data with producer, and executing analytics with Spark submit to produce and inspect parquet outputs.

Build a real-time fraud detection pipeline with flink guard, using Apache Flink and PyFlink to ingest Kafka data, detect location shifts and spending spikes via stateful processing, and emit alerts.

Explore Apache Flink, a stream-first open-source framework for real-time and batch processing of bounded and unbounded data, with stateful processing, data stream API, and checkpointing.

Set up windows subsystem for linux with debian, install java jdk 11 and python, and add tools like wget, git, curl, and llvm. Configure bashrc for environment variables.

Set up your data engineering environment by updating bash rc, installing Python and a virtualenv, then installing Apache Beam, Apache Flink, numpy, and arrow.

Set up and configure Apache Flink and Kafka in an environment, install Kafka 3.3.1 and Flink connectors, adjust Flink conf to 0.0.0.0, start the cluster, and access the Flink dashboard.

Configure the Kafka producer by setting server.properties, advertising the listener to the PC IP (192.168.1.125) on port 902, and verify end-to-end data flow via console producer and consumer.

Explore building a streaming data pipeline using Kafka producers, CSV to JSON conversion, and Flink for fraud detection, with real-time data flowing from CSV to Kafka to Flink.

Develop a Flink script to process Kafka data with PyFlink in WSL and VS Code, defining a schema and parsing JSON, while applying haversine distance in streaming.

Define a json schema with row and column types, parse kafka data into flink streams, apply a watermark strategy, and key by user id for stateful processing.

Learn to implement per-user stream processing in Flink by applying the Haversine distance formula in a Python UDF, detecting 500 km location jumps and high amount spikes as fraud alerts.

Run a streaming fraud-detection pipeline with Kafka, Zookeeper, and Flink, processing a prepared CSV to detect fraudulent patterns and verify results via job logs.

ShelfSync provides an end-to-end data pipeline that automates retail sales and inventory insights using Apache Airflow, Nifi, Spark, and Hadoop HDFS with parquet.

Learn to plan and implement a shelf sync data pipeline using Jira tasks, Kanban, and a practical stack with Apache Airflow, NiFi, HDFS, and Spark, from research to deployment.

Orchestrate complex data pipelines with Apache Airflow, an open source workflow orchestration tool for scheduling, coordinating, and monitoring DAGs as Python code.

Install and configure Apache Airflow in a virtual environment, including Python 3.11 prerequisites, dependencies, database initialization, user setup, and launching the web UI and scheduler.

Apache NiFi enables real-time, no-code data flows with provenance and scalable, secure data movement across on-prem and cloud environments.

Install and configure Apache NiFi on Debian/WSL with Java JDK 11, adjust nifi.properties for http, start NiFi, and explore the UI to publish to Kafka.

Explore the Hadoop distributed file system (HDFS) master-slave architecture, including the name node and data nodes, and learn how threefold replication enables fault tolerance and high throughput for data workloads.

Install java jdk 8, extract and move hadoop to /usr/local/hadoop, configure core-site and hdfs-site, format and start namenode and datanode, and verify via hdfs dfs.

Learn to build an Airflow DAG that generates and aggregates simulated store inventory and sales data, coordinates NiFi for data movement to HDFS, and prepares for later Spark processing.

Create a daily airflow dag for a sales pipeline that generates sample data for ten stores and fifty items, writes sales and inventory data to csv, and moves files.

Start the Airflow web server and scheduler, verify DAGs, and monitor tasks to generate and move simulated data from tmp retail data to NiFi and HDFS.

Build a NiFi ingest to HDFS workflow by creating a process group with list file, fetch file, and put HDFS to move csv files from a local directory to HDFS.

Trigger a daily airflow dag to move data from local tmp to NiFi, push to HDFS, verify provenance, and prepare a Spark job to read from HDFS into data/process_sales.

Learn to build Spark-based data pipelines by reading CSV inventory and sales data from HDFS, aggregating it, and writing results back to HDFS.

Build a Spark pipeline that reads sales and inventory data, casts types, joins on date, store_id, and item_id, and computes total sales, total quantity, and stock per day.

The lecture demonstrates creating an HDFS output path, partitioning by date, writing parquet with overwrite mode, running spark submit, and converting the data to csv for import via NiFi.

Demonstrate end-to-end data movement from HDFS to local using NiFi, building an HDFS-to-local process group with list HDFS, fetch HDFS, and put parquet files.

Orchestrate a robust streaming pipeline by coordinating Airflow, Nifi, and Spark to generate, move, and process sales and inventory data through HDFS, parquet, and local directories.

Boot airflow, nifi, and hdfs in the wsl debian environment to start the pipeline. Check the data flow from inventory to sales csvs via hdfs and the airflow dag.

Run the spark submit script to process data, write snappy parquet to HDFS, fetch it locally, review data provenance with a parquet viewer, and observe NiFi and Airflow workflows.

Develop a serverless predictive maintenance pipeline using AWS S3, Athena, and optional Glue Data Catalog to compute remaining useful life from NASA Cmaps FD001 jet engine sensor data.

Discover AWS and cloud computing fundamentals, including pay-as-you-go storage, compute, and databases. Learn beginner-friendly AWS services like S3, EC2, Lambda, RDS, and Athena.

Set up an AWS free-tier account, configure MFA and IAM, create an S3 bucket, and set up Athena with a Glue catalog database for queries.

Download the Cmap FD 001 data from data.gov, keep only FD 001, then convert to a CSV with Python for S3 upload and Athena-based rolling-window rule computation across 21 sensors.

Explore Amazon S3, the simple storage service with buckets, objects, and regions, and learn how durability, cross-region replication, and pay-as-you-go pricing power backups, logs, ML data, and analytics.

Convert the data to CSV, upload it to a new S3 bucket named cmap sensor data, create raw data and output folders, then start Athena to query results.

Amazon Athena, a serverless SQL query service that reads data directly from S3 using the AWS Glue Data Catalog for schema, with no infrastructure to manage.

Create an external table in Amazon Athena from CSV data on S3, using database Cmap and table Clean Synapse, with engine_id and 21 sensor readings as doubles, and run queries.

Compute remaining useful life (rul) as the max cycle per engine minus current cycle. Create a view using max cycle over partition by engine id to compute the average rul.

Compute per-engine average rule with window by engine ID. Create rolling average and rolling max for five cycles; save as snappy parquet to S3 and validate in Athena.