Teach on Udemy

Turn what you know into an opportunity and reach millions around the world.

Learn More

Your cart is empty.

Keep shopping

Hands-On End-to-End Big Data Projects

Name: Hands-On End-to-End Big Data Projects
Rating: 3.5 (5 reviews)

Work with Big Data Tools, SQL Databases, AWS, ETL, Data Integration Tools & more to master real-world Big Data Projects

Created byMD Imran

Last updated 2/2026

English

What you'll learn

How to Build a Scalable Data Pipeline using various Components
Data Warehouse Design
Data Preparation,Cleaning, Data Transformation and Manipulation
Industry Project Ready projects

Course content

5 sections • 112 lectures • 9h 38m total length

Exploration of the dataset4:11
Explore a sales data set by analyzing columns like region, country, item type, and sales channel, and learn to transform order date and ship date formats in a data pipeline.
Creating EMR Cluster6:52
Login into EMR part 18:20
Login into EMR part 22:36
Upload Data into Amazon S32:51
Create an S3 bucket, upload the sales data CSV, and prepare it for loading into the EMR cluster, setting up the extract and load steps before transformation.
using HIve as ETL Tool14:11
Hive Data Insertion2:05
Install Tableau Desktop6:27
Install Driver2:14
Connect Tableau to Amazon EMR Hive2:46
Add data schema and Table0:52
Set the default schema and final stage table, drag the final sales table onto the canvas. Preview data in Tableau with measures and dimensions.
plot charts in Tableau part 16:28
plot charts in Tableau part 29:27
plot charts in Tableau part 32:51
plot charts in Tableau part 44:38
plot charts in Tableau part 54:58
Building Dashboard and story14:58
Drag and drop charts to build a dynamic dashboard, set auto size, apply filters for interactive insights, and create stories with captions for presentations across the data source.
Resources0:02

Tour to Architecture diagram1:33
Exploration of the dataset2:52
Upload data to AWS S32:05
Create an S3 bucket named project three in the AWS management console, then upload the customer.csv file to store data in S3.
Create Postgresql in aws6:00
Download Pgadmin3:34
set up postgres sql and create schemas1:44
Create schemas and order table in your postgres instance10:38
set up infromatica cloud account4:57
Add S3 Connection part 110:31
Add S3 Connection part 20:45
Add postgres Connection2:48
Create customer Destination in Datawarehouse5:33
EL for aws s3 to data warehouse10:34
Create order Destination in Datawarehouse4:41
EL for app database to data warehouse4:39
Create DBT account1:47
dbt part 17:33
dbt part 27:01
dbt part 36:05
dbt part 42:00
dbt part 51:43
dbt part 66:32
dbt part 71:47
dbt part 88:32
Learn how dbt handles transformation within a PostgreSQL data warehouse by building staging and final models, validating with tests, and previewing results before connecting to Tableau.
Tableau and postgres set up2:33
How to Build charts in Tableau2:03

Tour to Architecture diagram3:15
Create EC2 Instance5:21
SSH into EC2 Instance2:14
Envirnoment setup with docker4:26
Copy Important folder from local to ec2 and give required permissions17:07
To connect to different services locally after port forwarding5:28
To get into bash shell of different containers7:46
Insert Nifi Template2:45
Upload and import the NiFi template, then explore the components and flow files to trace data movement from API retrieval through encryption, pre-processing, and outputs to Kafka and HDFS.
Data Extraction with Nifi11:10
Data encryption parsing9:43
Data sources hdfs kafka part 16:17
Push data into hdfs with the put hdfs processor and Hadoop configuration, automatically creating directories, while data also moves to kafka in parallel.
Data sources hdfs kafka part 26:03
Data sources hdfs kafka part 30:47
streaming data from kafka to pyspark5:40
pyspark streaming output kafka nifi hdfs part 14:52
Demonstrate a PySpark streaming workflow by deploying test.py to the Spark master, granting permissions, and pushing processed data into a Kafka topic named 'desired out', then verify the topic output.
pyspark streaming output kafka nifi hdfs part 27:36
Move Data HDFS to hive Table part 13:47
Move Data HDFS to hive Table part 23:07
Dataflow Orchestration with Airflow part 18:39
Dataflow Orchestration with Airflow part 21:01
Connecting with Data Visualization Tool1:41
Connect Tableau to Hive data using Hive connectors, enter server name, port, username, and authentication, then drag databases and tables to build visuals.
Plot charts1:00
Resources0:02

Project Architecture1:05
create ec2 instance3:33
ssh into ec2 machine1:32
Environment setup 14:16
Environment setup 20:50
Environment setup 36:02
Frequently used hdfs commands8:15
sqoop import part 18:38
sqoop import part 22:12
sqoop import part 39:06
sqoop export10:25
project part 18:32
Load Walmart store sales data from a csv into a MySQL container, then create a Hive table and analyze the dataset through a Docker-based Hadoop workflow.
project part 210:39
Navigate docker containers and mysql to load the Walmart sales dataset, create a retail database and Walmart_sales table, and import data with proper local files settings.
project part 31:25
project part 48:51
project part 53:18
Analysis Part 12:03
Identify the stores with the highest and lowest sales by summing weekly sales per store, rounding to two decimals, and ranking with dense_rank over total in descending order.
Analysis Part 20:46
Analysis Part 30:30
Analysis Part 41:48
Analysis Part 51:54

Project Architecture3:40
Explore the project architecture tying AWS services (EC2, Kinesis Firehose, S3) with Apache Airflow and Snowflake to move, transform, and load streaming data from landing to processed stages.
Creation of S3 Bucket1:57
Creation of IAM Policy4:40
Creation of EC2 Instance8:37
Creation of Kinesis Firehose Delivery4:34
Create two Kinesis Firehose delivery streams to load customers and orders data into the Snowflake Data Pipeline S3 bucket, using direct put as the source and landing and error prefixes.
Installing Kinesis Agent and Sending data to Firehouse part 114:57
Installing Kinesis Agent and Sending data to Firehouse part 25:53
Create Snowflake Account3:20
Setting up Snowflake Databases and Table Part 115:41
Setting up Snowflake Databases and Table Part 22:09
Create an AWS IAM role to connect Snowflake with S3, set the external ID, and attach S3 full access, then review the trust relationship before configuring a stage.
Setting up Snowflake Stages Part 18:49
Setting up Snowflake Stages Part 25:09
Understanding AIrflow DAG Part 115:07
Learn to define an Airflow dag with bash and Snowflake operators, configure default arguments, and move data from landing to processing to processed in s3 and Snowflake.
Understanding AIrflow DAG Part 24:43
Creating Airflow Cluster Part 16:17
Creating Airflow Cluster Part 22:36
Updating Airflow Role11:27
error fixing and running dag4:52
Fix table naming and account configuration in Snowflake, run the dag, and validate end-to-end data flow from S3 landing to processing, with logs and verifications.
Snowflake DB revision and Cleanup7:06
error found and fixing0:48
Creating Transformation in snowflake3:21
updating dag3:34
Running Airflow DAG with Transformation4:52
Learn how to run an airflow dag with data transformation, fix SQL errors, and verify results through logs, table creation, and a data build tool workflow.
checking the result1:58

Requirements

It is also beneficial to have prior knowledge of SQL, programming basics, data pipelines and ETL concepts

Description

The Big Data Projects course is designed to provide students with an in-depth understanding of the various tools and techniques used to handle and analyze large-scale data. The course will cover topics such as data preprocessing, data visualization, and statistical analysis, as well as machine learning and deep learning techniques for data analysis.

Throughout the course, students will be introduced to the Hadoop ecosystem, including technologies such as Hadoop Distributed File System (HDFS), MapReduce, and Apache Spark. Students will also gain hands-on experience working with big data tools such as Apache Hive, Pig, and Impala.

At the end of the course, students will have the necessary skills and knowledge to handle large-scale data and analyze it effectively. Students will also have a solid understanding of the Hadoop ecosystem and various big data tools that are commonly used in the industry.

A real data engineering project usually involves multiple components. Setting up a data engineering project, while conforming to best practices can be extremely time-consuming. If you are

A data analyst, student, scientist, or engineer looking to gain data engineering experience, but are unable to find a good starter project.

1. Wanting to work on a data engineering project that simulates a real-life project.

2. Looking for an end-to-end data engineering project.

3. Looking for a good project to get data engineering experience for job interviews.

Then this Course is for you. In this Course, you will

Learn How to Set up data infrastructure such as Airflow, Redshift, Snowflake, etc
Learn data pipeline best practices.
Learn how to spot failure points in data pipelines and build systems resistant to failures.
Learn how to design and build a data pipeline from business requirements.
Learn How to Build End to End ETL Pipeline
Set up Apache Airflow, AWS EMR, AWS Redshift, AWS Spectrum, and AWS S3.

Tech stack:

➔Language: Python

➔Package: PySpark

➔Services: Docker, Kafka, Amazon Redshift,S3, IICS, DBT Many More

Requirements

This course presume that students have prior knowledge of AWS or its Big Data services.
Having a fair understanding of Python and SQL would help but it is not mandatory.

Every Month New Projects will be added

Who this course is for:

People with some software background who want to learn the New technology in big data analysis will want to check this out. This course focuses on Various Big data Tools; we introduce some Data Engineering and data Science concepts along the way, but that's not the focus. If you want to learn how to Build Data Engineering Projects , then this course is for you.
Data analysts and Data Engineer who are curious about Big Data Tools and how it relates to their work.

Hands-On End-to-End Big Data Projects

What you'll learn

Explore related topics

Course content

Build ETL Data Pipeline on AWS EMR Cluster18 lectures • 1hr 37min

Build Modern ETL Data Pipeline using IICS26 lectures • 2hr 1min

Create A Data Pipeline based on Messaging Using PySpark and Airflow23 lectures • 2hr

Build Big Data Project using Sqoop, HDFS and Hive21 lectures • 1hr 36min

Build Data Pipeline using AWS, Snowflake, Kinesis and Airflow24 lectures • 2hr 26min

Requirements

Description

Who this course is for: