Data Engineering Bootcamp

Understand what data engineering is and why it’s an in-demand tech role, and learn to turn data from websites, apps, IoT, and social media into structured, semi-structured, and unstructured insights.

Explain how data engineers build the data pipelines to extract, load, and transform data, and compare etl with elt for flexible, cloud-based analytics.

Explore the modern data engineering ecosystem centered on ELT, covering data sources, ingestion, data lake and warehouse storage, transformation, orchestration, and data consumption with real-world examples.

Data engineering matters for business success by delivering clean, consistent data and enabling real-time decisions, advanced analytics, and faster market responses.

Data engineers are infrastructure specialists who build robust, scalable pipelines used by data analysts, data scientists, and machine learning engineers, with overlap across roles and SQL or Python skills.

See how data engineering powers real-time event capture, fraud detection, inventory updates, and personalized recommendations in a large e-commerce setting, directly impacting revenue.

Install git on your local system to clone the repository and begin work. Learn to download git, run the installer with default settings, and use git bash if desired.

Clone the public course repository to access the data pipeline structure, including config, infrastructure, src, tests, and data. Install VSCode and git, then clone and explore the project in VSCode.

Install the Python dependencies from the requirements.txt, including pandas, boto3, prefix library, and python dot env, using a virtual environment to keep projects isolated.

Download and install Docker Desktop for your operating system, choosing the correct Mac, Windows, or Linux installer; select a backend (WSL recommended) and verify the Docker engine is running.

Learn to build a data engineering pipeline: set up data, configure AWS, ingest to a data lake, preprocess into analytics-ready formats, and automate with orchestration.

Explore a synthetic e-commerce data set with csv, parquet, and json formats for customers, orders, products, order items, and reviews, and learn cloud upload steps for data lake readiness.

Create an AWS account, verify your email, enter personal and payment details, and sign up for basic free-tier access to the AWS console.

connect your local repository to AWS by creating an environment file with AWS credentials, including access key, secret key, region, and an S3 bucket for data lake pipelines.

Create an S3 bucket for the data lake, choose a unique name in us east one, and apply defaults like blocking public access and storing objects in the bucket.

Learn to test an AWS connection by writing a Python script that reads credentials from a dot env file, connects to S3 with boto3, and lists buckets.

Create a Python data ingestion script to upload raw CSV files from local storage to an AWS S3 bucket using boto3 and pathlib, with environment variables for credentials.

Define the csv data path with pathlib, locating data/raw relative to the repository. Apply defensive checks for existence and prepare to list csv files before uploading to an s3 bucket.

List CSV files from the data folder using a glob pattern, and add defensive checks for missing CSV files while preparing to upload files to S3 in the next tutorial.

Upload CSV files to a structured S3 raw data folder in the data lake, track progress with a counter, handle errors without stopping the pipeline, and report final upload results.

Verify uploads to S3 after data ingestion by listing bucket contents and calculating total data size in megabytes. Create a verification function reading from S3, printing progress, and handling errors.

Run and verify the data ingestion pipeline by uploading raw csv files to S3 and confirming file keys. Activate the virtual environment and review the verification step and data processing.

Explore data processing in a data engineering bootcamp by building ETL pipelines and comparing ETL with ELT, using Python for local extract, transform, and load from an AWS data lake.

Describe the etl pipeline structure, outlining four steps: download data from s3, transform data, create business metrics, and upload processed data back to s3, using pandas, boto3, dotenv, and DateTime.

Build an etl pipeline extract function that downloads csv files from s3, loads them into data frames, and organizes them in a data sets dictionary for later transformation.

Transform raw csv data into a clean, standardized customer dataset by applying defensive checks, email lowercase and strip, date conversion with pandas, and age group creation.

Apply the same etl pipeline to the products data, clean product names, convert prices to numeric, and create price categories (budgets to luxury) for marketing; generalize to other data frames.

Build ETL pipelines for orders and order items, cleaning data, converting order dates to datetime, and turning totals into numeric values, while engineering temporal features like year and month.

In this ETL pipeline lesson, you clean and transform the review data by converting dates and ratings to numeric, then create rating categories (poor to excellent) for business intelligence insights.

Explore how to extend etl pipelines into business metrics by transforming cleaned data into customer intelligence, including lifetime value, order metrics, and age group segmentation for insights.

Execute the product performance metrics exercise by downloading the hint PDF and following five steps, mirroring the customer workflow. Compare the code with the solution in the next video.

Validate datasets, aggregate by product for units sold, revenue, and orders; enrich with product data and store in a product metrics set to analyze category revenue.

Develop a monthly sales trend dataset that captures revenue and order volume over time, applying data validation, multi-level grouping, and time-series aggregation with business-friendly names and storage validation.

Validate and process data, group by order year and month to compute monthly revenue and order counts, then save as monthly sales and prepare to load to AWS S3.

Learn to complete the ETL cycle by building a function that uploads processed data and business metrics to AWS S3, with error handling, temporary CSVs, and clean-up for analytics-ready insights.

Upload datasets and business metrics to S3, implement exception handling with logs, track upload counts, and structure S3 keys for a reliable ETL workflow.

Execute and debug the etl pipeline, from downloading data from s3 to transforming it, creating business metrics, and uploading results back, then orchestrate with prefect.

Explore orchestration of data pipelines with Prefect, learning flow and task concepts, and implement a hello world Python example before applying scheduling, dependencies, and monitoring to a real ETL project.

Explore how prefect orchestrates a simple hello world function using flow decorators, automatic logging, and a dashboard to monitor flows, deployments, and tasks.

Explore how flows act as containers and tasks as units, enabling selective retries, reusability, and easier debugging with parallel execution in an etl pipeline.

Initialize a prefix YAML deployment file, configure a basic flow with a hello world task, and deploy it to the dashboard to prepare the orchestration for execution.

Learn to create and deploy a flow using a work pool in Prefect, linking a hello world task to a job board, workers, and a YAML-defined schedule every minute.

Learn to orchestrate a data engineering ETL pipeline with Prefect, implementing S3 download, data transformation, business metrics, and S3 upload.

Build and deploy a data orchestration workflow by creating a work pool, defining workers and deployments, and automating etl tasks with a Prefect flow and yaml configuration.

Learn how containerization with Docker solves the 'works on my machine' problem by packaging code, dependencies, and system libraries into portable containers for consistent development, testing, and production.

Learn how to containerize an end-to-end data pipeline using docker file and docker compose in a data engineering bootcamp, covering images, containers, dependencies, and health checks.

Run and orchestrate the data pipeline inside a Docker container by validating configurations, starting Docker Compose, and executing the prefect workflow to process data, metrics, and S3 uploads.