
Welcome to Data Engineering: From Zero to Hero, a hands-on course that takes you from the basics to building real data pipelines. Find out what you will build, which tools you will use (SQL, Python, Snowflake and Apache Airflow) and how to get the most out of each section.
Meet your instructor, a data professional with 10+ years of experience as a data analyst, data scientist and data engineer. See who this data engineering course is for, from complete beginners and career changers to junior data engineers, and what you will be able to do by the end.
A walkthrough of the full data engineering curriculum: the data ecosystem, data sources and destinations, data modeling, SQL with Snowflake, data wrangling with Python, and pipeline orchestration with Airflow. Use it as a roadmap to plan your learning path.
The software and free accounts you need to follow along: Python, VS Code, a virtual environment, Jupyter Notebooks and a free Snowflake trial. Everything in the setup lectures is optional, so you can skip what you already have installed.
tep-by-step guide to installing Python on MacOS for data engineering work. You will install the latest Python 3 version, check it from the terminal and get ready to run the scripts used throughout the course.
Install Visual Studio Code on MacOS and set it up as your Python and SQL editor. Learn which VS Code extensions help data engineers write, run and debug code more efficiently.
Learn how to create and activate a Python virtual environment (venv) on MacOS. Virtual environments keep each data project's libraries isolated, a best practice every data engineer should follow.
Install the core Python libraries for data engineering, such as pandas and requests, inside your virtual environment using pip. You will also learn how to keep a requirements file so your setup is reproducible.
Run a quick test script to confirm Python, VS Code, your virtual environment and pandas all work together on MacOS. Fix common setup errors now so nothing blocks you in the hands-on lectures.
In this lecture, I go through Jupyter Notebooks, a useful tool for a Data Engineer
An overview of the introduction to data engineering: the world of data, the main data roles, what data engineering is, what a data engineer does day to day, and the essential data engineering skills and tools.
A clear definition of data engineering: moving data from where it is created to where it is used, efficiently and without compromising data quality or integrity. Learn why data engineering is one of the most in-demand fields in analytics and why reports and dashboards depend on it.
What a data engineer actually does: building, maintaining and managing data pipelines that move data from operational systems to analytics. Through real scenarios (a new dashboard, real-time CEO metrics, cutting data warehouse costs) see how a data engineer thinks and works.
he data engineering skills and tools you need to know: SQL and Python, data modeling, ETL vs ELT, Git and CI/CD, and cloud platforms. Get a map of the modern data stack, including Snowflake, BigQuery, Redshift, S3, Kafka, Spark, Airflow, dbt and Power BI.
A quick recap of what data engineering is, the role of the data engineer and the key data engineering skills and tools, so you are ready to explore the data engineering ecosystem next.
Welcome to the data engineering ecosystem. This section covers the foundations every data engineer relies on: computer hardware, on-premises vs cloud computing, cloud services, data structures and formats, and OLTP vs OLAP systems.
The roadmap for this section: the basics of computers, on-prem vs the cloud, cloud services and providers, all things data (types, structures, formats, big data) and the operational vs analytical world.
Computer fundamentals for data engineers: CPU, RAM, storage (HDD and SSD), GPU and computer networks. Learn why separating storage, compute and memory matters, and how IP addresses, TCP/IP, HTTP and DNS move data between machines.
On-premises vs cloud computing: who owns, manages and maintains the infrastructure. Learn why companies are moving to the cloud for cost efficiency, scalability, accessibility and faster time to market.
The four core cloud services every data engineer uses: storage (data lakes such as AWS S3, Google Cloud Storage, Azure Data Lake Storage), compute (VMs, containers, serverless), managed databases and networking (CDN, load balancers, VPN).
What is data, and how does it turn into information and knowledge? Learn the definition of data as recorded events, activities and transactions, the raw material of every data pipeline.
Structured vs semi-structured vs unstructured data, with examples: database tables and CSV files, JSON, XML and logs, and text, images, audio and video. Learn which storage and use cases (BI, data interchange, AI/ML) fit each type.
The data types every data engineer works with: integer, float, character, string, boolean, date and timestamp, plus lists, dictionaries and tuples. Choosing the right data type prevents bugs and keeps pipelines efficient.
Data file formats explained: text-based CSV, TSV, JSON, XML and YAML vs binary formats Avro, Parquet, ORC and Protobuf. Learn when to use row-based vs columnar formats for streaming, storage and analytics.
The Vs of big data: volume, velocity and variety, plus variability, veracity, viscosity and value. Understand what makes data "big" and how it shapes data engineering decisions.
OLTP vs OLAP explained: operational databases (MySQL, PostgreSQL) vs analytical data warehouses (BigQuery, Redshift, Snowflake). Compare their purpose, query patterns, data freshness and ACID compliance (atomicity, consistency, isolation, durability).
Recap of the data engineering ecosystem: computer basics, cloud computing and services, data structures, types and formats, big data, and OLTP vs OLAP.
How does data travel from source systems to analytics? This section covers data sources, data repositories, ETL and ELT data pipelines, and batch vs real-time processing, with a hands-on API exercise in Python.
The roadmap for this section: data sources (databases, APIs, files, streaming), data repositories (data warehouse, data lake, lakehouse, data marts, NoSQL), data pipelines and batch vs streaming.
What are data sources in data engineering? Learn the main types you will extract data from: operational databases, REST APIs, files, streaming and IoT sensor data, and logs and monitoring systems.
Operational databases (OLTP) as a data source: how SQL and NoSQL transactional databases store day-to-day business data, why ACID compliance matters, and what to watch for when extracting data from them.
REST APIs for data engineers: how clients and servers communicate over HTTP, GET/POST/PUT/DELETE methods, endpoints, headers, query and path parameters, authentication (API key, bearer token, basic auth) and HTTP status codes like 200, 401, 404 and 500.
More data sources for your pipelines: flat files loaded from local disks, data lakes or SFTP servers; streaming, sensor and IoT data; and semi-structured application logs and monitoring data.
Step-by-step solution to the earthquake API assignment: set up a virtual environment, use Python requests to query the USGS API, flatten GeoJSON with pandas and save the results to CSV.
Where does data go? An introduction to data repositories in the analytical world: data warehouses, data lakes, data lakehouses, data marts, NoSQL databases and API or FTP destinations.
Data warehouse vs data lake vs data lakehouse explained: structured vs raw data, schema-on-write vs schema-on-read, and BI vs ML use cases. Examples include Snowflake, BigQuery, Redshift, Amazon S3, Databricks and Delta Lake.
What is a data mart? Learn how a data mart is a focused subset of a data warehouse, built for one department or business area such as sales, finance or marketing reporting.
NoSQL databases explained: document databases (MongoDB), key-value stores (Redis, DynamoDB), wide-column stores (Cassandra, HBase) and graph databases (Neo4j). Learn when each NoSQL type is the right choice.
APIs, FTP and files as data destinations: how data engineers deliver data to other systems and partners through APIs, file exports and SFTP transfers for reporting and integration.
What is a data pipeline? Learn the extract, transform, load steps, the difference between ETL and ELT, their pros and cons, and popular tools such as AWS Glue, Azure Data Factory, Fivetran, dbt and Snowflake.
Batch processing vs real-time streaming: latency, cost and complexity trade-offs. Learn about message brokers, pub/sub vs message queues, Kafka, Kinesis and RabbitMQ, and the Lambda architecture.
Recap of data sources, data repositories, ETL and ELT data pipelines, and batch vs real-time processing, before we move into data modeling.
Welcome to data modeling for data engineers. Learn how to design data models that make analytics fast and easy, from the star schema and snowflake schema to Data Vault, the medallion architecture and BI performance tuning.
The roadmap for this data modeling section: what data modeling is, star schema, snowflake schema, Data Vault, medallion model, building a data model with BEAM, and optimizing your BI solution.
What is data modeling? Learn how designing the structure of your data enables efficient analysis, reporting and decision-making, and why good dimensional modeling makes analytical queries faster and easier for analysts.
The star schema explained: the most widely used dimensional model, with one central fact table surrounded by dimension tables. Learn what grain (event granularity) means and why star schemas power most data warehouses and BI dashboards.
Fact tables in dimensional modeling: measures, foreign keys and grain. Learn the three types of fact tables (transactional, periodic snapshot and accumulating snapshot) with practical sales and order examples.
Dimension tables explained: descriptive attributes, natural keys vs surrogate keys, and slowly changing dimensions (SCD Type 0, Type 1, Type 2 and Type 3) with clear customer examples you can apply in any data warehouse.
Build your first star schema step by step: identify the business process, define the grain, design the fact table and its dimension tables, and connect them with surrogate keys.
Snowflake schema vs star schema: how normalizing dimension tables into sub-dimensions reduces redundancy, and the trade-offs in join cost, ETL complexity and query performance.
Turn a star schema into a snowflake schema: normalize a customer dimension into city and country tables and see how the extra joins change your SQL queries.
Data Vault modeling explained: hubs, links and satellites. Learn why Data Vault 2.0 offers scalability, flexibility and auditability for enterprise data warehouses with changing business rules.
The medallion architecture explained: bronze (raw), silver (cleaned and validated) and gold (business-ready) layers. Learn how this layered approach organizes data pipelines in a data lakehouse such as Databricks.
Build your first medallion architecture: land raw data in bronze, deduplicate and validate it in silver, and aggregate business metrics in gold, ready for BI tools.
How to build a data model with the BEAM methodology (Business Event Analysis and Modeling): identify business events, define their grain, answer the 7Ws (who, what, when, where, why, how, how many) and design fact and dimension tables.
Apply BEAM to a real business event, "customer buys a product": turn the 7Ws into a sales fact table and customer, product, store and date dimensions.
Data warehouse performance tuning for BI: indexes, table partitioning and partition pruning, materialized views and denormalized tables. Learn when each optimization helps and when it costs more than it saves.
Recap of data modeling: star and snowflake schemas, fact and dimension tables, slowly changing dimensions, Data Vault, the medallion model, BEAM and BI optimization. Practice with the airline flight star schema exercise.
Welcome to the SQL crash course for data engineers. You will write real SQL in Snowflake, from SELECT basics to joins, CTEs and window functions, and finish by building a sales data mart with an ELT pipeline.
The roadmap for this SQL section: SQL fundamentals, Snowflake setup, data retrieval, aggregations, joins, subqueries, CTEs, window functions, query execution order, data cleaning, transformations and building data marts.
What is SQL (Structured Query Language)? Learn why SQL is the standard for relational databases, the main SQL dialects (PostgreSQL, MySQL, Oracle, Snowflake SQL) and the four command groups: DDL, DML, DQL and DCL.
Core relational database concepts: primary keys, composite keys and foreign keys; one-to-one, one-to-many and many-to-many relationships; databases, schemas, tables and views (regular, materialized and secure); and common SQL data types.
What is Snowflake? An introduction to the cloud data platform: its three-layer architecture (cloud services, compute, storage), separation of storage and compute, virtual warehouses, stages, and roles and access control.
Create your free Snowflake trial account step by step: pick a cloud provider and region, choose an edition and log in to Snowsight, ready for the hands-on SQL exercises.
A guided tour of the Snowflake UI (Snowsight): worksheets, databases and schemas, virtual warehouses, query history and loading data, so you can find your way around Snowflake with confidence.
Set up Snowflake for the course: create a warehouse, database, schema and tables, and load the sample customers, products and orders data used in every SQL practical example.
SQL SELECT basics: choosing columns and aliases, filtering with WHERE, comparison and logical operators (AND, OR, NOT), IN, BETWEEN and LIKE, sorting with ORDER BY and limiting results with LIMIT and OFFSET.
Write your first SQL queries in Snowflake: select, filter, sort and limit customer and product data, and learn why SELECT * is fine for exploration but not for production pipelines.
our first SQL assignment explained: retrieve all orders above $200 from January 2024 and sort them in ascending order, using WHERE, date filters and ORDER BY.
Step-by-step solution to the basic data retrieval assignment, with tips on filtering dates correctly and writing clean, readable SQL.
SQL aggregate functions COUNT, SUM, AVG, MIN and MAX, grouping with GROUP BY, and filtering groups with HAVING. Learn how aggregations power KPIs, dashboards and analytics.
Hands-on GROUP BY examples in Snowflake: count orders by country, calculate average price by category and use HAVING to keep only the groups that matter.
he aggregation assignment explained: calculate total and average order amount per month using SQL aggregate functions, date functions and GROUP BY.
Step-by-step solution to the monthly aggregation assignment, using DATE_TRUNC, SUM, AVG and GROUP BY in Snowflake.
SQL joins explained with diagrams: INNER JOIN, LEFT JOIN, RIGHT JOIN, FULL OUTER JOIN and CROSS JOIN. Learn when to use each join for fact-to-dimension queries, data quality checks and data reconciliation.
Hands-on SQL join examples in Snowflake: combine orders and customers with INNER and LEFT JOIN, find customers without orders, and avoid the duplicate rows bad joins create.
The joins assignment explained: list every customer with their total number of orders and total sales, including customers who have never ordered.
Step-by-step solution to the joins assignment, using LEFT JOIN, GROUP BY and COALESCE to handle customers with no orders.
SQL subqueries explained: nested queries in the WHERE clause, scalar subqueries in SELECT, and derived tables in FROM. Learn what each type returns and when a subquery runs once vs once per row.
Hands-on SQL subquery examples in Snowflake: filter orders with IN, compare each order to the average with a scalar subquery, and aggregate inside a derived table.
Common Table Expressions (CTEs) with the WITH clause: how CTEs make SQL readable, modular and easy to debug, and why they encourage the step-by-step transformation mindset used in ELT pipelines and dbt.
SQL CTE The CTE assignment explained: find the top 3 customers by total purchase value using a WITH clause.
Step-by-step solution to the CTE assignment, building the query in readable steps with WITH, SUM, ORDER BY and LIMIT.
SQL window functions explained: the OVER clause, PARTITION BY and ORDER BY, and how window functions differ from GROUP BY by keeping row-level detail. Covers aggregate, ranking (ROW_NUMBER, RANK, DENSE_RANK, NTILE) and offset (LAG, FIRST_VALUE, LAST_VALUE) functions.
Common SQL window function patterns: running totals, full-partition totals, 7-row moving averages and forward-looking windows. Learn window frames (UNBOUNDED PRECEDING, CURRENT ROW, FOLLOWING) and ROWS vs RANGE.
Hands-on window functions in Snowflake: deduplicate with ROW_NUMBER, rank customers with RANK and DENSE_RANK, compare orders with LAG and calculate running totals.
The window functions assignment explained: calculate each customer's cumulative spending over time and rank all customers by total spending.
Step-by-step solution to the window functions assignment, combining cumulative SUM with PARTITION BY and ORDER BY and customer ranking.
SQL order of execution explained: FROM and JOIN, WHERE, GROUP BY, HAVING, SELECT, ORDER BY, then LIMIT. Understand why a column alias can't be used in WHERE and how execution order helps you debug queries.
Data cleaning in SQL: handle NULLs with COALESCE, NULLIF, IFNULL and CASE WHEN; clean strings with TRIM, UPPER, LOWER, INITCAP, REPLACE and regex; and fix types and dates with CAST, TRY_CAST and DATE_TRUNC.
Data transformation with SQL: calculated columns, ratios and flags, CASE WHEN bucketing, PIVOT, and combining aggregations with window functions to reshape raw data into analytics-ready tables.
Build an ELT pipeline and a sales data mart in Snowflake: load data with stages and COPY INTO, automate with Snowflake tasks and stored procedures, and model a star schema with fact_sales, dim_customer, dim_product and dim_date.
Recap of the SQL section: SQL and Snowflake fundamentals, data retrieval, aggregations, joins, subqueries, CTEs, window functions, data cleaning, transformations and ELT data marts.
Welcome to data wrangling: turning raw, messy data into clean, structured data you can trust. You will clean, transform and validate data with SQL and Python pandas through hands-on examples and assignments.
The roadmap for this data wrangling section: what data wrangling is, data cleaning (deduplication, missing values, standardization), data transformation (sorting, aggregation, filtering, window functions) and data validation.
What is data wrangling (data munging)? Learn how data engineers bring raw data into a structured, analysis-ready format by cleaning and transforming it, and the tools used to do it: SQL, pandas, Apache Spark, dbt, Great Expectations and Soda.
Why data cleaning matters: real-world data is rarely accurate, complete or consistent. Learn the goals of data cleaning and the most common data quality problems you will fix in every pipeline.
How to remove duplicate records with SQL and Python: SELECT DISTINCT, ROW_NUMBER() in a CTE to keep the latest record, and pandas drop_duplicates() with subset and keep options.
How to handle missing values (NULLs and NaNs): when to drop rows, and imputation methods such as constant placeholders, mean, median or mode, interpolation and machine learning. Includes SQL and pandas fillna() examples.
Data standardization techniques: fix inconsistent capitalization, date and timestamp formats and currencies using casting functions and regular expressions (regex) in SQL and Python.
A complete data cleaning example in SQL: remove duplicates, fix invalid ages, standardize emails, fill missing country values and replace NULL income with the average.
The Python data cleaning assignment explained: use pandas to deduplicate records, handle missing values and standardize formats in a messy dataset.
Step-by-step solution to the Python data cleaning assignment using pandas drop_duplicates, fillna, string methods and to_datetime.
What is data transformation? Learn why raw data is rarely in the right shape for analysis and how data engineers restructure it with sorting, aggregation, filtering and window functions.
Sorting data in SQL and Python: ORDER BY with ascending and descending order and multiple columns, and pandas sort_values() for ordering DataFrames.
Data aggregation in SQL and pandas: totals, averages, minimums and maximums with GROUP BY and pandas groupby().agg(), the building blocks of analytical reporting.
Filtering data with SQL WHERE and HAVING clauses and pandas boolean indexing and query(). Learn the difference between filtering rows before and after aggregation.
Window functions for data transformation in SQL and pandas: cumulative sums, rankings, LAG and LEAD, first and last values, using OVER/PARTITION BY and pandas groupby with cumsum, rank and shift.
The Python data transformation assignment explained: for completed orders only, find the average, minimum and maximum amount per category, sorted alphabetically.
Step-by-step solution to the data transformation assignment using pandas filtering, groupby, agg and sort_values.
Data validation for reliable pipelines: accuracy, consistency, completeness and integrity checks. Learn schema, domain, range, uniqueness and null checks, and tools like custom SQL tests, pandas, Great Expectations and Soda.
Recap of data wrangling: data cleaning, data transformation and data validation with SQL and Python pandas, before moving on to orchestrating data pipelines.
Welcome to data pipeline orchestration with Apache Airflow. Learn how data engineers schedule, automate and monitor pipelines, and build your first Airflow DAG with tasks, dependencies, operators and sensors.
The roadmap for this section: what orchestration is, an introduction to Apache Airflow, DAGs, tasks and dependencies, operators and sensors, and the Airflow UI.
What is data orchestration? Learn how orchestrators manage task dependencies, scheduling, retries and failures across systems, and compare Apache Airflow with Dagster, Prefect, Luigi, AWS Step Functions and Google Cloud Composer.
Apache Airflow explained: why it is the industry-standard orchestrator and how its architecture works: the metadata database, scheduler, executor, workers, web server and triggerer.
A quick hands-on tour of Apache Airflow: see a running Airflow environment, where DAG files live and how the scheduler picks them up and runs your tasks.
Airflow DAGs (Directed Acyclic Graphs) explained: how to define workflows in Python, schedule them with cron expressions (minute, hour, day of month, month, day of week) and trigger them manually or with event-based triggers, sensors and datasets.
Airflow tasks and dependencies: set sequential and parallel tasks with the >> and << operators, use depends_on_past and trigger rules (all_success, one_failed, none_skipped), and understand task states such as queued, running, up_for_retry and deferred.
Airflow operators and sensors explained: PythonOperator, SQL and cloud transfer operators such as GCSToBigQueryOperator, and FileSensor and ExternalTaskSensor. Learn poke_interval, timeout, and poke vs reschedule mode.
Build a complete Airflow DAG step by step: import libraries, set default arguments, define the DAG and its schedule, create tasks with operators and wire up task dependencies.
The Apache Airflow web UI explained: monitor DAG runs, pause and unpause DAGs, trigger runs manually, read task logs to debug failures and mark task states.
A hands-on tour of the Airflow UI: the grid and graph views, task logs, run history and the daily actions you will use to monitor and troubleshoot production data pipelines.
Learn data engineering from scratch and build the skills to design, build and orchestrate real data pipelines, with no prior data engineering experience required.
Data Engineering: From Zero to Hero is a hands-on course for anyone who wants to understand how data is collected, stored, modeled, transformed and delivered in modern companies. You will learn the concepts, then apply them with the tools used in real data teams: SQL, Snowflake, Python (pandas) and Apache Airflow.
Whether you are a data analyst moving into engineering, a software engineer curious about data, or a career changer starting from zero, this course gives you a clear, practical path into one of the most in-demand roles in tech.
What you will learn
Data engineering fundamentals: what a data engineer does, the core skills and the modern data stack
The data ecosystem: cloud computing (AWS, Azure, GCP), structured vs unstructured data, file formats like JSON, Parquet and Avro, and OLTP vs OLAP
Data sources and destinations: operational databases, REST APIs, data warehouses, data lakes, lakehouses, data marts and NoSQL
Data pipelines: ETL vs ELT, and batch vs real-time streaming with Kafka and pub/sub
Data modeling: star schema, snowflake schema, fact and dimension tables, slowly changing dimensions, Data Vault and the medallion architecture
SQL with Snowflake: from SELECT basics to joins, subqueries, CTEs and window functions, ending with a star schema sales data mart
Data wrangling with SQL and Python: deduplication, missing values, standardization, transformations and data validation
Pipeline orchestration with Apache Airflow: DAGs, scheduling, tasks and dependencies, operators, sensors and the Airflow UI
Learn by doing
Every major topic has a practical example, a hands-on assignment with a full video solution, and a quiz. You will pull earthquake data from a real API with Python, write SQL in your own free Snowflake account, clean and transform datasets with pandas, and build an Airflow DAG.
Taught by a practitioner
Your instructor has more than 10 years of experience working as a data analyst, data scientist and data engineer, and focuses on what you will actually use on the job.
By the end of this course you will understand how data flows through an organization, and you will be ready to build your own data pipelines or take the next step toward a data engineering role. Enroll now and start your data engineering journey.