Apache Iceberg: Complete Hands-On Masterclass

Name: Apache Iceberg: Complete Hands-On Masterclass
Rating: 5.0 (6 reviews)

Master Apache Iceberg with hands-on labs using PySpark, Databricks, and Google Colab—no setup or data needed

Role Play

Created byDE & AI Engineering Simplified

Last updated 7/2025

English

English [Auto],

What you'll learn

Learn why Apache Iceberg is redefining modern data lakes and how it overcomes limitations of Hive, Delta Lake, and Hudi.
Set up Iceberg in Databricks & Google Colab—no local install or cloud budget needed—and start building hands-on from your browser.
Perform DDL and DML operations (insert, update, delete) on Iceberg tables and explore internal metadata structures such as snapshots, manifests, and partitions.
Master Iceberg’s time travel and metadata tables to build scalable, version-controlled, cost-efficient data pipelines.
Understand Iceberg’s architecture under the hood—how it handles schema, partition evolution, and decouples compute from storage.
Explore real-world debugging, rollback, and auditing use cases using Iceberg’s powerful snapshot and metadata tracking system.

Course content

8 sections • 30 lectures • 4h 14m total length

Why To Learn Apache Iceberg6:07
Explore how Apache Iceberg delivers true acid compliance, scalable metadata management, and schema evolution, partition evolution, and time travel for cloud-scale, multi-engine data lakes or lakehouses.
Course Roadmap: What You’ll Learn, Step by Step3:10
Explore Apache Iceberg from setup to time travel through practical Databricks and Colab hands-on demos, covering Hive tables versus Iceberg tables, DDL and DML, metadata tables, and architecture.
Course Technical Pre-requisite2:01
Engage in an interactive, hands-on exploration of Apache Iceberg using PySpark, Databricks Community Edition, and Google Colab, with zero infrastructure setup and self-contained data generation.
Downloadable Resources: Diagrams, Code & Data Files2:33
Download the complete course content zip to access python notebooks, architectural diagrams in pdf, lessons learned in png, and chapter resources; unzip, validate, and explore five sections in VS Code.

Introduction - File Format & Table Format With Apache Spark2:17
Explore how Spark writes Parquet and CSV, compare with Apache Iceberg table format, and perform DDL and DML operations in a hands-on PySpark Databricks setup.
Databricks Community Edition - Self Registration1:17
Register for Databricks community edition with a personal email via web, verify with a six-character code, log in to Databricks notebook, and upgrade to pro for unlimited clusters and collaboration.
Databricks Web Interface Walkthrough3:50
Take an interactive tour of the Databricks web interface, covering the side panel, user profile, security, compute, catalogs, and notebooks, with hands-on setup and future compute attachment.
Databricks Freemium Edition - Support For Iceberg Table & Limitations3:48
Explore Databricks freemium edition and Iceberg limitations. The video explains why Iceberg work uses the community edition due to Unity Catalog requirements and support gaps.
Create Compute & Notebook in Databricks10:15
Create a Databricks compute cluster and notebook to explore memory, storage, dbfs, and run Spark with Iceberg runtime on community edition.
Apache Spark With Parquet File Format11:14
This chapter demonstrates a PySpark program that creates dummy inventory data, writes it to parquet and CSV, partitions by category, and contrasts file format with the Iceberg table approach.
Quiz - PySpark Data Handling Fundamentals
PySpark With Iceberg File Format26:30
Create an Apache Iceberg table with PySpark and configure the Spark environment and Iceberg runtime. Compare Iceberg file format with table format and explore versioned metadata and time travel.
Quiz - PySpark + Apache Iceberg: Fundamentals

PySpark On Google Colab Notebook1:41
Learn to run PySpark with Iceberg on Google Colab by setting up Spark and Hadoop, checking PySpark config, and creating and validating the inventory table and metadata files.
Free Google Colabe Notebook Setup To Run PySpark6:31
Set up a Google Colab notebook for PySpark with Apache Iceberg by installing Java, Hadoop, and Spark 3.5.1, configuring the Iceberg Spark runtime, and verifying memory, disk, and catalog setup.
Apache Iceberg With PySpark On Colabe Notebook14:57
Learn how to use Apache Iceberg with PySpark on Colab notebook, enabling the iceberg spark extension, creating inventory tables with metadata, data folders, and time-travel queries.
Choosing the Right Platform: Databricks CE, Colab, or Databricks Freemium for Apache Iceberg PoC

OLAP & Role Of File & Table Formats7:07
Explore the Apache Iceberg architecture and how table formats, storage engines, catalogs, and compute engines optimize OLAP workloads across CSV and Parquet data stored in cloud or distributed file systems.
Data Lake (To Be modified)2:09
Explore data lake architectures that store all data types and formats, from CSV and Parquet to images, while examining Hadoop, HDFS, MapReduce, and Hive external tables for analytics.
Hive Table Format7:52
Create two external tables over a csv file and a parquet file in Databricks using a Hive table format with header and inferred schema, enabling seamless queries via the catalog.
Data Lake2:49
Explore how the data lake platform provides visibility into storage and hive table format, supports multiple file formats, and flexible storage locations, while weighing pros and cons against data warehouses.
Why Iceberg Table Format3:38
Discover how lakehouse combines storage and compute, and how the iceberg table format enables ac transactions, time travel, and partition evolution through explicit configuration.
Convince Me: Why File Format & Table Format Matter in Spark-Based Datalake?

Create & Insert Operation With Apache Iceberg Table16:32
Configure a Databricks cluster with the Apache Iceberg Spark runtime, generate 100k inventory records, and create an Iceberg table partitioned by category; insert data and inspect metadata and manifest files.
Insert Operation (DML) With Apache Icerberg10:24
Perform an insert on an existing Apache Iceberg table, adding 300k records, and review how data files, metadata, and snapshot history reflect the update.
Delete Data From Iceberg Table With Spark 3.222:39
Explore delete operations on Iceberg tables with Spark 3.2. Learn how metadata governs logical deletes, time travel, and snapshot files.
Delete Data From Iceberg Table With Spark 3.5 (Non Partitioned Data)17:11
Demonstrates deleting records from an Apache Iceberg table without partitioning using Spark 3.5 in a Google Colab notebook, and analyzes resulting metadata and data file changes.
Iceberg Built-in Metadata Tables14:21
Explore iceberg built-in metadata tables—history, metadata log entries, snapshots, and manifests—and see how inserts, deletes, and updates reveal data evolution, enabling time travel and strong observability.
Time-Travel Feature With Iceberg4:20
Master the time travel feature of Apache Iceberg tables by using version as of or timestamp as of to view and roll back data via snapshot IDs and metadata table.

Apache Hive Partition Evolution Limitations14:27
Understand Apache Hive partition limitations, including static and dynamic partitions, and how partition pruning improves query performance and cost, setting the stage for Iceberg's partition lineage.
Hidden Partition & Partition Evolution in Apache Iceberg19:55
Explore Apache Iceberg's hidden partition and partition evolution, compare with Hive, Delta Lake, and Hoodie, and practice in a Colab notebook using Spark.

Iceberg System Procedure - Intro1:54
Explore Apache Iceberg's built-in SQL procedures with Apache Spark to rollback to snapshots or timestamps and cherry-pick changes, all via hands-on Google Colab.
Iceberg Procedure - Rollback Feature (To a snapshot & a timestamp)11:50
Explore iceberg rollback features with snapshots and timestamps, using stored procedures to revert data by snapshot or timestamp, and understand metadata and version history in a spark 3.5.1 environment.

Requirements

Basic understanding of Python programming is helpful.
Some familiarity with PySpark and big data file formats like Parquet is recommended.
Familiarity with file formats like Parquet or ORC is useful but not mandatory.
No need for cloud subscriptions or local installations — Databricks Community Edition and Google Colab are used (both are free).

Description

This course offers a practical, hands-on introduction to Apache Iceberg, the modern open table format designed for today’s large-scale data lakes and lakehouses. Whether you’re a data engineer, developer, or architect, this course will help you understand and apply Iceberg concepts through real-world exercises—without the need for any infrastructure setup.

You’ll learn to create, query, and manage Iceberg tables using PySpark in both Databricks Community Edition and Google Colab—two free platforms accessible from your browser. We cover everything from understanding table formats, DDL and DML operations, partition evolution, schema evolution, metadata tables, and Iceberg’s powerful time travel capability.

All code and sample data are provided chapter by chapter. You’ll generate data on the fly, inspect table structures, and compare metadata files using VS Code and online JSON viewers. No local installation, no external datasets—just clear, interactive learning.

What You’ll Learn

Key differences between file formats and table formats in big data
How to create and manage Apache Iceberg tables using PySpark
Comparing Hive tables and Iceberg with practical demos
Running Iceberg on Databricks and Google Colab (setup included)
Performing DDL and DML operations (insert, update, delete)
Using Iceberg’s built-in metadata tables to inspect file-level and snapshot info
Implementing time travel to query historical data versions
Understanding how Iceberg handles schema evolution and partition changes
Comparing Iceberg with Delta Lake and Hudi in practical scenarios

By the end of the course, you’ll have a strong working knowledge of Apache Iceberg and be ready to use it in real-world data projects with confidence.

Who this course is for:

This course is designed for data engineers, data architects, and developers who work with large-scale data pipelines and are looking to modernize their data lake or lakehouse architecture. If you’re currently using Hive, Delta Lake, or Hudi and want to explore a more flexible, scalable, and engine-agnostic table format, this course is for you.
It’s also ideal for those who want to gain hands-on experience with Apache Iceberg using free tools like Databricks Community Edition and Google Colab — without needing complex infrastructure or cloud setup.
Whether you’re building batch or streaming pipelines, working on schema evolution, or just want to understand Iceberg’s time travel and metadata capabilities, this course will help you build practical skills with real-world applications.
No prior experience with Iceberg is required — just a willingness to learn and some basic familiarity with Python or PySpark.

Apache Iceberg: Complete Hands-On Masterclass

What you'll learn

Explore related topics

Course content

Introduction To Apache Iceberg4 lectures • 14min

How PySpark Works With File Format & Table Format - Using Databricks Notebook7 lectures • 59min

Apache Iceberg Hands On Using Google Colab4 lectures • 23min

Apache Iceberg Architecture - Deep Dive6 lectures • 24min

Apache Iceberg Hands on Lab & Deep Dive6 lectures • 1hr 25min

Thank You1 lecture • 1min

Bonus Section - Advanced Topic2 lectures • 34min

Bonus Section - Iceberg System Procedures2 lectures • 14min

Requirements

Description

Who this course is for: