
Explore the Apache Iceberg architecture and hands-on setup, including docker-based local deployment, Minio object storage, time travel with snapshots and partitioning, and insights into schema evolution and Nessie.
Explore Apache Iceberg, an open table format for big data that adds structure on top of raw files and enables acid transactions, time travel, and scalable schema changes.
? Ubuntu users:
Please make sure to manually create the ./warehouse and ./notebooks directories before starting the setup. Then, visit http://localhost:9001 to access MinIO and kindly create a bucket named warehouse.
Explore Apache Iceberg’s catalog, metadata layer, and storage layer, with metadata pointers tracking the latest version. Learn how metadata files, manifest lists, and data files enable fast, file-level scans.
Partition nested json fields directly with Apache Iceberg, avoiding json flattening and simplifying data writing and reading compared to Delta Lakes.
Explore how Apache Iceberg achieves file pruning beyond partitioning with a rich metadata layer, manifest files, and min and max stats encoded in base64 to scan only relevant parquet files.
Explore Iceberg schema evolution, showing metadata-only changes that leave data files untouched while adding, dropping, or renaming columns, and using immutable field IDs to preserve mapping.
Clone the repo, configure yaml and Dockerfile to load Nessie integrations jars for Spark 3.5, and start the dockerized spark-iceberg-minio-nessie stack to enable Nessie as a version catalog.
Create and switch Nessie branches and namespaces, using Pinus to base a feature branch on the main hash, while noting spark caching may require restarting the catalog to switch branches.
Explore PyNessie issues with switching between feature and main branches, and apply workarounds using separate catalogs for main and feature branches or a spark session to isolate namespaces and tables.
Learn to merge a feature branch into main and delete the feature branch in Nessie using Pinus. Create and delete tags, understand tag immutability, and manage main and feature namespaces.
USE CODE - ICEBERG3
Welcome to "Apache Iceberg: Your End-to-End Practical Course" — the most comprehensive, hands-on course designed to help you master Apache Iceberg and build modern data lakes efficiently and effectively!
Apache Iceberg is a revolutionary open table format that is transforming the way organizations manage massive datasets. It is designed to solve the challenges of big data, including schema evolution, hidden partitioning, and time travel, all without the limitations and complexities of traditional systems like Hive and even Delta Lake. Iceberg empowers you to manage petabyte-scale data in a simplified and performant way — making it a critical skill for anyone working in modern data engineering and analytics.
In this course, you will start from scratch and learn everything you need to become proficient in Apache Iceberg, whether you are a beginner or an experienced data engineer looking to upskill. Through step-by-step practical demonstrations, you'll learn how to create Iceberg tables, perform schema evolution without rewriting data, leverage hidden partitioning for faster queries, use time travel to access historical data snapshots, and manage data efficiently with features like compaction and snapshot expiration.
We focus on real-world use cases, showing you how Iceberg integrates with Spark, SQL, and cloud storage systems like S3 or MinIO. By the end of this course, you will be confident in using Apache Iceberg to build robust, scalable, and future-proof data lakes.
If you’re ready to master one of the most powerful data lake technologies available — this course is for you!