Apache Iceberg: The Beginner's Guide

Apache Iceberg's background and architecture spotlight a metadata layer between storage and compute engines, enabling time travel and atomic snapshots that provide ACID transactions.

Create an Apache Iceberg table with PyIceberg in Python, define a schema and namespace, insert data transactionally, then select records and explore partitioning, schema evolution, and time travel.

Learn to query Apache Iceberg tables with PyIceberg by setting up a local catalog, loading tables, and selecting only required columns in a Jupyter notebook for efficient data access.

Filter data in Iceberg tables using storage-layer predicates to skip non-matching files, leveraging a local catalog and product table with row filters like equal, greater than, and in.

Compare pandas and polars to process billions of rows on commodity hardware, revealing polars' faster columnar processing and easier API for data manipulation and filtering.

Explore using Apache Iceberg with Polars in Python to load a local Iceberg catalog, scan a table, and execute lazy queries that read only required columns for efficient data analysis.

Connect to a local Iceberg catalog, load a table, convert it to a Polars lazy data frame, and filter by id, name, or multiple ids.

Explore how Apache Iceberg integrates with Spark across local catalogs and AWS S3 storage, coordinating metadata and data layers for production environments and petabyte-scale parquet datasets.

Install PySpark in a Jupyter Notebook to enable reading, writing, and managing Iceberg tables with the Python API.

Initialize a spark session with iceberg support, create a local catalog product table with six columns, insert 25 rows, and demonstrate iceberg features like schema evolution and time travel.

Query iceberg data with PySpark by reading the local catalog’s product table into a Spark DataFrame and selecting the name, then product ID, name, and stock.

Explore PySpark aggregate functions on Iceberg tables, set up a Spark session with Iceberg integration, and compute total, max, min prices, counts, and distinct categories.

Explore duckdb, an open source in-memory analytical database designed for workloads, embeddable with zero configuration, SQL support, columnar storage, and vectorized execution.

Explore DuckDB's in-memory execution and columnar storage for ultra-fast analytical queries. Learn its serverless, embedded design, multi-language support, native file access, and cross-platform open-source nature for streamlined data workflows.

Implement a production-ready iceberg workflow using AWS Glue as the catalog, S3 storage, and Spark processing. Create a Glue database, define a product schema, and build an iceberg table.

Learn to read iceberg tables in S3 with DuckDB, using the iceberg and httpfs extensions and the iceberg_scan function for an S3 catalog.

Learn how to insert data into iceberg tables with spark using dataframe write, append, and coalesce, while examining metadata, snapshots, and copy-on-write behavior for time travel.

Explore time travel with Apache Iceberg by querying historical snapshots via version as of, using snapshot metadata for auditing, debugging, and data analysis in spark read-only queries.

Configure and verify Apache Iceberg table properties to optimize storage and query performance, including parquet storage, snappy compression, and aligning session and table level settings.

Iceberg enables hidden partitioning that decouples storage from queries, letting users filter with natural expressions while engineers manage partition transforms—time based like year, month, day—and partition evolution without rewriting data.

Learn how hidden partitioning in Apache Iceberg boosts performance without changing queries. See Spark integration, automatic partition pruning by day, and transforms like day, month, and hour.

Create databases and tables with Nessie catalog, insert data, and track operations with commits in Nessie UI and AWS S3.

Merge dev changes into main with Nessie, verify the merge, and clean up by deleting unused branches, mastering the full data lake development lifecycle.