Apache Iceberg: Multi-Engine Lakehouse, Spec to Production

The 2024–2026 inflection points reveal a rapid shift to a single interoperable iceberg ecosystem, with Polaris and Unity catalogs enabling cross-engine reads and one governance layer across three engines.

Apache governance with distributed committers and a public process, ensuring open collaboration, vendor independence, and robust architecture against dominant vendor strategy changes.

Bridge the first two modules by reframing the lakehouse and outlining five key insights. Compare Iceberg, Delta, and Hoodie, emphasize governance, and prepare for the spec internals exploration.

Understand how Iceberg turns raw parquet files into a queryable table via a five-layer stack: catalog, metadata.json, manifest list, manifests, and parquet data, enabling fast pruning and time travel.

Default to Parquet for new Iceberg tables, as it beats ORC and Avro for data files, delivering columnar reads and strong compression; target 128–512 MB files with JSTD by default.

Explore how Iceberg uses manifest files as a metadata index to prune data at query time, using Avro manifests with per-file and per-column statistics and status flags.

Understand how a manifest list ties each snapshot to its manifests, and how the snapshot tree with parent links enables reading audit log in Spark, expire snapshots to control storage.

Learn metadata-only schema evolution in iceberg: add, rename, and drop columns with stable IDs. Renames update metadata only; drops are permanent without time travel, and safe type changes require migration.

Discover how Iceberg replaces Hive partitioning with hidden, function-based transforms that prune by date and id, and how you can evolve partition strategies without rewriting data.

Sort order and partitioning work together to speed queries. They create tight min and max bounds for file pruning, with future writes sorted and existing data preserved.

Evolve an Iceberg table with schema changes, shift partitioning from months to days, and add a sort on order date and customer id, validating metadata.

Unify engines through a single REST catalogue driven by a versioned OpenAPI spec inside Apache Iceberg, enabling any engine to attach with unified auth and a shared protocol.

Master the three-layer rest catalog—namespaces, tables, commits—through get, post, and delete, with atomic commits. Use config, tokens, and metrics endpoints, and curl to validate catalogs.

Apply governance with a three-level model—identity, namespace, and table—by aligning grants, naming conventions, and authentication patterns to manage access and operations.

Spin up a local Iceberg rest catalog server, connect Spark to create a namespace and a table, and observe live api calls in logs.

Polaris unifies governance across Spark, Trino, and Snowflake with a central rest catalog and a shared grant system, enabling a single governance layer for multi-engine environments.

Explore Unity Catalog open source, a lineage-capturing data catalog for Delta and Iceberg, enabling multi-engine access via direct Iceberg REST or Uniform, ideal for Databricks-centric teams.

Nessie provides git-like catalog-level branching for Apache Iceberg, enabling safe multi-table testing and atomic merges across the catalog. Use cases include audit publish, experimentation, and disaster recovery.

Leverage the AWS Glue catalog as the IWS-native default catalog for Athena analytics, IAM-centered security, and S3 table integration, with practical setup tips.

Hive metastore remains the backbone of many data lakes, with Iceberg supporting HMS as a catalog; consider gradual migration to rest catalogs as needs grow across engines, regions, and governance.

Learn data frame writes with Apache Iceberg, including branch writes, audit and publish to main, and using snapshot properties for traceability and atomic commits.

Apply the write-audit-publish (WAP) pattern to keep production data trustworthy, using a WAP branch for staging, audit validation, and atomic publish to zero downtime for readers.

Set a target file size and distribution mode (hash or range) at table creation to prevent small files sprawl in iceberg writes. Use write-time and maintenance compaction to protect performance.

Build a real iceberg bronze layer by ingesting raw json shipment events into a day-partitioned, event-time sorted table with idempotent partition overwrites and pruning for fast queries.

Coordinate parallel Flink writers with a single coordinator to commit at each checkpoint, balancing freshness, file counts, and operational maintainability in Iceberg sinks.

Master exactly-once semantics in streaming pipelines using Iceberg aware sinks. Atomic checkpoint commits and idempotent replay ensure no duplicates or drops, even after failures.

Learn upsert patterns with merge in Apache Iceberg, using Bronze CDC and Silver Current to keep a single source of truth with incremental, idempotent merges and row_number for latest events.

Learn how to implement slowly changing dimension type 2 on Iceberg using a single merge to preserve history with valid from, valid to, and is current, while handling out-of-order events.

Learn how to handle out-of-order events in a CDC pipeline using watermarks, late-event strategies, and Iceberg features like atomic merge, time travel, and partition rewrites.