Become a Data Architect: Design, Decide & Defend Platforms

Explore how data mesh enables domain ownership with autonomous data products and a self-serve platform, while data fabric weaves metadata across systems for integration, automation, and governance via contracts.

Explore how a governed warehouse and a data mesh fit workload, budget, and skills. Use a platform selection scorecard to choose the architecture that best matches your business.

Discover the six data platform layers: source, ingest, store, transform, serve, and consume, and learn to defend their clear boundaries to build scalable, tool-agnostic architectures.

Master recall under pressure by using the six box diagram and five-step draw order, with the metadata bar and batch vs stream paths for design reviews, interviews, and data mesh.

Redraw the six-layer diagram per domain to implement a data mesh through replication. Add contracts on seams and convert the serve layer into a versioned data product with an SLA.

Visualize the course as one diagram, a six-box subway map where each box holds a module and its decisions, mapping ownership from ingest to capstone.

Explore lambda versus kappa architectures for batch and stream data, comparing two-path versus single-path designs, trade-offs, and the role of a durable, replayable log as the source of truth.

Understand how columnar formats Parquet and ORC reduce data reads by selecting columns, versus row formats like Agro for streaming, then use Avro for writes and Parquet for analytics.

Analyze how compression and layout trade-offs affect read cost, cpu, and storage. Use splittable codecs like snappy and lz4, and leverage row groups, sorting, and footer stats to skip data.

Delta, Iceberg, and Hoodie differ in governance and portability, not surface SQL; evaluate lock-in risk, portability, engine ecosystem, and community gravity.

Learn how the open catalogue enables true multi-engine read interop on an open lakehouse, while guarding against new catalogue lock-in by validating single writer plus multi-reader writes.

Assess a real decision to standardize a one-table format across Spark, Databricks, and a cloud warehouse to avoid lock-in, and recommend Iceberg plus an open catalog for cross-engine portability.

Declare one row equals one order line to define grain, then attach facts and dimensions to that grain. Avoid undeclared grain to prevent double counting and flawed schemas.

Explore star versus snowflake schemas, balancing query speed and storage in columnar warehouses. Normalize only when a dimension is large enough; star wins for most queries, while compute drives costs.

Design data warehouses by selecting SCD types per attribute to preserve or overwrite history, using Type 2 versioning, effective dates, end dates, and is current flags to support accurate reports.

Explore how data vault's hub, satellite, and link model delivers auditability and parallel loading, explaining why the 400-table explosion occurs only with many sources and regulatory mandates.

Understand how wide tables drive denormalization as a cost optimization, balancing storage, build compute, and read time with disciplined incremental rebuilds and dropping unused columns.

Choose ingestion patterns intentionally using a four-dial framework—latency, volume, mutability, and cost—guided by a decision tree that maps sources to full reload, incremental, change data capture, or streaming.

Name the log, processor, and sink as the core streaming triad; the log provides durable, ordered replayable events, the processor applies a stateful transform, and the sink stores results.

Delivery semantics - exactly-once and at-least-once - are a trade-off, not a correctness dial. Use at least once with an idempotent sink as default, reserving exactly once only for exceptions.

Assess streaming only when real-time freshness justifies the tax of cost, ops, and cognitive load. Apply the earn the stream test with conditions A, B, and C before provisioning brokers.

Compare Lambda and Kappa architectures for unifying batch and stream processing, weighing batch layer accuracy against speed layer freshness, and decide based on logic stability and replay cost.

Illustrate the medallion pattern with bronze, silver, and gold layers, each enforcing guarantees—from immutable raw to cleaned joins to business-ready aggregates—driving trusted analytics via automated boundaries and dbt.

Learn how the dbt paradigm makes transformation as code, with modularity, tests, docs, and lineage, enabling automatic boundary checks and governance for scalable data platforms.

Make transforms idempotent and safe to rerun by using merge on a business key, partition overwrite, or deterministic full rebuild; consider incremental versus full refresh for trade-offs.