Data Engineering 101: The Beginner's Guide

Explore how source systems generate data and distinguish structured tabular data from unstructured files, guiding storage, search, and processing choices for traditional analytics and deep learning.

Explore the main database types: relational databases, NoSQL, and key-value stores, and learn how choosing the right system affects performance, scalability, and data structure for transactional data and flexible schemas.

Learn how third-party tools generate data via rest, GraphQL, and gRPC APIs, how event streams move data from producers through brokers to consumers, including idempotency and at least once delivery.

Explore storage options in data pipelines, from memory and ssd to hdd, and understand latency, persistence, durability, replication, and cloud networking including egress charges.

Learn serialization to turn data into byte streams for efficient storage and transport, comparing JSON and CSV with Parquet and Arrow, and using compression and caching.

Explore distributed storage and compute across multiple machines, enabling parallel processing with Spark, Trino, and object storage like S3, and contrast strong versus eventual consistency with acid versus base models.

Compare five storage systems—file, block, object, cache, and streaming storage—and understand their use cases. See how object storage like S3 uses a flat structure and immutability for scalable, analytics-friendly workloads.

Compare row-based and columnar storage to reveal OLTP versus OLAP use cases, showing how transactions and analytics drive performance, with HDDs and SSDs shaping trade-offs.

Compare data warehouse, data lake, and data lakehouse to see Apache Iceberg, Apache Hoodie, and Delta Lake enable analytics on structured, semi-structured, and unstructured data with storage and compute separation.

Ingestion moves data from source systems to storage using batch, microbatches, or streaming to support latency needs. ETL vs ELT contrasts loading and transforming data for warehouses and data lakes.

Explore streaming ingestion and its latency, ordering, and duplication challenges, including late data, at-least-once vs exactly-once guarantees, and idempotency. Learn how buffering, retention, and dead letter queue protect the pipeline.

Learn to build scalable ingestion pipelines with auto scaling and buffers, handle outages and data spikes, and manage schema changes through validation and proactive communication with upstream producers.

Explore diverse data ingestion approaches, from direct database connections via JDBC and change data capture using database logs, to APIs with managed connectors and real-time streams with Kafka or Kinesis.

Learn how to shape raw data into useful business data by executing and optimizing SQL queries, exploring DDL, DML, and DCL, and planning, explain plans, and streaming window techniques.

Master data modeling as conceptual, logical, and physical stages to map orders, customers, and products; compare normalization with denormalization and explore keys, link tables, and y denormalized table patterns.

Learn how to compose and orchestrate multiple SQL transformations to convert raw data into business conform data, using dbt, views, and materialization concepts for analytics.

Explain the events versus states distinction, where events are immutable and states are derived from them. Compare streaming and batch data, their processing differences, and when to apply each approach.

Serve data to downstream consumers through analytics, ML and AI, and reverse ETL, by providing dashboards, notebooks, and feature stores to support useful metrics and catch errors through monitoring.

Explore data ops as the data counterpart to dev ops, focusing on robust data pipelines, observability, automated CI/CD, and orchestrators to replace fragile cronjobs and prevent silent data failures.

Orchestration coordinates multiple data engineering jobs to run efficiently and reliably, enabling end-to-end visibility and data lineage while scheduling cadence, retries, and timeouts for complex pipelines.

Explore security and privacy considerations for data engineers, including authentication, authorization, least privilege, and layered defenses, plus data privacy practices and robust data quality through validation and observability.

Define a development and production workflow with an isolated local environment. Prioritize rapid iteration, production parity, and local data samples to boost productivity.

Assess the key qualities of good data architecture—performance, scalability, reliability, security, and modularity—while comparing Lambda, Kappa, and the modern data stack for end-to-end pipelines.

Design your data architecture for speed with modular, open source and cloud tools, starting simple to enable rapid value while ensuring scalable integration and avoiding vendor lock-in.

Compare data warehouse and data lake house architectures for end-to-end data pipelines, focusing on ingestion, storage, and orchestration, with S3 as a landing zone and Snowflake, Redshift, or BigQuery choices.

Explore Doc DB as an in-memory OLAP database with analytical columnar storage, enabling performant SQL queries as a local replacement for Snowflake, while noting its lack of distributed scalability.

Explore orchestrators for the BI stack, comparing airflow with dexter, and contrast declarative versus imperative approaches, highlighting local development, cloud native deployment, and CI/CD benefits.

Explore ingestion options in a data warehouse, comparing custom Python connectors with managed tools like Fivetran, Airbyte, and DLT, including pre-built connectors and CSVs in S3.

Explore the BI stack in a data warehouse: transform ingested data with SQL in Snowflake, optionally using dbt, then visualize with BI tools like Tableau or Power BI.

Examine a data lakehouse BI stack with Apache Iceberg storage and Trino compute, illustrating storage‑compute separation, SQL on object storage, and the open source advantage over Snowflake.

Explore the compute engine in a data lakehouse with Trino, a federated, pure compute engine using connectors to BigQuery, Snowflake, Postgres, and more; compare with Spark.

Design a streaming stack for unbounded events driven by latency needs, from real-time analytics to same day shipping, and understand Kafka as event log and buffer, with Pulsar as alternative.

Ingest data with Kafka and process it in real time using Flink. Enrich streams with Postgres, store in Iceberg, and query with Trino, exploring latency versus completeness.

Clarify the differences between AI, ML, and deep learning, and contrast structured data with unstructured data. Explain training vs inference, labels, and the data and compute needs in data engineering.

Explore how the ML pipeline extends the data engineering workflow through a feature store that centralizes features, enabling models to predict housing prices from columns like rooms and location score.

Train machine learning models by exploring data in the lakehouse and feature store, using notebooks like Jupyter and Ray for distributed training, while tracking experiments, hyperparameters, and data lineage.

Explore model serving and inference methods, including online and batch inference, with Bento ML; compare resource use, monitoring, and drift to keep ML models current.

Compare the deep learning pipeline with traditional ML, highlighting unstructured data and millions of features that demand more storage and compute, illustrated by a facial recognition scenario.

Explore the deep learning pipeline from labeling and data preparation to training and inference, including the iceberg table, GPU-powered distributed training, and model serving with BentoML and edge deployment considerations.

Build robust data pipelines with type, constraint, and code validations, plus unit and statistical testing. Enhance observability and deploy with CI/CD using Terraform and Plumie, as well as GitHub Actions.

Data engineering adopts software engineering practices and open source integration layers for faster, local, code-first pipelines. The lecture shares opinions on AI’s future and learning paths.

Plan your data engineering journey forward with a book recommendation and ML resources, then build end-to-end data pipelines using Docker and the open modern data stack.