Intro to Data Engineering

Data engineers develop and maintain systems that transform raw data into high quality information for analysis and machine learning. They unify disparate sources, enforce security, and orchestrate data pipelines.

Explore the data engineering lifecycle from data generation and ingestion to transformation, storage, serving, and reverse etl, with emphasis on security, access controls, monitoring, and orchestration.

Explore how data engineering relates to data analyst, data scientist, and software engineer roles, highlighting how pipelines, optimization, and collaboration shape each career path and the potential to switch fields.

Explore the three data engineering service models—service, startup, and product—and see how they shape team organization, workflows, and analytics delivery.

Master core data engineering skills including expert SQL and Python, Bash shell scripting, version control with Git, data modeling, infrastructure as code with Terraform and CDK, and data orchestration.

Explore data architecture as the blueprint for collecting, storing, securing, and using data, covering data integration, storage strategies, rbac, compliance, pipelines, data quality, and trade-offs between cost, speed, and security.

Explore a concrete data architecture, from external data sources and an S3 bucket to a data warehouse, raw tables, and a data mart, orchestrated by Apache Airflow for secure analytics.

Learn how data lakes store raw data at scale for analytics and machine learning, while data warehouses offer structured, fast queries for business intelligence, and data marts serve department-specific needs.

Explore the modern data stack as a cloud-based, consumption-based philosophy guiding tool choices for data pipelines, highlighting ELT over ETL, open-source bias, and component-based architectures versus turnkey solutions.

Connect data from emails, databases, and external services into a data warehouse. Enforce standardized, consistent data flow that fails loudly and stops dependent processes when issues arise.

Learn principles of data architecture: select common components wisely, plan for failure, design for scalability and leadership, keep systems loosely coupled, enable reversible decisions, prioritize security, and embrace FinOps.

Learn how data orchestration coordinates, schedules, and runs data workflows using dependency management, automation, and security to improve data quality and compliance.

Explore Apache Airflow, an open-source data orchestration tool from the Apache Software Foundation, using DAGs, tasks, operators, and a scheduler and executor to manage data pipelines.

Explore relational databases, where data rests in tables of rows and columns; learn primary keys and foreign keys, and how SQL enables analysis.

Explore how relational databases organize data using servers, databases, schemas, tables, and views. Understand how public schemas group objects and how views support reusable query results.

Compare OLTP and OLAP databases: OLTP handles fast inserts, updates, and deletes for transaction oriented apps with high concurrency, while OLAP supports large, complex analytics queries, often with data lag.

Learn how SQL, the primary method for interacting with relational databases, creates, manipulates, and queries data using a declarative language. See a practical example with select, where, and order by.

Explore the acid properties—atomicity, isolation, consistency, and durability—and how they guarantee transactional integrity in relational databases, with comparisons to NoSQL and SQL choices.

Explore document databases, a flexible NoSQL style that stores JSON-like documents with schemaless structure, enabling fast development and horizontal scaling—yet limiting complex analytics without a relational layer.

Explore key-value databases as generalized document stores, where a key maps to any value, with DynamoDB as an example, and see how data moves into relational databases for SQL analysis.

Explore object storage, a flat hierarchy that stores files like CSV or text in a bucket, enabling fast retrieval and simple staging between systems, like AWS S3.

Explore columnar databases and their analytics optimization over row-based relational storage, showing how accessing a subset of columns reduces IO and speeds queries, with Cassandra DB as an example.

Discover graph databases that store data as nodes, edges, and properties to model relationships like social networks and fraud detection. Examples include AWS Neptune, Neo4j, and Azure Cosmos DB.

Explore NoSQL concepts with practical questions on ACID durability, graph database edges, JSON document storage, and why columnar databases excel at analytical queries and data warehousing.

Explore horizontal vs vertical scaling in data engineering: vertical scaling is simple and cost-effective at small scale, while horizontal scaling scales across many servers but requires re-architecting and coordination.

Learn Python as a core data engineering tool, covering basics from variables to loops and pandas, and how Python drives workflows with Apache Airflow and Spark.

Rest APIs act as standard interfaces between software, enabling data engineers to use get and post calls to fetch data, store JSON in S3, and load into warehouses.

Master shell scripting basics for data engineers to interact with the terminal, navigate remote systems with pwd, ls, cd, mkdir, and use ssh, scp, and ssh keys for AWS access.

Explore cron, the time-based job scheduler essential in data engineering, and learn to define schedules with five fields, where star means all and star slash 15 specifies intervals.

Learn how version control tools like Git and Mercurial enable collaborative coding, branching, pull requests and merge requests, and an auditable history to track changes and revert when needed.

Reduce the risk of shipping bad or non-functional code by applying unit testing and integration testing, and learn how these guardrails boost code quality and safe refactoring within the SDLC.

Explore containerization with Docker to encapsulate applications and dependencies into portable containers that run identically across environments, enabling isolation and simplified deployment in data engineering and ci cd workflows.

Learn how infrastructure management at scale avoids costly mistakes from GUI-only workflows, by using infrastructure as code with Terraform, Ansible, and CDK for versioned, auditable control.

Understand what big data is, characterized by volume, velocity, and variety, and the data engineer's role in building large-scale processing architectures using Hadoop, Spark, and Kafka.

Explore Hadoop as an evolution of MapReduce for distributing data storage and batch processing with HDFS across machines, enabling scalable analytics on logs and social media data for insights.

Explore Apache Spark as an in-memory big data processing framework enabling stream processing, real-time analytics, data exploration, MLlib for scalable machine learning, and graph processing with PySpark and Spark SQL.

Learn Kafka as a stream processing tool for event-driven architectures, decoupling data ingestion from consumption and enabling real-time data pipelines, log aggregation, and microservices communication.

Explore conceptual, logical, and physical data models and learn how data engineers translate business processes into scalable database designs, including entities, keys, and ERDs.

Explore entity relationship diagrams as the universal language of database design, showing tables with primary and foreign keys, and one-to-many, one-to-one, and many-to-many connections.

Explore data normalization, reducing redundancy by separating related data into tables and joining them as needed. Learn 1NF, 2NF, and 3NF and their impact on queries.

Explore the Inman top-down data warehouse with normalized data marts, and the Kimball bottom-up approach using denormalized fact and dimension tables to handle quantitative data.

This lecture stresses the non optionality of security and privacy in data engineering, covering preventable breaches, PII, the right to be forgotten, and regulations like CCPA and GDPR.

Identify direct and indirect PII and apply data minimization, encryption at rest and in transit, and strict access controls. Emphasize transparency and training to prevent PII leakage with real-world examples.

Apply the principle of least privilege by granting minimum access, using role-based access control, just-in-time rights, and privileged access management to reduce attack surface and phishing risk.