Data Engineering on AWS - The complete training

Meet Ashish, the trainer behind data engineering on AWS. He delivers classroom-style instruction with simple analogies, why-focused explanations, and whiteboard demos that connect concepts to real business needs.

Explore how data analytics transforms raw data into insights by data engineers building end-to-end pipelines, sourcing, ingestion, storage, processing, analytics, and visualization on AWS.

Learn core data engineering terms, including database schema, data warehouse, OLAP vs OLTP, ETL, and ODBC/JDBC, to understand how data is modeled, stored, and analyzed on AWS.

Learn how AWS Glue provides a serverless data integration catalog of databases and tables discovered by crawlers, with S3 data, classifiers, partitions, and permissions.

Discover AWS Glue as a serverless ETL platform that uses Spark in-memory processing to extract, transform, load data, join tables, and perform streaming and batch analyses via Glue Studio.

Learn to configure AWS Glue ETL Part 2 by connecting to Aurora with Secrets Manager, building crawlers to create the catalog, and running ETL jobs in Glue Studio.

Join sales, customers, and products data in a visual Glue ETL workflow to aggregate sales by zip code and product type, then store the results as parquet in S3.

Explore how AWS Glue workflows coordinate crawlers and jobs into a single orchestrated ETL. Use triggers and nodes to schedule, run, and monitor complex pipelines in the console.

Explore how AWS Glue bookmark tracks processed data to enable incremental processing, with rewind, backfilling, and reset options, controlled via a simple job-level checkbox in the console.

Explore AWS Glue execution types: standard for fast job start time, auto scaling for cost efficiency, and flex for time-insensitive workloads using spare capacity.

Explore AWS Glue data quality, a serverless pay-as-you-go tool that profiles data, auto-generates and customizes rules, and checks data quality at rest and in transit for reliable ETL outcomes.

Explore data quality at rest in AWS Glue Data Catalog, using data quality rules and the Dcdl language, with automated recommendations for ETL jobs and data in transit.

Explore AWS Glue Databrew, a visual data preparation tool with a point-and-click interface for cleaning and normalizing data for analytics and ML, featuring over 250 transformations, data sets, and recipes.

Explore AWS glue's additional features, including Secrets Manager for credential management, VPC endpoints for private S3 access, worker types for faster processing, and a central schema registry for versioned schemas.

Explore Amazon Redshift as a purpose-built, fully managed data warehouse for petabyte-scale analytics, with columnar storage, OLAP, and massive parallel processing.

Explore redshift architecture, including leader and compute nodes, execution plans and a C plus plus file, plus storage options with Ra3, Aqua acceleration, and slices for scalable analytics.

Explore how to create a provisioned redshift cluster, choose node types dc2 and Ra3, set up networking, IAM role, and access via JDBC/ODBC with a cluster endpoint.

Explore Redshift query editor v2, a web-based SQL client to connect to your Redshift cluster, author and run queries, save and share them, and visualize results.

Explore how Amazon Redshift distributes data across cluster slices using distribution styles—key, all, even, and auto—to enable massive parallel processing and minimize data movement for joins and queries.

Learn how to manage a persistent AWS Redshift cluster: perform elastic or classic resize, pause and schedule maintenance, edit parameters, create snapshots, and monitor upgrades and quotas.

Learn how Redshift data API enables SQL queries via an HTTP endpoint without persistent connections. Use Secrets Manager or temporary credentials for asynchronous, event-driven workflows with AWS services.

Discover how to query structured and semi-structured data directly from S3 using redshift spectrum, avoiding data copy, via external schemas and catalogs (Glue, Athena, or Hive Metastore).

Explore Amazon Redshift serverless, using namespaces and workgroups to separate storage and compute, enabling self-service analytics with pay-per-use pricing and automatic provisioning, scaling, and maintenance.

Explore how Redshift serverless uses namespaces and workgroups to manage storage and compute, connect to data sources, and monitor usage with auto-scaling and data sharing.

Learn how materialized views in Redshift speed dashboards by storing precomputed results, with auto or manual refresh, incremental updates, and automatic query rewriting.

Explore how redshift workload management (WLM) uses queues with memory and slots to route etl, ad hoc, and reporting queries, with automatic or manual mode and concurrency scaling.

Share live data across redshift clusters without copying or etl by using redshift data sharing, enabling producer and consumer roles across accounts and regions with cost efficiency.

Integrate data API events with EventBridge to trigger Lambda or SNS actions and notify endpoints. Learn stored procedures, scheduling with Query Editor 2.0, vacuum and copy to optimize Redshift performance.

Explore streaming data, its high-volume, near real-time processing, and how streaming pipelines with producers, storage, and consumers deliver timely insights in AWS.

Explore AWS streaming services, including the Kinesis family with data streams, data firehose, and video streams, and learn how producers, storage, consumers, and destinations form a streaming pipeline.

Explore Amazon Kinesis as a platform for streaming data on AWS to load and analyze streaming data, a family including Kinesis Data Stream, Data Firehose, and Apache Flink's managed service.

Learn how Amazon Kinesis Data Streams handles data with producers and multiple consumers across shards, using partition keys and data blobs to route and process records.

Understand Kinesis stream capacity modes—on-demand and provisioned—and compare costs and scalability. Learn to create a data stream in the console, configure shards, and set retention.

Create a Kinesis stream, put and get records via shard iterators in AWS CLI, using trim horizon, sequence number, after sequence number, and timestamp.

Generate and flood an AWS Kinesis stream with dummy data using the Amazon Kinesis Data Generator, which deploys via CloudFormation and Cognito, and customize templates for sensor or person data.

Explore sending data to a Kinesis stream via Kinesis Producer Library, AWS SDK methods (put record, put records), or Kinesis Agent, with options EventBridge, IoT Core, Aurora, and tools.

Consume data from a Kinesis data stream by building custom applications with the Kinesis Client Library (Kql) or via managed services like Kinesis Data Firehose, Flink, Glue, and EMR.

Learn how enhanced fan out in AWS Kinesis data streams guarantees two MB per second per consumer, supports up to twenty consumers, reduces latency, and introduces per-consumer cost considerations.

Amazon data firehose, a stream integration service that ingests data from producers or Kinesis streams, transforms with Lambda to Parquet or ORC, and delivers to S3, Redshift, OpenSearch, or Snowflake.

Learn how dynamic partitioning streams sensor data to Amazon S3 via Firehose, automatically partitioning by sensor ID for efficient Athena queries and streamlined data organization.

Distinguish Kinesis data stream and data firehose by ingestion and storage versus transfer and format conversion, with on-demand scaling, data retention differences, replay capabilities, and near real-time delivery.

Discover how the fully managed Amazon managed service for Apache Flink enables real-time streaming analytics with SQL, Java, Scala, or Python, using diverse sources and sinks and automatic scaling.

Build a streaming data pipeline with Apache Flink on AWS: replay taxi data from S3 to Kinesis, process with a Java Flink app, and visualize results in Kibana via OpenSearch.

Explore Amazon Managed Flink Studio to interact with streaming data using a Zeppelin-powered notebook, enabling interactive development with SQL, Python, or Scala and AWS Glue Catalog integration.

Discover how Apache Kafka works as a distributed event streaming platform on AWS, including clusters, brokers, topics, partitions, replicas, ZooKeeper, and Kafka Connect data sources and sinks.

Migrate on-premises Apache Kafka to AWS with Amazon MSK, a managed service that provisions Kafka clusters and Kafka Connect, and handles VPC connectivity and operational overhead.

Create an Amazon MSK cluster using quick or custom creation, choosing provisioned or serverless. Configure brokers, VPC, subnets, security groups, roles, and set up producer and consumer workstreams.

Learn how to set up an MSK access role, deploy an EC2 client, configure security groups and a VPC, and create and describe Kafka topics using bootstrap endpoints.

Learn to send and receive messages on an AWS MSK Kafka cluster using console producers and consumers, monitor metrics in CloudWatch, and clean up resources after testing.

Explore Amazon MSK's serverless option to run Kafka clusters without right sizing, with auto scaling, endpoint access, and throughput-based pricing, and compare serverless with provisioned mode.

Compare MSK provisioned versus serverless clusters to choose based on workload predictability and control. Serverless auto scales and eliminates capacity configuration, while provisioned offers broker-level control and pay-for-use pricing.

Learn how Amazon msK connect provides a fully managed kafka connect deployment to connect sources and sinks, auto scales connectors, and integrate with s3 and confluent connectors.

Compare Amazon Kinesis data stream with Amazon MSK by mapping stream to topic, shard to partition, and sequence number to offset, noting serverless versus provisioning and replication implications.

Explore how AWS enables processing big data workloads by leveraging volume, variety, and velocity to collect, store, process, and analyze data from diverse sources in real time.

MapReduce enables distributed and parallel processing of huge data sets by splitting work into map and reduce phases, with AWS helping run these jobs to generate centralized reports.

Explore the big data ecosystem from ingestion and storage to processing, analytics, and visualization, and how AWS supports these workloads with EMR and other distributions.

Explore Amazon EMR, a managed cluster platform that simplifies running big data frameworks like Hadoop and Spark, with primary, core, and task nodes for scalable processing.

Explore storage options for Amazon EMR, including HDFS on core nodes for fast, data-aware processing and EMRFS on S3 for persistent, decoupled storage across clusters.

Learn to create an EMR cluster on AWS console, planning storage and selecting a big data framework. Launch, connect, and run a PySpark job with results stored in S3.

Learn how to launch and manage an AWS EMR cluster, submit a PySpark step, monitor running and completed jobs, and retrieve streaming results from S3 without incurring idle costs.

Migrate an on-prem Hadoop cluster to Amazon EMR using AWS migration tools, and configure an external Hive metastore with Glue Data Catalog or Aurora.

Leverage Amazon EMR Serverless to run Spark and Scala without cluster management, paying only for provisioned resources. Automatically scales with multi-AZ resiliency and integrates with Airflow and AWS Step Functions.

Explore cost optimization strategies for an EMR on EC2 workload by reserving primary and core nodes, using spot task nodes, Graviton instances, and auto scaling to tailor capacity.

Explore how AWS data lakes use S3 storage to capture structured, semi-structured, and unstructured data, enabling batch or streaming ingestion and analytics with Redshift, Athena, QuickSight, EMR, and SageMaker.

Compare data warehouse and data lake, detailing their load versus understand philosophies, schema handling, data types, and user roles, and see how lakehouse architecture enables co-existence on AWS.

Explore AWS Lake Formation and its centralized security and governance layer for data lakes on S3, plus simplified S3 permissions, blueprint-driven ingestion, and Glue integration.

Understand how AWS Lake Formation builds a data lake with ingestion, security, and cataloging on S3 and Glue. Discover granular access controls, tagging, and roles for data governance and auditing.

Set up a secure data lake on AWS by registering S3 storage, creating a Glue database, and granting Lake Formation permissions, with hybrid IAM and Lake Formation access.

Configure a glue crawler to populate the data lake in S3, assign S3, Glue, and CloudWatch permissions, and resolve Lake Formation permissions to populate the data catalog with tables.

Discover how AWS Lake Formation enables fine-grained data lake permissions, combining named data catalog and tag-based access, with column, row, and cell level controls and data filters.

Learn how tag based permissions use Lake Formation tags to control access to databases, tables, and columns by environment and data sensitivity, with inheritance and GDPR tagging.

Learn how open table formats such as Delta Lake, Apache Iceberg, and Apache Hudi enable broad data operations and AWS tool integration for read and write, batch and streaming ingestion.

Discover how Amazon Athena provides a serverless SQL interface to query data stored in S3 using ANSI SQL, with DDL and DML support.

Explore how Amazon Athena uses the AWS Glue Data Catalog as a metastore to query data stored in Amazon S3 in Avro, Parquet, or JSON via tables and a crawler.

Learn to optimize AWS Athena queries by using columnar formats, partitioning, compression, and query result reuse to reduce data scanned and costs, including create table as select conversions.

Explore how to optimize Athena queries by comparing CSV and Parquet formats, using saved queries, query reuse, and create table as select to reduce data scanned and improve performance.

Learn how to use Amazon Athena workgroups to isolate workloads, set data usage and cost limits, monitor with CloudWatch metrics, and enforce access through IAM policies and alerts.

Discover federated query across multiple data sources, enabling analytics without moving data. Learn how Athena uses Lambda data source connectors and spill bucket to query S3, MySQL, DynamoDB, and more.

Discover how Athena federation connects diverse data sources via preconfigured connectors and Lambda-backed data sources, enabling a single query across DynamoDB, Redis, Edge Base, and more.

Explore data visualization with Amazon QuickSight, a serverless, portal-based modern bi dashboard that connects to AWS and on-prem sources, supports analysis, dashboards, ml insights, nlp, and embedded analytics.

Learn how to get started with Amazon QuickSight, including signing up, creating data sets, uploading a CSV, and building interactive dashboards with SPICE for high-performance analytics.

Learn to visualize Amazon Athena results in QuickSight by configuring data sources, managing permissions, and troubleshooting access to S3, KMS keys, and federated queries.