Complete AWS Certified Data Engineer Associate - DEA-C01

Discover how AWS S3 uses buckets and objects stored per region with unique names and strict naming rules. Examine use cases, storage classes, durability, availability, lifecycle rules, and versioning.

Learn how to create an S3 bucket by selecting a region, choosing a globally unique bucket name, and configuring initial options like versioning and cost-tracking tags; then upload an object.

Learn hands-on how to upload files to an S3 bucket by creating a folder, dragging and dropping from your local PC, and viewing object details like region and key.

Compare streaming and batch ingestion, highlighting real-time processing for time-sensitive data like fraud detection. Default to batch for large volumes and cost efficiency, with Kinesis, Glue, and Lambda.

Explore AWS Glue, a fully managed, serverless ETL service that visually builds data pipelines across S3 and Redshift, and uses the Glue Data Catalog and crawlers for schema inference.

Configure an AWS Glue crawler to detect metadata from an S3 folder and populate the AWS Glue data catalog with databases and tables. Use Athena to query the metadata-backed tables.

Explore how AWS Athena delivers serverless, pay-as-you-go SQL queries on data stored in S3, including CSV, Avro, and parquet formats, with Glue data catalog and Kinesis integration, visualization via QuickSight.

Query data stored in S3 with Athena, after configuring a query result location in S3 and exploring the customers table for quick previews and simple sql calculations.

Explore federated queries that let you run sql across relational, non-relational, and object data sources beyond s3 through Athena connectors for ad hoc analysis and custom data sources.

Pay only for the queries you run, based on data scanned, and improve performance with partition pruning, partition projection, and query result reuse, plus Parquet or Apache Awsy formats.

Explore Athena workgroups to isolate queries for different teams and use cases, configure query execution and result locations, manage access with IAM policies, and optimize performance and cost.

Learn to set up and configure a work group in Athena, choose the engine (Athena SQL or Spark), specify a result bucket, and apply usage alerts for cost control.

Learn how AWS Glue costs accrue through dpus-based compute for crawlers and ETL jobs, billed by seconds with a ten-minute minimum, plus catalog and interactive notebook costs.

Learn to set AWS budgets in the cost and billing center, configure alarms and email notifications for cost and usage budgets, including forecasted exceedances, saving plans, and reservations.

Set up budgets in the AWS billing console using Cost Explorer, create a zero spend budget and a 5 USD monthly budget, and configure alert thresholds and email notifications.

Set up a Glue etl job to move data from an s3 csv source to a parquet destination using a visual etl editor with schema inference and data catalog updates.

Schedule etl jobs and crawlers in a hands-on approach, creating schedules with daily or monthly frequencies and managing runs to control costs.

Explore stateful vs stateless systems and data ingestion, showing how bookmarks in AWS Glue enable incremental load and how Amazon Kinesis Data Streams support real-time processing.

Explore stateless versus stateful data ingestion in AWS Glue, perform incremental loads with bookmarks, and verify results in Athena after ETL jobs process new and existing files.

Enable bookmarks in AWS Glue to create a stateful data ingestion with incremental loads, tracking loaded data so only new records load on subsequent runs.

Explore AWS Glue ETL transformations, from discovery and extract to transform and load, including data catalog updates, filtering, joining, aggregations, and ML-based find matches.

Explore aws glue data quality to improve integrity with rule-based checks and anomaly detection for data in S3 or in transit. Configure the rule set editor, validate data, and preview.

Master glue workflows to orchestrate crawlers and jobs with triggers, conditions, and monitoring, and extend control with step functions, EventBridge, and lambda-driven, scheduled, or on-demand triggers.

Explore glue workflows in practice by orchestrating crawlers and ETL jobs with triggers. Set up on-demand or scheduled runs, cron expressions, and event-based ETL using the graphical interface.

Explore glue job types, from spark-based visual workflows and notebooks to script-from-scratch, python shell, and ray engines, including dpu and gpu options, spark streaming, and flex execution.

In this hands-on demo, see how the engine is automatically set by tasks and data sources in the visual editor, with two dpus minimum, and explore flex execution.

Partition data on AWS Glue to speed queries and reduce iops by scanning only relevant partitions, while enabling independent ETL on partitioned data in S3.

Discover AWS Databrew's no-code visual interface for data preparation, cleaning, and transformation with 250+ steps, recipes, and automated, scheduled jobs that integrate with S3, Redshift, AWS Lake Formation, and IAM.

Explore essential AWS Glue DataBrew transformations for data preparation and engineering, including nest and unnest map, array, and struct, pivot and unpivot, transpose, joins, splits, filters, and type conversions.

Explore AWS Glue DataBrew in practice to clean, normalize, and transform data interactively for analytics, reporting, and machine learning, with project recipes and scheduled outputs.

Explore AWS Lambda as a serverless, scalable compute service that runs code automatically in response to events, supports multiple languages, and handles data ingestion and processing from S3 or Kinesis.

Set up event-driven data ingestion with AWS Lambda triggered by S3 uploads. Move files from a source bucket to a destination bucket using Python, with IAM permissions and monitoring.

Explore how lambda layers package dependencies into a zip file and share code across multiple functions, enabling centralized management and easier updates.

Learn how replayability enables re-ingesting data in streaming workflows to fix errors, adapt to schema changes, and ensure robust, consistent results with AWS services like Kinesis.

Explore Amazon Kinesis and its data streaming services for ingesting high-volume data with Kinesis Data Streams, delivering to destinations via Kinesis Firehose, and real-time analytics with managed Apache Flink.

Explore how Amazon Kinesis Data Streams ingests data via producers, formats records with partition keys, scales with shards, and connects to consumers through Kinesis Data Firehose and AWS Lambda.

Understand throughput and latency in data streaming with aws kinesis: how shards scale throughput, how propagation delay affects end-to-end processing, and how polling intervals influence data availability.

Set up and configure a Kinesis data stream, provision two shards, publish and consume records, and process data with a lambda function, exploring producers, consumers, and monitoring.

enhanced fan-out for Amazon Kinesis Data Streams gives each consumer a dedicated two megabytes per second throughput via http2, enabling up to 20 consumers per shard and reducing latency.

Demonstrate pulling and consuming stream data with CLI records, partition keys, and shard iterators, decoding base64 data, and outline a lambda to process and write results to S3.

Connect a Lambda function to a Kinesis data stream by creating an IAM role, configuring a trigger, and deploying Python code to write to S3 in real time.

Explore common issues in streaming data pipelines, from slow writes and throttling to hot shards and read delays, with practical fixes like partition key strategies, batching, and adjusting shard counts.

Explore Amazon Kinesis Data Firehose, a fully managed, near real-time data delivery service that buffers, transforms with AWS Lambda, and loads data to S3, Redshift, OpenSearch, or other destinations.

Create an Amazon Data Firehose stream using a Kinesis Data Streams source and an S3 destination, with optional transformations, buffering, cost calculation, and demo data testing.

Transform data on the fly in a firehose stream by integrating an aws lambda function, selecting a blueprint, and configuring the output format.

Explore Amazon managed service for Apache Flink to process streaming data with near real-time analytics. Integrate Kinesis Data Streams, MSK, and S3; enable stateful processing, anomaly detection, and event-driven actions.

Discover Amazon MSK, a managed Apache Kafka service for real-time data streams, with brokers, zookeepers, multi-az replication, and up to ten megabytes, plus producers and consumers.

Explore MSK Connect and MSK Serverless, fully managed Kafka Connect options for connectors to databases, search indexes, and file systems, with auto-provisioning and pay for what you use.

Partition data in S3 using time-based folders or attributes to improve query performance and reduce scanned data, with Glue crawlers and a data catalog managing partition keys and metadata.

Create partitions with Glue by organizing S3 data into London and New York folders, then crawl and query with Athena, illustrating automatic partition creation and efficient partition pruning.

Explore S3 life cycle management and storage classes—standard, intelligent-tiering, one zone infrequent access, and glacier instant retrieval, flexible retrieval, deep archive—driven by data age, access patterns, and cost.

Explore how S3 lifecycle rules automate data management by transitioning objects to cheaper storage classes after a set time. Enable AWS to delete expired data automatically.

Explore S3 storage classes by uploading a file, setting standard, infrequent access, or Glacier, and automate transitions with lifecycle rules to optimize costs and data retrieval.

Learn to create s3 lifecycle rules to move data - from intelligent-tiering to glacier - based on prefix, tags, or size, and to expire after 720 days.

Explore AWS S3 intelligent-tiering, auto transitioning data between frequent, infrequent, and archive tiers, with customizable archival rules and lifecycle settings to optimize costs.

Cross-region replication copies data across regions to boost disaster recovery, reduce latency, and improve availability. Enable one-way replication with deletions not mirrored by default and versioning for both buckets.

Enable cross-region replication in AWS data lake by creating a source and destination bucket, enabling versioning, configuring a replication rule, and validating one-way replication and delete marker behavior.

Explore encryption in transit and at rest in s3, using ssl/tls, server-side encryption with s3 keys or kms, dual-layer and customer-provided keys, plus key management and rotation.

Configure server-side encryption for S3 buckets, choosing between S3 managed keys and AWS Key Management Service keys, override defaults for specific objects, and practice bucket creation, upload, and cleanup.

Explore how bucket policies control access to S3 bucket objects with a JSON policy, detailing version, statement, effect, principle, action, resource, and conditions such as IP range or HTTPS.

Explore s3 access points that create customizable entry points to a bucket, each with its own policy and dns name, supporting internet and vpc origins, for secure, scalable access control.

Explore how object lambda in S3 transforms data on the fly using lambda functions for redaction, format conversion, and data augmentation via object lambda access points, without duplicating data.

Discover S3 event notifications and how bucket events trigger actions—such as lambda functions, SNS/SQS, or EventBridge—for object create or delete, with prefixes and suffixes filtering directories and file types.

Configure S3 event notifications by creating a bucket, an SNS topic, and a subscription, then route object creation events to the topic and verify via email.

Understand the data mesh concept, decentralizing data ownership and treating data as a product with governance, enabled by AWS services like S3, Glue, Redshift, Lake Formation, Athena, and API Gateway.

Explore AWS data exchange, a centralized catalog to find, subscribe to, and use third-party data in the cloud, with marketplace and Lake Formation integration for finance, healthcare, geospatial, and retail.

Explore amazon elastic block store (EBS) as durable, scalable block storage for EC2. It delivers persistent data, replication within an availability zone, volume attachments, snapshots, encryption, and pay-as-you-go pricing.

Learn to provision EBS with snapshots and capacity options, including incremental block-level backups, encryption at rest with KMS, lifecycle policies, and cross-region copy to optimize cost and durability.

Explore creating and attaching EBS volumes to an EC2 instance, set up a gp2 volume alongside the root, and manage delete-on-termination and availability zone rules.

Explore Amazon elastic file system, a serverless, elastic shared storage that scales automatically, provides high availability across multiple availability zones, and enables concurrent access from multiple EC2 instances via NFS.

Manage backups from a single console with AWS backup, apply policies across EC2, RDS, and EFS, and leverage vault lock for immutable, compliant data protection.

Explore hands-on AWS backup to centrally manage and automate backups with backup plans, vaults, encryption keys, and rules, life cycle, regions, and resource assignments for S3 buckets.

DynamoDB, a fully managed AWS NoSQL database that scales horizontally, stores schema-less data, and delivers millisecond read and write latency.

Walk through hands-on creation of a DynamoDB table, defining a partition key (and optional sort key), choosing a table class, capacity mode, and basic encryption.

Explore how DynamoDB builds data into tables of items with scalar attributes, highlighting schema-less design, primary keys, and secondary keys with a book example.

Explore how to create items in a DynamoDB table, defining a partition key, adding attributes, and using schemaless items with optional author and title fields.

Explore how DynamoDB uses a fixed, scalar primary key to uniquely identify items, and compare partition keys with composite keys (partition key plus sort key) for efficient retrieval.

Demonstrate how primary keys define a unique partition key and optional sort key, show you cannot change a primary key after table creation, and introduce composite keys and secondary indexes.

Explore DynamoDB secondary indexes, comparing local secondary keys with global secondary indexes, including partition key rules, flexibility, and capacity considerations for category and manufacturer access patterns.

Explore projecting attributes to DynamoDB secondary indexes, choosing all, key only, or include up to 20 attributes per index to optimize access patterns, storage costs, and DynamoDB streams.

Explore DynamoDB streams to capture changes in real time, with records per write operation and options like keys only, new image, old image, or both, enabling Lambda and cross-region workflows.

Enable DynamoDB streams to capture changes with new and old image for 24 hours, then process them via a lambda trigger and monitor in CloudWatch.

Learn the four high-level DynamoDB apis—the control plane, data plane, streams, and transaction api—and perform crud operations with Partiql and classic apis to manage tables and ensure data integrity.

Explore DynamoDB accelerator (DAX), an in-memory cache that lowers read latency and boosts throughput for DynamoDB. Configure a DAX cluster, manage cache hits and misses, and handle throttling with retries.

Understand DynamoDB capacity modes, on demand and provisioned, and how throughput, cost, and performance guide workload decisions; learn read capacity unit and write capacity unit concepts, auto scaling, and throttling.

Explore capacity modes for tables, including provisioned and on demand, and configure auto scaling with min/max and target utilization for reads and writes, including global secondary indexes.

Explore how DynamoDB's write and read capacity units (WCUs and RCUs) determine throughput, how to calculate them, and how strongly and eventually consistent reads influence capacity and throttling.

Explore provisioned and on-demand capacity in DynamoDB, calculate read and write capacity units using the capacity calculator, compare pricing, and optimize throughput for steady workloads.

Explore hot partitions and throttling in DynamoDB, and apply even partition keys, exponential backoff, burst capacity, adaptive capacity, CloudWatch monitoring, and caching to optimize performance and cost.

Explore TTL in practice by using an expiry date attribute to auto delete items based on time, reducing storage and read/write costs while improving performance.

Explore Amazon Redshift, a fully managed petabyte-scale data warehouse for analytical workloads, with columnar storage, massive parallel processing, and ANSI SQL support.

Explore how redshift clusters use a leader node and compute nodes, with replication, backups, automatic recovery, serverless vs provisioned configurations, and ODBC/JDBC communication.

Create a Redshift cluster and load sample data, then explore permissions with an IAM role and S3 integration, while noting defaults, costs, and snapshot options.

Access and query an AWS Redshift cluster using the query editor v2, connect with database credentials or JDBC/ODBC, and run queries across schemas and sample tables in the console.

Compare Ra3 and Dc2 node types and their storage models, with decoupled compute and S3-backed managed storage on Ra3 and local SSD on Dc2.

Explore elastic resizing and classic resizing to scale a cluster by adjusting compute node slices and node type, with the leader node managing data distribution across slices.

Understand Amazon Redshift snapshots as point-in-time backups stored in S3 with SSL encryption. Explore automated incremental snapshots, manual snapshots, retention defaults, and cross-region sharing without copying.

Resize Redshift clusters and manage snapshots, including manual and automatic backups, restoration to provisioned or serverless namespace, and elastic resize as the recommended method.

Apply the Redshift vacuum to reclaim disk space and reorder data, reducing fragmentation from updates and deletes and boosting query performance with options like full vacuum, sort only, or reindex.

Explore Amazon Redshift integrations with S3, Redshift Spectrum, and Kinesis for streaming ingestion; load from EMR, DynamoDB, or Aurora Zero-ETL, and automate data movement with AWS Data Pipeline and Lambda.

Demonstrates loading data from an S3 bucket into Amazon Redshift using the copy command and a wizard, including creating tables, handling headers and delimiters, and unloading data back to S3.

Learn to unload data from Redshift to S3 using the unload command with a query, bucket, delimiter, and role, producing a single-file output by turning off parallelism.

Compare ETL and ELT for moving data to a Redshift data warehouse. Use in-database SQL transformations, stored procedures, and user-defined functions, or integrate with external ETL tools via JDBC/ODBC.

Use federated queries to run live SQL against external relational sources from redshift, avoiding ETL and data duplication.

Explore materialized views in Redshift to auto refresh pre-computed query results for faster access. Create, refresh, and optimize streaming materialized views using external schemas with Kinesis or Kafka sources.

Learn to use federated queries with external schemas and create materialized views to accelerate complex queries, including how to query and refresh materialized views and compare with standard views.

Query data in S3 with Redshift Spectrum using external tables that reference files without loading them, enabling large scale analytics with standard SQL.

Explore redshift spectrum with a practical step-by-step demo of creating an external schema linked to the glue data catalog and querying an external table from an s3 path.

Explore Redshift system tables and views to monitor health, view running queries, and improve performance. Learn key examples like SSIS query history, STL alerts, vacuum, and SVCS and SVL views.

Learn how the Redshift Data API provides a lightweight https interface to run asynchronous SQL queries against Redshift from serverless and web apps.

Share live data across clusters, accounts, and regions without copying, using outbound and inbound shares. Choose standard, AWS data exchange, or Lake Formation shares with encryption; consumers pay their compute.

Explore Redshift workload management to optimize query performance via priority-based resource allocation, query queues, user groups, and automatic or manual configurations with concurrency scaling.

Accelerate short running queries by default in workload management, prioritizing create table and read-only select statements; machine learning predicts execution time to run short queries in space for faster performance.

Understand Redshift serverless versus provisioned clusters, with automatic provisioning, workload-based scaling, and managed encryption, plus cost control and absence of manual resizing.

Redshift ml lets you create, train, and deploy machine learning models directly in Redshift with SQL, generating in-cluster predictions without data movement.

Enforce authentication and IAM, deploy Redshift inside a VPC with zero-trust default state, enable encryption at rest with KMS or your own keys, and secure data in transit with SSL.

Explore how redshift uses IAM for authentication and manages privileges with users, groups, and roles, including RBAC, nested role permissions, and row-level security and data masking.

Explore fine-grained access control with column-level and row-level security, enabling precise data access. Apply dynamic data masking through masking policies and tied policies with priorities to enforce strongest access rules.