AWS Data Engineer Bootcamp: The Complete Guide

Enable multi-factor authentication on AWS by registering a virtual MFA device, installing an authenticator app, scanning a QR code, and using MFA codes to log in securely.

Use AWS glue to read data from S3, apply a filter where payment status equals success, and write the result to S3 in parquet.

Learn how to use AWS Glue aggregate transformation to group data by country and compute sums, averages, maximum values, and counts, with a practical CSV example.

Master the derived column transformation in AWS Glue to modify existing columns or derive new ones with expressions. Apply load date, year, month, and case-based logic in your ETL pipelines.

Create an s3 bucket, upload an employee csv, and build a glue catalog using a crawler. Then query the data in aws athena and set a query output location.

Launch and connect to an Ubuntu Linux EC2 instance, configure the free tier, set up a key pair and network, and practice basic Linux commands.

Set up an S3 bucket to trigger an SNS notification on file upload, create an SNS topic with an email subscription, and verify email delivery of the notification.

Unifies data lake and data warehouse in a single open-format platform, enabling acid transactions, schema enforcement, time travel, and fast analytics on all data types with the Databricks Lakehouse.

Explain the difference between managed and external tables in Databricks. Managed stores metadata and data in Databricks; external keeps data in cloud storage and metadata in Databricks, with drop behavior.

Set up a PySpark workspace in Databricks, create a catalog, database, and volume, upload a CSV, and prepare a notebook to practice reading files and building a data frame.

Learn to use the PySpark group by transformation to group rows by state and apply aggregations such as sum, average, max, min, and count to calculate totals and statistics.

Learn to connect to external data sources by configuring a catalog, creating a foreign connection catalog, and establishing credentials for mysql or other systems, then query external tables via unity catalog.

Discover how lakehouse jobs in Databricks automate notebooks, SQL queries, and pipelines, scheduling tasks at specific times and enabling ETL-like orchestration with monitoring and retries.

Set up databricks job notifications to alert on failures due to schema changes or cluster issues, with email or Microsoft Teams alerts for success, failure, or warnings.

Explore what a database is, compare relational and non-relational databases, and learn SQL basics, plus how to create a database using SQL Server Management Studio.

Master filtering and sorting data in SQL by using where and order by clauses, filtering for salary above 70,000 and null locations to drive analysis.

Learn to perform sql update operations: set null locations to India, set Siam salary to 1 lakh, and update null salaries, using update, set, and where clauses.

Explore lead and lag window functions in SQL to access previous and next row values, using over and order by transaction date, with practical sales table examples.

Learn how views in sql act as virtual tables that generate data on demand, simplify complex queries, secure data by limiting columns, and maintain consistency with underlying data.

Learn how to reverse a Python list using built-in methods and without them, including using list.reverse and a for loop, with practical examples.

Master duplicate handling in PySpark by removing duplicates with distinct, and identifying duplicates via group by and count on customer id and transaction id.

Practice solving common Python coding questions for data engineer interviews, including finding the second largest element, reversing numbers, missing numbers, reversing words, and checking anagrams, palindromes, and merging lists.

Solve python coding interview questions: remove duplicates from a list by building a unique result, and sort a list without built-in functions using a swap-based loop.

Compare data warehouse and data lake to understand structured data versus unstructured and semi-structured data, ETL vs ELT processing, and how each supports reporting and raw data storage.

Normalization reduces redundancy by splitting data into multiple tables, while denormalization stores data in a single table with duplication to speed up queries.