Mastering AWS Elastic Map Reduce (EMR) for Data Engineers

PowerShell is an advanced version of the DOS prompt. The lecture covers launching PowerShell on Windows 11 and 10, customizing settings, and using SSH to remote machines without PuTTY.

Download and install Visual Studio Code on Windows, add it to the path, then launch the editor and open a folder as a workspace.

Install the remote development extension kit in Visual Studio Code on Windows to enable remote ssh, wsl, and containers, and access files via ssh or wsl.

Plan an emr cluster to support data engineering and analytics. Schedule with airflow and jupyter for ad hoc queries, with auto scaling, and spark jdbc reporting to BI tools.

Create an EC2 key pair in the AWS console, save the .pem file to ~/.ssh, and set 600 permissions to enable SSH to EC2 and EMR clusters.

Configure an EMR 6.3.0 cluster with Spark and Glue catalog using advanced options, enable EMR managed scaling, logging to S3, spot instances, and a 1-master, 2-core, 0-task configuration.

Review how an AWS EMR cluster is created, focusing on the summary tab, master node access, configuration details such as EMR version, Hadoop 3.2.1, and Spark 3.1.1, and security groups.

Access EMR cluster application interfaces, including the YARN timeline server and Spark history server, via open ports or an ssh tunnel on the master node.

Review emr cluster monitoring by examining cluster, node, and io views, noting insights appear after jobs run. Adjust graph size and granularity to troubleshoot using time ranges and dashboards.

Review EMR cluster hardware, including master and core configurations, vcore, memory, and storage, and explore EMR managed scaling with minimum and maximum values and the option for custom automatic scaling.

Review emr cluster configurations by inspecting overridden properties and json, noting glue data catalog for spark metastore, and troubleshoot via runtime overrides in the cluster dashboard.

Review EMR 6.3.0 cluster events from creation to resizing, noting Hadoop debugging steps and the role of the AWS Glue Data Catalog for Spark metadata in troubleshooting.

Configure and review EMR cluster steps, including Hive and Spark jobs, add custom jars or scripts, review executed steps, and control outcomes with actions on failure.

Review bootstrap actions in EMR clusters, configure and run shell scripts from S3 to install prerequisites on all nodes, and monitor status via the EMR dashboard.

Connect to the EMR master node via ssh with the public DNS and private key, then use sudo to access s3 and CLIs like Spark, Shell, Pyspark, Spark Sequel.

Learn to disable termination protection and terminate an emr cluster using the console or aws cli, emphasizing the two-step process to prevent accidental termination in production BI applications.

Clone an EMR cluster by duplicating the terminated cluster’s configuration, including services like Hadoop and Spark, instance groups, scaling, and logging, then connect to the new master via SSH.

List s3 buckets and objects from the emr cluster master node using the preconfigured aws cli and emr roles. Inspect the itv-github bucket with landing and ghactivity folders.

Explore listing and viewing AWS S3 buckets and objects from an EMR cluster using HDFS dfs and Hadoop fs commands, comparing S3 paths, outputs, and CLI behavior.

Learn to manage s3 data on an EMR cluster using hdfs/hadoop cli, copying files from local to s3, verifying with hdfs and aws s3 commands, and deleting by date.

Explore how AWS EMR spins up clusters with Hadoop and Spark to build scalable data pipelines, and review the Glue catalog databases and tables, including Retail DB, for EMR access.

Create an emr cluster with spark and hadoop, configure glue data catalog integration, and validate access to retail_db database and its tables using spark sql.

connect to the master node of the AWS EMR cluster via ssh to start spark-sql with yarn and access glue catalog databases, retail db tables and sample queries.

Learn to use the PySpark CLI on an AWS EMR cluster to process data in S3 with Spark dataframe APIs, group by status, convert to lowercase, and explore SQL-style queries.

Launch spark-shell on AWS EMR to process data with dataframe APIs in Scala, reading JSON from S3, previewing schema, and counting by status using group by and lower.

Attach notebooks to an AWS EMR cluster to run code interactively with Spark. Configure notebooks with Jupyter Enterprise Gateway and Glue Data Catalog to enable connected development and testing.

Set up a remote development environment for an AWS EMR cluster with a bootstrap script staged on S3. Install boto3 via bootstrap actions and prepare the cluster for future runs.

Provision an elastic IP in AWS EC2 and map it to the EMR cluster master node for a stable public address, including allocation steps in the AWS console.

Provision an EMR cluster with a bootstrap script and Elastic IP, using EMR 6.6.0 with Hive, Jupyter Enterprise Gateway, and Spark 3.2.0, integrated with the Glue data catalog.

Diagnose EMR bootstrap failures by inspecting s3 logs for bootstrap actions and diagnosing pip not found errors in bootstrap.sh.

Troubleshoot aws emr bootstrap by cloning the cluster, removing bootstrap actions, and recreating it with a corrected bootstrap.sh using /usr/bin/pip3 and boto3, then verify master access.

Clone and validate an emr cluster with a bootstrap action, review logs, and ensure boto3 is installed across nodes for reliable development access.

Set up a python virtual environment in a VS Code workspace using python 3.9 named me-venv, then activate it. If needed, select interpreter to ensure the venv integrates with workspace.

Learn to use Python boto3 to manage EMR clusters, including associating an elastic IP with the master node, and explore EMR APIs via a Jupyter notebook in VS Code.

Install and verify boto3 in a python virtual environment, upgrade pip, and test AWS access by creating an s3 client and listing buckets to prepare for EMR cluster API exploration.

Set aws_profile using a .env file in Visual Studio Code to switch to the analytiqs account and explore the boto3 APIs to manage EMR clusters, starting with listing active clusters.

Learn to list emr clusters with boto3, filter by running or waiting states, obtain the cluster id, and describe the cluster to retrieve the master public dns name.

Retrieve the EMR master node's EC2 instance id by listing instances with the cluster ID and matching its public DNS to the master DNS, enabling elastic IP association.

Use boto3 to locate the EMR master instance id and its elastic IP allocation id, then associate the elastic IP with the master node using AWS EC2 APIs.

Learn to associate an elastic IP with the EMR master node using boto3 after obtaining the master instance id and allocation id with ec2 client.

Learn to set up an EMR notebook environment on a cluster using an IAM user, run Python3 or PySpark kernels, and access S3 data with Spark for development.

Review the EMR master node connection via VS Code remote development, configure ssh, and set up the repository on the cluster to support the application development lifecycle.

Set up a git repository on the EMR master node by installing git with yum, cloning from GitHub, and configuring a VS Code workspace for development.

Learn best practices for configuring a centralized AWS EMR development cluster for multiple developers, using individual master-node accounts, passwordless access, and VS Code remote development.

Install Pylance and Python extensions in the remote Visual Studio Code window to enable Python syntax highlighting and auto complete for EMR cluster development; validate boto3 and s3_client usage.

Explore how EMR cluster uses global Python and Spark binaries, verify Python 3.7.10 and Spark 3.2.0 in /usr/lib/spark, and run pyspark to start your first Spark application.

Master the development lifecycle on an AWS EMR cluster by running Spark Submit in local or yarn mode, and adjust util.py get_spark_session to use yarn for dev and production.

Develop your first spark application on an EMR cluster using VS Code with a remote connection, configuring the Python path for pyspark.

Create a modular spark session function in Python, supporting dev and prod yarn modes, using util.py and a main guard, then run with spark-submit to execute Spark SQL queries.

Upload gharchive data from gharchive.org to an aws s3 bucket for emr-driven development. Stage locally, validate uploads, and manage json.gz files with recursive s3 transfers.

Develop dynamic read logic for spark applications by building from_files with a spark session, using format and load to read json files from s3, controlled by environment variables.

Apply spark APIs to transform a dataframe by deriving year, month, and dayofmonth from created_at using withColumn, and save the transformed data via a process.py workflow in EMR.

Partition data by year, month, and dayofmonth, coalesce to 16 files, and write the transformed dataframe to s3 in parquet format using spark's dataframe writer with append mode.

Resize AWS EMR cluster via the web console and adjust minimum and maximum core and task nodes. Explore elastic scaling and its impact on cost and validation when applying changes.

Validate productionized code by running spark submit in prod, resize emr cluster, process json to parquet in s3, verify counts by date with pyspark, and monitor emr capacity.

Back up mastering EMR to S3, optionally as a zip, and terminate the cluster during extended breaks. Create a new cluster, clone the repository, and resume with an elastic IP.

Clone and resize an emr cluster to deploy spark applications on non development clusters, configure uniform master/core/task nodes, enable autoscaling, and prepare a zip file on the master.

Set up the AWS EMR master node, map an elastic IP, and prepare a code repository to build and validate a zip of Spark applications for dev and non-dev clusters.

Adjust EMR cluster capacity by setting the minimum on-demand limit and adding task instances to resize for larger data sets, balancing on-demand and spot provisioning to ensure timely processing.

Build a zip of the python program files for an EMR Spark application, clean old zips, and validate locally in client and cluster modes before EMR deployment.

Validate a spark application in client mode on EMR using a zip file and app.py, then verify results and prepare for cluster mode deployment.

Run a spark application in EMR cluster mode by propagating environment variables with --conf and passing zip files and app.py via --py-files. Validate the target location to ensure successful execution.

Copy the zip file and app.py to s3 and run spark submit in cluster mode on emr with deploy mode cluster. Verify no exceptions by listing s3 and confirming files.

Learn how to deploy a spark application as a step on an existing EMR cluster using cluster mode and spark submit options, with artifacts stored in S3.

Learn to configure a state machine to delete multiple S3 objects at once by creating test files in cloud shell, copying them to S3, and validating with object listing.

Validate a spark application deployed as a step on an AWS EMR cluster by reviewing STD ERR logs and using aws s3 ls to confirm the target files.

Master how to manage AWS EMR with boto3 by setting up a Mac or WSL development environment in VS Code, and configuring AWS profiles for notebooks and S3 access.

Provision an AWS EMR cluster with Hadoop and Spark using the AWS web console and AWS CLI, configured for EMR 6.6.0 with one master and two core instances.

Mastering EMR clusters with the aws cli, learn to list active clusters, describe clusters, list instances and steps, modify cluster step concurrency, and terminate clusters, then prepare for boto3 programming.

Explore how to manage AWS EMR clusters with boto3 by listing clusters, viewing statuses, and using run_job_flow and terminate_job_flows to create, describe, and terminate clusters.

Using boto3, learn to create and start an EMR cluster with run job flow, configure instances, release label, logUri, applications, configurations, and manage steps and bootstrap actions.

Learn to create an AWS EMR cluster or job flow using boto3, configuring run_job_flow with applications, configurations, and instances, including master and core roles and keep-alive settings.

Prepare data sets for a Spark step on the EMR cluster by cleaning the S3 location, validating source files for 2021-01-13 to 15, and adding the Spark step with boto3.

Learn how to add spark application as a step to an existing AWS EMR cluster using boto3, by configuring HadoopJar steps with jar and arguments, and using add_job_flow_steps.

Learn how to add a Spark application as a step to an AWS EMR cluster using boto3, validate execution with PySpark counts, and troubleshoot via a second EC2 key-paired cluster.

Terminate the AWS EMR cluster after adding steps using add_job_flow_steps, choosing API, AWS CLI, or the web console, and verify status with describe cluster.

Create an AWS EMR cluster for 2021 January 13th to 15th with steps, run steps and Spark app, then terminate the cluster with one command, validating data at source and target.

Set up a local development environment to build EMR pipelines with AWS Step Functions, using the workflow studio and web console; pull latest changes from the matching-emr repository.

Explore AWS step functions for coordinating distributed applications. Learn the core terms—state machines, activities, and pipelines (workflows)—and how to manage end-to-end pipelines via the console, CLI, or boto3.

Explore EMR based pipelines with AWS Step Functions by creating state machines in Workflow Studio. Start executions, use pass for input, and integrate EMR cluster steps to automate workflows.

Review the AWS IAM role generated for a basic state machine, inspect related policies and permissions, and adjust trust relationships as needed in the IAM console.

Learn to build an EMR cluster workflow with create cluster, add step, and terminate cluster using AWS Step Functions and Workflow Studio, including production considerations and run job flow details.

Learn to create an EMR cluster with AWS Step Functions by translating run job flow parameters into JSON for the create cluster action, including name, logUri, release label, and instances.

Attach the step function role with permissions to create an AWS EMR cluster, including service, job flow, and auto scaling roles, and attach needed policies.

Learn how to add a spark step to an AWS EMR cluster using a state machine, configure JSON arguments, and pass the cluster ID dynamically while waiting for completion.

Validate the successful execution of a step added to an aws emr cluster via step functions, review logs and s3 outputs, and plan termination handling in the next lecture.

Add a terminate cluster action to the EMR state machine, pass cluster id from input to output, and verify success by inspecting the state machine execution.

Learn how to validate a state machine that runs a Spark application on AWS EMR by passing step input, monitoring cluster creation, executing steps, and verifying outputs and logs.

Orchestrate workflow with AWS Step Functions and EMR, creating a cluster, running a spark application as a step, cleaning target location, and terminating the cluster to reach a succeeded state.

Enhance the EMR-based data pipeline by validating target files, deleting existing ones, creating the cluster, running the Spark application, and terminating clusters, while building a new state machine for integration.

Create a state machine in AWS Step Functions to validate S3 locations and delete files when present. Learn to configure minimal S3 access policies for the execution role.

Configure an inline s3 policy for a step functions role to validate and delete files in the aiGitHub bucket, granting list, read, and delete permissions.

Create an empty json file in S3, copy it into aiGitHub/ghsm demo, and validate list objects using a state machine in Workflow Studio to confirm permissions and successful execution.

Learn how boto3 APIs relate to AWS Step Functions and S3-based states, enabling state machines to manage and delete S3 objects using bucket and prefix parameters.

Add the ability to delete single or multiple S3 objects by key or JSON array, configure the bucket and permissions, and troubleshoot access denied errors in the state machine workflow.

Troubleshoot and fix permissions to delete objects in S3 via a state machine, updating IAM policy to grant read, write, and delete on all objects in the aiGitHub bucket.

Learn how to pass input to an AWS Step Functions state machine, parameterize bucket, prefix, and key, and propagate input to delete objects in S3.

Set up and validate a state machine to delete multiple S3 objects by creating test files, copying them to S3, and iterating commands to delete all at once.

Learn to process multiple s3 objects with a map state in an AWS step function, wiring list objects output to map and passing bucket and prefix for execution.

Learn to extract s3 object keys from list objects using a Step Functions map and pass with input path, then pass keys to s3 delete object.

Implement delete S3 object logic in the AWS step function state machine, replacing pass with S3 delete object, wiring bucket and key inputs, and noting lambda-assisted array augmentation.

Develop an AWS lambda function to update an array with bucket and S3 object details using Python 3.9, deploy, test with events, and prepare contents for state machine data processing.

Add a lambda function to the state machine to pass s3 bucket and key details for deleting objects, test the workflow, and resolve permission issues preventing invocation.

Learn to extend an AWS step function state machine using a choice rule in workflow studio to handle s3 list objects gracefully, invoking lambda only when contents exist, else end.

Explore how map concurrency in AWS Step Functions enables parallel deletion of S3 objects, with practical guidance on setting maximum concurrency and validating parallel execution.

Learn to invoke a secondary AWS Step Functions state machine from another state machine to validate a target S3 location before Spark writes files, using ARN and input.

Integrate S3-based state machine with EMR state machine to validate S3 objects, delete them if present, then create EMR cluster, run Spark as a step, and terminate the cluster.

Integrate two AWS step functions state machines—an EMR-based ghactivity converter and a validate S3 target location—ensuring validation precedes EMR pipelines, with backups via copy-to-new and git versioning.

Learn to grant aws step functions permissions via iam roles for cross-state machine invocation, enabling emr spark jobs and s3 target location validation.

Learn to update an AWS Step Functions state machine to invoke an EMR workflow that validates an S3 location, using Workflow Studio and proper input parameter handling.

Propagate original input through the EMR and AWS Step Functions state machine to add a Spark step to the cluster, discarding output and routing input to downstream states.

Validate an AWS EMR-based step function state machine execution by inspecting each state for file presence, confirming deletion, and reviewing cluster spark deployment and logs to ensure successful execution.

Run the AWS step function state machine to validate S3 object deletion, re-running with the same inputs to confirm files are removed as the cluster is created and Spark runs.

Learn to add source location validation in the AWS Step Functions state machine, ensuring the pipeline only runs when the S3 source has files; otherwise it fails.

Update AWS step function state machine to validate source S3 location by using a list objects condition in Workflow Studio, parametrize bucket and prefix, then route to validated or path.

Learn to validate an s3 source location within a step function state machine, run updated executions, and manage positive and negative paths with source and target consistency checks.

Develop an AWS Lambda function to verify 24 files exist in S3 for a given date, then trigger EMR Spark processing to write parquet to the target.

Attach an inline policy to the state machine role to invoke the lambda function, then review and specify the function ARN and name to enable execution.

Develop and test a state machine that invokes a lambda to validate the source file count. Manage permissions and routing between check source count, pass, and validate s3 target.

Master best practices for AWS Step Functions state machines orchestrating EMR clusters and Spark jobs, including avoiding hard-coded inputs, enabling incremental runs, and integrating with organizational job-tracking frameworks.