Docker on AWS (Beginner-to-Advanced)

In this lecture, we will discuss some basic Docker concepts including its Docker’s definition, Container, Containerization and its benefits, Images, Dockerfile, and most importantly: differences between Containers and Virtual Machines.

Let’s get familiar with some ECS terms and concepts like definition of ECS, launch types: EC2 and Fargate, benefits, Task Definitions , Tasks, ECS Cluster, Container Instance and Service.

Now the time has come for us to get our hands dirty in order to understand how ECS works. For this, our best strategy would be is to learn and practice how an ECS cluster is deployed. The part one shows the entire deployment of an ECS cluster using AWS Management Console.

In the second part, we shall explore all components deployed after our cluster was launched, including ECS Instance(s), CloudFormation Stack and Auto Scaling Group (ASG). Also, we will initiate an SSH connection to our ECS instance and execute few commands.

In this demonstration, we will manufacture a cluster prior to creating a new Task Definition.

Let’s finish this process of creating a Task Definition.

Now, we will run a task using the task definition we created in previous lectures.

In this demo, we will create cluster first prior to creating an ECS service from scratch.

We will now finish setting up our ECS service and will then deploy a container through it.

Let’s see how an ECS Service can compensate for stopped tasks by launching new one, and thus we can achieve fault tolerance and high availability through its usage.

In this demo, we will make use of Dynamic Port Mapping strategy while including an Application Load Balancer to work on the top of an ECS Service.

In this second part, we will finally deploy an Application Load Balancer.

In this theoretical lecture, we will understand how the concept of Auto Scaling can be applied to ECS containers deployed using an ECS Service. Also, we will go through different types Auto Scaling policies.

We will make configure a Target Tracking Policy within our ECS Service while using CPU Utilization as a metric.

Now, we will install and run ApacheBench (ab) utility to send thousands of requests to our Application Load Balancer, so that it spikes the CPU usage of our containers, and consequently the number of containers are increased automatically.

Let us now check whether our configured tracking policy can scale out and scale in the number of instances automatically.

This time we will configure a Step Scaling policy to scale out the number of instances step-by-step based on spike in the CPU Utilization of our container instances.

Let us now make use of ApacheBench (ab) utility to test our configured Step Scaling policy

Amazon ECS capacity providers enable you to manage the infrastructure the tasks in your clusters use. Each cluster can have one or more capacity providers and an optional default capacity provider strategy. The capacity provider strategy determines how the tasks are spread across the cluster's capacity providers. When you run a task or create a service, you may either use the cluster's default capacity provider strategy or specify a capacity provider strategy that overrides the cluster's default strategy.

In this demo, we will see how Amazon ECS capacity providers can use Auto Scaling groups to manage the Amazon EC2 instances registered to their clusters. You can use the managed scaling feature to have Amazon ECS manage the scale-in and scale-out actions of the Auto Scaling group or you can manage the scaling actions yourself. This enables you to effectively use cluster auto scaling.

In this demo, we will learn how ECS supports the ability to schedule your containers on either a cron-like schedule or in a response to CloudWatch Events.

When a task that uses the EC2 launch type is launched, Amazon ECS must determine where to place the task based on the requirements specified in the task definition, such as CPU and memory. Similarly, when you scale down the task count, Amazon ECS must determine which tasks to terminate. You can apply task placement strategies and constraints to customize how Amazon ECS places and terminates tasks. Task placement strategies and constraints are not supported for tasks using the Fargate launch type. By default, Fargate tasks are spread across Availability Zones. With all other tasks, default task placement strategies depend on whether you are running tasks manually or within a service

A task placement strategy is an algorithm for selecting instances for task placement or tasks for termination. For example, Amazon ECS can select instances at random, or it can select instances such that tasks are distributed evenly across a group of instances.

A task placement constraint is a rule that is considered during task placement. For example, you can use constraints to place tasks based on Availability Zone or instance type. You can also associate attributes, which are name/value pairs, with your container instances and then use a constraint to place tasks based on attribute.

Let's now see how we can make use of different task placement strategies and constraints using AWS Management Console.

Amazon Elastic File System (Amazon EFS) provides a simple, scalable, fully managed elastic NFS file system for use with AWS Cloud services and on-premises resources. It is built to scale on demand to petabytes without disrupting applications, growing and shrinking automatically as you add and remove files, eliminating the need to provision and manage capacity to accommodate growth.

As we have gone through the basics of EFS, let's not see how we can collaborate it with ECS containers.

Let’s get our hands dirty and implement seven steps to use EFS with ECS. To start with, we will create two security groups – one for ECS container instances and another one for EFS file system, and thereafter manufacture an Amazon ECS Cluster.

Let’s now create an EFS file system and push some content to it using an external EC2 Linux Instance.

In this last part of our demo, we will create a Task Definition using a custom JSON template, run our containers and view the outcome.

In this tutorial, you set up a simple AWS Lambda function that listens for Amazon ECS task events and writes them out to a CloudWatch Logs log stream.

To start with, let us set up an ECS Cluster and configure a Task Definition from scratch.

Let’s now author a new Lambda function by using our custom python code snippet so that it can listen to Amazon ECS task events and write them out to CloudWatch Logs log stream.

Finally, we will create an event rule in CloudWatch and put it to test. We can easily see all the logs delivered to our CloudWatch Logs log group after launching and stopping our task.

Let’s start with some basics and understand the components of Amazon SNS.

In this tutorial, you configure a CloudWatch Events event rule that only captures task events where the task has stopped running because one of its essential containers has terminated. The event sends only task events with a specific stoppedReasonproperty to the designated Amazon SNS topic.

Let’s get started by setting up a new ECS Cluster and creating a Task Definition using a custom JSON template.

Now, we will create an Amazon SNS Topic and create a subscription.

In this last part of our demo, we will create a an event rule in CloudWatch using a custom event pattern and put it to test.

Amazon ECS enables you to inject sensitive data into your containers by storing your sensitive data in AWS Secrets Manager secrets and then referencing them in your container definition. Sensitive data stored in Secrets Manager secrets can be exposed to a container as environment variables or as part of the log configuration.

In the past, when you created a custom application to retrieve information from a database, you typically embedded the credentials, the secret, for accessing the database directly in the application. When the time came to rotate the credentials, you had to do more than just create new credentials. You had to invest time to update the application to use the new credentials. Then you distributed the updated application. If you had multiple applications with shared credentials and you missed updating one of them, the application failed. Because of this risk, many customers choose not to regularly rotate credentials, which effectively substitutes one risk for another.

Secrets Manager enables you to replace hardcoded credentials in your code, including passwords, with an API call to Secrets Manager to retrieve the secret programmatically. This helps ensure the secret can't be compromised by someone examining your code, because the secret no longer exists in the code. Also, you can configure Secrets Manager to automatically rotate the secret for you according to a specified schedule. This enables you to replace long-term secrets with short-term ones, significantly reducing the risk of compromise

Amazon ECS enables you to inject sensitive data into your containers by storing your sensitive data in AWS Secrets Manager secrets and then referencing them in your container definition.

The following tutorial shows how to create a Secrets Manager secret, reference the secret in an Amazon ECS task definition, and then verify it worked by querying the environment variable inside a container showing the contents of the secret.

In this tutorial we will be creating a basic secret for storing a username and password to reference later in a container

In order for Amazon ECS to retrieve the sensitive data from your Secrets Manager secret, you must have the Amazon ECS task execution role and reference it in your task definition. This allows the container agent to pull the necessary Secrets Manager resources.

Let us now create a task definition that references a Secrets Manager secret, and then create an ECS Cluster.

For this tutorial we will be running a task using the EC2 launch type, using the cluster we created in the previous step. Also, we can verify whether all of the steps were completed successfully and the environment variable was created properly in our container