Chaos Engineering

Discover why chaos engineering shifts from disaster recovery to proactive resilience by injecting faults in a controlled environment to test, strengthen distributed systems.

Explore how chaos engineering runs experiments to reveal real world system behavior under failure, detailing five steps from steady state (the given state) to hypothesis, small experiments, and measuring impact.

Explore aws fault injection service (fis) as a managed chaos engineering tool for fault injections, with experiments for ec2, s3, eks, rds, and az outages, integrated with cloudwatch and iam.

Explore fault injection experiments, including EC2 terminations behind auto scaling, disk fills, Aurora reader node reboots, pod deletions, high I/O on ECS and Fargate, availability zone interruptions, observing application impact.

Explore how the AWS fire security model uses two IAM roles—console access and the fire service role—to assign permissions for experiments and resource actions like EC2 termination and RDS failover.

Create the FIS service role to run chaos experiments, granting EC2 termination and availability zone changes, and attach CloudWatch permissions for monitoring and logging.

Break down the first experiment: given architecture across availability zones fronted by an auto scaling group, and the hypothesis that terminating an EC2 instance won’t affect the app.

Explore chaos engineering by deploying a fault injection setup in AWS console, with an EC2 launch template, an Auto Scaling group across two availability zones, and a CloudWatch log group.

Build an ASG based architecture by creating an EC2 launch template with experiment ready tag and configure a two-AZ auto scaling group with capacity 1–4 and a CloudWatch log group.

Create a FIS experiment template in the fault injection service to terminate 50% of running EC2 instances in the auto scaling group, using tag and state filters for precise targeting.

Chaos engineering demo runs an FIS experiment to terminate a targeted EC2 instance, previews the target, and fixes a misconfigured auto scaling group by scaling to two instances.

Increase the auto scaling group to two and minimum capacity to two, terminate an instance, and spin up a new one, with the app running and no impact.

Use a CloudWatch dashboard to track healthy versus unhealthy hosts in an auto scaling group during fizz experiments; EC2 termination drops to one, then minimum desired capacity restores to two.

Demo shows deploying a fault injection framework using a CloudFormation template to create an FIS experiment that terminates an EC2 instance and spins up another, with a preview step.

Explore a real life pet adoption application built as microservices on AWS, featuring EKS, EC2, Fargate, DynamoDB, S3, and Aurora with load balancers and auto scaling across zones.

Deploy the pet site, log in, and review adoptions; update cube config for cube CTL, monitor with CloudWatch trace map, and run ten minutes of traffic to verify steady state.

learn how to preconfigure an ec2-based environment for chaos engineering deployment by creating an iam role, setting up the instance, installing prerequisites, and deploying the application with cdk bootstrap.

Chaos engineering demo guides you through setting up architecture, deploying the pet site application, and generating real-world load with ECS Fargate, K6, and chromium to measure CloudWatch real user metrics.

Plan your fault injection experiment, or game day, by defining objectives and choosing a dev or test workload. Align with the AWS well-architected reliability pillar; define steady-state behavior and metrics.

Formulate a hypothesis from your architecture and fault. Define the experiment with blast radius tags, duration, stop conditions, and metrics, then run in development before production and perform blameless post-mortem.

Establish steady state metrics with CloudWatch, RUM, and X-Ray trace maps by monitoring page loads, load time, and ECS performance. Use baselines to assess fault injection impacts.

Demonstrates CloudFormation deployment to provision an EC2 environment with IAM role, VPC, and security group, then bootstrap the CDK, install dependencies, and deploy serverless and observability stacks.

Simulate an EC2 disk failure to test your application’s resilience across availability zones, a load balancer, and an auto scaling group, validating that the web page stays available.

Run a disk fill fault injection on an EC2 instance using AWS FIS and Systems Manager, monitoring CPU and network metrics before and after the experiment in chaos engineering.

Run the fizz experiment and verify the site stays responsive; monitor cpu and network metrics showing a failing disk EC2 instance drops activity while a healthy EC2 instance handles traffic.

Explore chaos engineering by rebooting the reader node in an Aurora PostgreSQL multi-az cluster, observing no user impact as traffic shifts to the writer and the reader recovers.

Assess the current state of the Aurora PostgreSQL database, including writer and reader instances, latency, and request rate using CloudWatch traces; then create an IAM role for Fault Injection Simulator.

Demonstrates creating a Fizz Experiment template to reboot an RDS Aurora reader node and verify the reboot has minimal impact, with writer operations continuing.

Test the resilience of an ECS Fargate-based pay-for-adoption API by injecting I/O stress with AWS Fault Injection Service, verifying continued availability of the pet payment and adoption web app.

We examine steady state metrics from CloudWatch Container Insights for ECS service on Fargate, tracking CPU, memory, and network over 30 minutes, then recheck after introducing an I/O stress fault.

Create an IAM role for ECS Fargate via the GUI, name it ECS Fargate Firestore, and attach CloudWatch Logs, SSM, and Fault Injection Simulator access for AFI experiments.

Create an AWS FIS experiment template to run an ECS Fargate I/O stress test for ten minutes on selected ECS tasks, with CloudWatch logs.

In this demo, run io stress tasks to generate traffic and monitor user metrics. Results show no major impact, with cpu and memory utilization dropping due to the application's design.

Chaos engineering uses fault injection to simulate memory utilization in an eks cluster for the pet adoption microservices app. Verify that redundancy across pods maintains availability under memory stress.

Set up prerequisites for memory stress testing on EKS by creating and mapping an IAM role for the X fault injection simulator, configuring kubectl, and deploying Kubernetes objects.

Establish the steady state of an Eks application by collecting baseline cpu and memory metrics from CloudWatch container insights, dashboards, and rum, before the memory-stress demo on part 2.

Create a Fizz experiment to apply memory stress to a pod in the ECS cluster, using an experiment template for nine minutes at 50%, and monitor with CloudWatch logs.

Chaos engineering demo shows memory stress on an ecs pod, raising latency from about 90 ms to 120 ms with a small drop in requests, while end-user impact stays limited.

simulate pod delete on eks to practice chaos engineering, validating kubernetes restarts and ensuring pod termination triggers a prompt restart with minimal user impact.

Monitor steady state with cloudwatch container insights, tracking pod count, cpu, and memory before deleting a pod on eks. Evaluate end-user experience with cloudwatch rum and web vitals during load.

Demonstrates running a pod delete experiment on EKS using the fault injection service, leveraging a template from the scenario library and validating with CloudWatch logs and metrics.

During the pod delete demo, kubectl get pods shows a new pod for the pet site after deletion, with two running pods and no error spike in CloudWatch.

Explore how availability zones provide isolated, redundant data centers within an aws region, enabling fault tolerance and failover across zones or across regions through cross-region replication.

Conduct cross-service failure tests across multiple availability zones by simulating a power interruption; verify the pet site remains available with degradation, using tags to target impairment.

Create an IAM role with a trust policy for AFI's to assume it, attach the easy experiment and office AC task policies to simulate an AZ power interruption.

Demonstrate chaos engineering by simulating an az power interruption for a us east 1a aurora writer using a prefilled scenario library, shared parameters, and CloudWatch logging.

Explore chaos engineering by executing a controlled easy power interruption to observe steady state performance, RDS failover, and resilience across writer and reader nodes, with real-time metrics.

Empty seven s3 buckets and permanently delete their contents, delete rds snapshots for cloudformation deletion, then destroy cdk resources, auto scaling group, launch template, cloudwatch log group, and ec2 servers.

Apply chaos engineering with the AWS fault injection service to advance the resilience lifecycle for always-on systems and distributed apps.