DevSecOps on AWS: Defend Against LLM Scrapers & Bot Traffic

Explore how to run a Flask demo app in a local environment using docker compose, a docker file, and a minimal setup with Python, virtual environments, and security practices.

Navigate Flask app code in the demo folder, including init file, routes, and test HTML template, and explain rendering templates and the WAF integration URL used for AWS deployment.

Install Terraform and manage versions with tfenv, selecting 1.6.x (e.g., 1.6.6); switch, list, and uninstall versions on Linux or macOS, with Windows caveats.

Configure an AWS profile and Terraform environment to spin up a coding lab, deploy a test Flask app, and set Terraform environment variables while verifying access with AWS get-caller-identity.

Build a docker image and push it to AWS ECR using provided scripts, configure the ECR repository, and verify the image in the AWS console for future Terraform deployment.

Run terraform scripts step by step with diagrams and a quick code-structure overview as we deploy a Flask demo application on AWS, highlighting CloudFront WAF integration.

Pre init and apply terraform modules to provision a vpc with public subnets, route tables, and security groups; configure terraform state on s3, logs on dynamodb, and sns alerts.

Configure domain delegation via Route 53 by creating a hosted zone, updating NS records, and validating SSL certificates with the ACM Terraform module for ELB and CloudFront.

Deploy an application load balancer with Terraform, configuring remote state, VPC, security groups, and certificates, while preserving ALB logs in S3 and redirecting HTTP to HTTPS.

Clarify that some variable name prefixes differ due to another security course, but the concepts and coding practices remain relevant for DevSecOps on AWS lessons.

Deploy a dockerized Flask app on EC2 via an auto scaling group, using a launch template and user data to pull from ECR, enable CloudWatch logs, and follow security best practices.

Configure an aws alb target group for ec2; define vpc id, health checks, and host header routing to a dockerized flask app in an auto scaling group.

Explore deploying a flask app on AWS EC2 via the AWS console, configure auto scaling with a load balancer, and SSH into the instance to run Docker with port mappings.

Explore how Amazon CloudFront functions as a content delivery network, detailing distributions, behaviors, origins, edge locations, and policies such as cache, origin request, and response headers.

Explore building a CloudFront-based delivery setup with Terraform, integrating Route53, ALB, and S3, plus modular policies, maintenance pages, and logs.

Explore deploying a CloudFront backed AWS WAF with a Terraform module, configuring WAF ACLs and logging pipelines via S3, Kinesis Firehose, and a Lambda processor for Athena-ready logs.

Deploy CloudFront with Terraform, configure Route53 DNS, and integrate WAF in the AWS console. Implement maintenance mode that serves a 503 page from an S3 bucket and routes traffic accordingly.

Explore how auto scaling on AWS Fargate and Kubernetes handles traffic spikes, reveal inertia and cooldown delays, and why reactive scaling falls short against high-velocity AI bot traffic.

Analyze aws fargate autoscaling in response to ai bot spikes using alb request count alarms and 100-request per container scaling, plus edge defenses like waf and cloudfront caching.

Cache content at edge locations to accelerate delivery. Define cache policies with time-to-live (ttl) and keys based on headers, cookies, and query strings, using CloudFront functions and Lambda@Edge.

Define a degraded content strategy that routes bots and non-human traffic to lighter versions of your site, using cache or simpler infrastructure, while keeping the main app for real users.

Show how cloudfront and lambda differentiate real users from bots via user-agent, forwarding bots to pre-rendered s3 content while serving the full app to humans, optimizing cost and performance.

Route bot traffic with Lambda@Edge to a secondary CloudFront distribution backed by S3. Use Terraform and Python to detect bot user agents and rewrite to index.html on the secondary origin.

Route bots with a lambda@edge in CloudFront to forward traffic to an S3 bot origin using a cache policy and test x test header.

Split caching between main and secondary distributions to avoid degraded content; use a CloudFront viewer-request function to add a bot header to the cache key.

Enable cloudfront edge tagging of bot requests with a lightweight javascript function, normalize user agents, and add a custom header to the cache key, creating separate bot and human caches.

Explore how the missing assets issue arises when deploying updates and how CloudFront caches HTML while avoiding assets. See why versioned filenames or hashes keep assets in sync after deployment.

Deploy the application and invalidate CloudFront caches to observe asset versioning and cache behavior, including how per pop caching can cause missing assets and 404s.

Discover how CloudFront origin shield creates a centralized caching layer to reduce origin requests and cache misses, boosting asset caching efficiency, and learn to enable it in Terraform.

Enable immutable asset deployments by versioning and storing assets in S3, ensuring old assets remain accessible and preventing 404s during builds by avoiding overwrites.

Learn how to defend against llm scrapers and bot traffic by delivering self-contained content with inline css and js, avoiding external assets and requests.

Explain how degraded content splits traffic between humans and bots using full and pre-rendered versions. Highlight cache separation with CloudFront tagging and immutable assets to prevent missing assets after deployments.

Understand how a web application firewall protects apps from cross-site scripting, sql injection and bot traffic using AWS WAF managed and custom rules, deployed as a reverse proxy on CloudFront.

Use WAF blacklists and allowlists with IP sets to control AI bot traffic. Learn why blacklists are safer than whitelists and how to implement IP-based rules.

Block select countries with a WAF geo-country rule and whitelist bot networks via IP sets, using AWS Web ACL with CloudFront and Athena to analyze WAF logs.

Configure athena to query aws waf logs in s3, create a table, and extract real bot geo data including country, ip, and user agent from waf logs with live examples.

Use ga4 fingerprinting to power rate limiting by key in a five-minute window, not ip. Forward ga4 from cloudfront, configure waf rules with terraform, and tune thresholds using real traffic.

Apply JA4-based rate limiting and first-stage filtering using 95th percentile analysis of fingerprint bursts. Use top fingerprints and bot-header scoped queries to set practical GA4 thresholds and investigate spikes.

Learn to implement URL-scoped throttling with AWS WAF granular rate rules, using Terraform to define per-URL and OR-based conditions, block aggressive scrapers on category pages and API endpoints.

Apply AWS managed rules to act as a bot filter layer, leveraging the Amazon IP reputation list to block botnet, compromised, and dirty traffic while allowing selective subrule control.

Explore AWS WAF bot control foundations, distinguishing common and targeted bot detection modes, with focus on identifying clients, bot types, and applying scalable protection for sensitive endpoints.

Turn on bot control in common mode to safely test traffic, collect metrics, and apply category labels in the web ACL before blocking.

Prepare the app layer for targeted bot control by integrating AWS WAF tokens via the SDK, enabling real-user validation and token-based API protection.

Explore how AWS bot control dashboards with label based insights help defend against bot traffic and LLM scrapers, using real traffic and test requests to classify bots versus humans.

Identify AWS WAF bot control categories using Athena queries on real logs, extract and aggregate user agent data, and produce a practical bot identification report for policy decisions.

Define a three-tier bot policy with whitelist and blacklist. Allow Googlebot and beam bot, block others, route bots to degraded S3 content, while trusted bots get full pages.

Switch from observability to enforcement for aws managed rules, blocking known bots using a two ip-set approach and a user-agent regex, with test curl and a Waveboard updater.

Build a multi-layered bot defense on AWS using CloudFront, WAF, and machine learning detections to classify, block, or route bot traffic, informed by Athena logs.