Practical System Design: Build Real Systems in AWS

Design and demonstrate a URL shortener workflow that creates shortened links from long URLs, supports custom aliases, and redirects users to the original destinations.

Set up a foundational MySQL database on AWS with a security group and RDS instance. Create the urls table and insert sample data to validate the URL shortener schema.

Design and test the read service for the URL shortener, querying the database by short key to return the long URL, and deploy a Lambda on AWS with local tests.

Create a redis-backed url shortener with a custom aliases set for duplicates and a global counter to generate unique short keys, deployed on AWS Lambda and tested end-to-end.

Implement Redis caching for the read service in a URL shortener by adding a Redis hash set for short key to long URL mappings, reducing database load and speeding redirects.

Expose your back-end services through API gateway with get and post routes, connect read and write lambdas, enable http redirects, and test with basic logging.

Migrate the database layer to Aurora MySQL by separating read and write traffic with writer and reader endpoints, then clean data, provision a cluster, and update connections for end-to-end test.

Configure a change data capture pipeline for Aurora MySQL by creating a parameter group, enabling bin logs, setting retention, and creating DMS source endpoint with a replication task to Kinesis.

Review the monthly cost of the url shortener on AWS, identify cost drivers—Aurora, DMS, and VPC—and clean up the related resources.

Explore the Ticketmaster two-phase workflow, from homepage to booking, where users discover events, view tickets, and reserve to hold for ten minutes. Confirm payment via Stripe to finalize.

Builds a read-only event discovery service behind AWS Lambda that exposes three get endpoints for home, event, and ticket data via JSON from MySQL.

Learn to implement a two-phase checkout: phase one reserves tickets with a Redis lock (NX, TTL 600s) to prevent race conditions, and phase two confirms bookings.

Turn existing services into a live browser app with an api gateway, returning html pages for home, event, and ticket views and enabling reserve and confirm flows via lambda endpoints.

Build a fully dynamic seat map for high traffic events by wiring in a high traffic handler and a web socket API layer, enabling real-time updates to connected clients.

Deploy a real-time seat map system on AWS EC2 using a bootstrap script and systemd. Update the high-traffic UI via WebSocket to reflect reserved, confirmed, and available seats.

Implement a ticketmaster search with an OpenSearch cluster and a dedicated search service lambda. Seed the event index with data and power match phrase queries on event names via /search.

Explore an event-driven e-commerce workflow using a real Kafka cluster, from producers and consumers to topics, including the order, payment, and notification services, with code in the GitHub repository.

Set up a MySQL database for e-commerce microservices with Terraform, provisioning an RDS instance and security group, bootstrapping the schema, and validating the deployment with a database endpoint.

Spin up a local Kafka cluster with Docker Compose, create order and payment topics, and examine K-raft versus Zookeeper-based architectures.

Build the order service for an e-commerce app, create a MySQL order record, and publish an event to Kafka via a producer, with end-to-end verification.

Extend the e-commerce app with a payment service that consumes order events from the order topic and emits a payment event to the payment topic.

Launch and operate the notification service as a Kafka consumer for the payment topic, sending user notifications on payment success and validating the end-to-end flow.

Deploy a full stack to AWS by dockerizing the e-commerce app, pushing the image to Docker Hub, and provisioning resources with Terraform, including Kafka, MySQL, and EC2 instances.

Deploy a full-stack AWS setup with Terraform and Docker, initialize and apply, verify MySQL, Kafka topics and consumer groups, perform end-to-end testing of orders with single-partition ordering, then tear down.

Update the Kafka-esque broker to support a producer API with per-partition locks, log end offsets, and deterministic partitioning by key, returning topic, partition, and offset.

Build the kafka-esque consumer API and migrate the ecommerce app to consume order placed events, trigger payment success processing, and notify users via topics with offset commits.

Implement and test the Kafka-esque consumer API in an eCommerce app, wiring payment and notification services to order and payment topics, and validating offsets, topic creation, and end-to-end messaging.

Increase partitions per topic to two for order, payment, and internal consumer offsets topic to enable parallel processing while preserving per-partition ordering through key-based routing and single-consumer partition handling.

Increase consumers per group to two to enable parallel processing across two partitions, with independent handling of order, payment, and notification streams, plus heartbeat and rebalance mechanics.

Introduce replication in kafka-esque cluster with broker a as leader and broker b as follower; implement replication loop and replica fetch endpoint to keep partitions in sync and enable failover.

Increase replication factor to two for data and offsets, launch brokers, produce orders, and verify that replication cycles align logs and offsets across brokers.

Strengthen replication guarantees by implementing in-sync replicas and high watermarks, enforcing minimum in-sync replicas on writes, and guiding reads to the high watermark across data topics.

Explore how in-sync replicas and high watermarks govern writes and reads across two brokers with a fixed leader, including min ISR checks and replication progression.

Introduce zookeeper into a kafka-esque cluster and implement controller election using ephemeral sequential z-nodes. Observe how the lowest sequence number becomes controller and how re-election occurs on node failure.

Demonstrate min ISR 1 and min ISR 2 scenarios in a Kafka-esque setup, testing proxying, failover, and coordinated leadership across broker A and broker B.

Deploy a full kafka-esque stack to AWS with Terraform and Docker, including RDS MySQL, Zookeeper, and two brokers on EC2, plus a dockerized e-commerce app.