Software Architecture & Technology of Large-Scale Systems

Identify how performance problems arise from queue buildup in network, db, or cpu resources, and design systems to prevent bottlenecks by avoiding inefficient processing, serial access, and limited capacity.

Understand how performance problems arise from request queue buildup and apply the three principles: efficiency, concurrency, and capacity to optimize serial and concurrent requests.

Contrast internet and intranet latency, noting internet paths add more hops and less reliability while intranets are faster; identify latency sources as wire transfer, TCP connection creation, and TLS/SSL handshakes.

Minimize memory access latency by reducing memory bloat, optimizing heap usage, choosing proper garbage collectors for batch versus live processes, and applying normalization and compute over storage.

Explains how cpu latency arises from context switching between processes on a single cpu, showing how inevitable i/o events cause evictions and wasted time, and explores minimization strategies.

Explore how shared resource contention drives latency in large-scale systems, and examine queues, threads, connection pools, disks, and locks as the main bottlenecks to reduce.

Compare optimistic locking to pessimistic locking for handling shared resource contention, showing how fetches occur without locks, followed by verification, back-off retries, and eventual commits.

Explore the compare and swap mechanism as an optimistic locking technique supported by hardware and software, enabling non-exclusive updates via atomic operations in Java and NoSQL databases.

Explore deadlocks in concurrent systems: ordering related deadlocks resolved by a global order, and load related deadlocks from high request or database connection pressure, with prevention strategies.

Explore caching to reduce latency by serving requests from cache instead of executing them, improving efficiency for single and concurrent requests.

Explore caching's role in system architecture to boost performance, including persistent connections, response compression, and efficient encoding, plus thread pools, DB pools, and session caching.

Explore a typical web system with a web browser as its client, a business application, a service, and a database, and learn concepts to scale it to millions of users.

Explore scalability principles for large-scale systems, focusing on decentralization and independence to enable horizontal scaling, avoid monoliths, and minimize coordination bottlenecks among many specialized workers.

Explore replication for horizontal scalability by multiplying web app, service, and database instances, distributing read loads and managing stateless versus stateful replication across layers.

Leverage stateless replication in the web layer by using a shared cache like Memcached or Redis to store session data, boosting scalability and reducing latency.

Stateless replication in the service layer mirrors web layer replication, using a shared cache to reduce latency and a database lock table with row locks to coordinate concurrent writes.

Break the monolith into independent services with a gateway to simplify client interactions. Use rest or soap for external clients and internal protocols like gRPC, while managing a shared database.

Use message queues to decouple interfaces, reliably deliver writes, and offload database load, enabling asynchronous processing and scalable order processing with inventory checks and notifications.

Learn how asynchronous processing boosts scalability by decoupling the order service from the database with a message queue, creating a backlog that drains later.

Explore database partitioning to achieve extreme scalability, from vertical designs to horizontal range and hash partitioning, including node distribution, NoSQL vs RDBMS tradeoffs, and the impact on asset transactions.

Range partitioning suits range-based queries, distributing data by value ranges across nodes. Hash partitioning targets key-based fetches, delivering faster equality lookups but costly range queries in distributed databases.

Learn methods for horizontal scalability, including breaking a monolith into scalable services and choosing stateful versus stateless replication. Explore partitioning, asynchronous processing, caching, and database partitioning as needed.

Discover how external load balancers are found via DNS resolving to an external IP, while internal load balancers are discovered through IP addresses, registries, or local configuration.

Layer-7 load balancers act as reverse proxies, terminate ssl, and route requests by content to static content servers or dynamic web applications, while applying round-robin or least-connections load balancing.

Implement active-active global data replication with master-master, peer-to-peer replication across regions, using synchronous within a region and asynchronous cross-region writes, plus conflict resolution and DNS failover.

Explore the rationale for micro-services in large-scale systems, examining why to adopt micro-services and why not to adopt them.

Explains service oriented architecture, its benefits, and how microservices improve it through decentralization and independent scaling. Highlights dependencies from common interface and database schemas and the resulting bottleneck.

Explore how microservices coordinate an order booking across inventory, order, and shipment services using local acid transactions and the saga pattern for compensating transactions.

Learn how micro-services communicate through synchronous and asynchronous models, and why an event-based transaction approach enhances scalability, reliability, and availability in systems.

Define reliability as a system property that ensures operation and availability despite partial faults. Use an airplane example where a remaining engine keeps the flight, illustrating reliability during time interval.

Define availability as the probability a system works correctly and is ready. Minimize downtime, recover quickly, and measure uptime over total time or successful requests.

Understand redundancy as spare capacity in a three-tier system, and see how primary components, web, business, and database, are backed by secondary instances to maintain availability.

Explore three redundancy types: active (hot spare), passive (warm spare), and cold (backup), and how each affects availability, load distribution, and response time in large-scale systems.

Identify and classify faults in client-server systems by examining response failures, timeout failures, incorrect responses, crash failures, and the limited scope of arbitrary response failures.

Explore health checks for multi-instance services, including external ping monitoring and internal cluster heartbeat monitoring. See how heartbeat exchanges reveal unhealthy nodes and help determine appropriate responses.

Explain how internal cluster monitoring uses heartbeats between primary and secondary load balancers to detect failures and enable failover without external monitoring.

Examine stateful failover strategies, including virtual IP and registry-based approaches, with floating IPs, heartbeat monitoring, and DNS TTL or router-based routing to promote standby when the primary fails.

Examine hot standby in a master-slave database, using synchronous replication to prevent downtime and data loss. Understand primary and secondary roles, failover, and how network lag affects latency.

Explore how to design highly available and reliable systems with redundancy, stateless versus stateful components, automated fault detection and failover, and resilience patterns like timeouts, retries, circuit breakers, shed load.

Explore network security as the fundamental part of security for any software system and its role in protecting software.

Explains how ssl/tls secures client-server communication over https by exchanging a server public key, then establishing a symmetric key for fast encrypted messaging, using certificates for verification.

Learn how SSL and TLS secure TCP communication, enable HTTPS and app protocols like HTTP, POP, IMAP, LDAP, FTP over SSL, and how certificates and public keys prove server identity.

Understand how a chain of trust forms from root ca to tier 2, tier 3, and tier 4 end-entity certificates, and how browsers verify signatures with public keys during ssl/tls.

Configure firewalls to protect a four-subnet network, exposing external access to demilitarized zones only on port 443, and opening port 8080 for internal services while db uses port 1522.

Explore stateful and stateless authentication, how client credentials transfer and are verified, and server-side and client-side storage options for credentials and tokens.

Stateless authentication issues tokens from an authorization service, containing user details, signed for verification by a private/public key pair, enabling decentralized, scalable access via cookies and authorization headers.

Learn access management and authorization, modeling access to your system and implementing controls to regulate who can access resources.

explains role-based access control by organizing users into groups, assigning business roles with permissions to access catalog, inventory, and order services, while system resources rely on service credentials.

Explore how OAuth2 tokens are granted by walking through client, resource owner, authorization server, and resource server roles, and see how an authorization code exchanges for a token.

Explore OAuth2's two grants: authorization code flow for internet apps and password flow for intranet apps. The authorization code flow uses redirects to exchange a short-lived code for access tokens.

Secure data at rest by hashing passwords and using transparent data encryption with a data encryption key; optionally add client data encryption for sensitive columns with a key storage system.

Secure a large-scale system by terminating tls at load balancers for https and using certificates for intranet tls, while applying stateless jwt authentication with oauth2 and rbac for access control.

Explore how SQL injection exploits unsafe queries via category parameters, and how prepared statements ensure only values substitute into queries, preventing data leakage and table drops.

Explore deployment challenges in large-scale systems, including replication and partitioning of databases, queues, caches, and services. Learn automated, continuous deployment strategies to manage complexity across RDBMS and NoSQL.

Explore modern ways of deploying an application and compare deployment approaches for learning purposes today.

Compare container deployment with virtual machines, highlighting lightweight images and no guest OS. See how a container runtime translates system calls to the host, enabling fast start and image movement.

Explore how to make infrastructure deployment simpler for large-scale systems, focusing on practical techniques that streamline deployment processes within modern software architectures.

Learn how to assign names and IP addresses to application component instances, and resolve them with DNS and internal DNS, enabling load balancers and clients to locate services.

Kubernetes, an open-source and popular deployment system, provides service naming via internal DNS, health checks, automatic rollouts and rollbacks, container restart and replacement, load balancing, resource limits, and storage mounting.

Explore canary deployment as a production testing strategy that releases a feature to a small subset of nodes, monitors impact with limited user requests, and gradually rolls out if safe.

Review deployment of large-scale systems, provisioning infrastructure, deploying applications with containers and cloud, and managing Kubernetes operations, while applying upgrade strategies: recreate, rolling update, canary, blue-green, and A/B testing.

Explore architectures for static and dynamic content, including Apache and Nginx servers. Compare Jetty and Tomcat, Node.js, and Spring Boot, with caching, Varnish, and CDN distribution via reverse proxies.

Explore how Nginx acts as a reverse proxy and cache, fronting Apache servers, caching static resources, and load balancing requests across multiple backends in large-scale web architectures.

Explore how Node.js brings JavaScript to the server, handling asynchronous IO with a single-threaded event-driven model to support many IO-bound connections, using JSON across browser, server, and databases.

Explore building services with restful and microservices patterns, focusing on caching and asynchronous processing through messaging queues, and hosting on containers with scalable load balancing via Engine X.

Explore how to design scalable services using web containers, object caching with memcache and Redis, and asynchronous messaging with Redis, RabbitMQ, and Kafka, plus deployment via Kubernetes.

Memcached serves as a centralized key-value cache across nodes, storing objects as blobs with TTL and using expired data first, then LRU eviction to maximize hits for frequently accessed data.

Memcache operates as a cluster of nodes with a client library using DNS to route keys by hash, following a cache-aside pattern with sub-millisecond latency and horizontal scalability.

Explore cloud caching solutions that mirror on-premises options for Memcached and Redis, including infrastructure as a service and fully managed services like AWS Elastic Cache and Google Memorystore.

Explore Rabbit MQ architecture as a general purpose message broker with vertical scalability. Learn about persistent and transient messaging, exchange routing, and master-slave replication for high availability.

Redis pub/sub provides a fast, non-persistent in-memory messaging queue with publishers, subscribers, and channels, delivering messages to connected clients in real time for use cases like live leaderboards.

Explore how RDBMS scale through vertical upgrades, vertical partitioning across services, and read replicas, then compare with NoSQL solutions to address consistency and scalability limits.

Examine DynamoDB's peer-to-peer architecture with no master, using consistent hashing and gossip protocols for extreme high availability and scalable reads and writes of small data chunks.

Google Big Table, a NoSQL store akin to Apache HBase, uses column families, row keys, and timestamps to provide sparse, sorted, multi-version data with atomic reads and writes.

MongoDB stores nested JSON-like documents and is horizontally scalable and eventually consistent. It updates fields, indexes top-level and nested fields, and enables fast queries by ID.

Explore MongoDB architecture, including indexing for fast searches, the write overhead trade-off, horizontal sharding, and master-slave replication, with Jason-based document storage for node.js applications.

Explore platform products for building analytics layer, move log data with Logstash and Fluentd, store in Hadoop HDFS or Elastic Search, and review stream processing with Kafka as a buffer.

Discover stream processing for real-time analytics on streaming data, using technologies like Kafka, Storm, Flink, Spark Streaming, and discuss fault tolerance, latency, throughput, and handling out-of-order or missing data.

Explore the architecture of large-scale systems, covering front-end layers, caching with Redis or Memcached, content delivery networks, service options, databases, and analytics tools as discussed in the lecture.