Software Architecture & Technology of Large-Scale Systems

Identify how performance problems arise from queue buildup in network, db, or cpu resources, and design systems to prevent bottlenecks by avoiding inefficient processing, serial access, and limited capacity.

Understand how performance problems arise from request queue buildup and apply the three principles: efficiency, concurrency, and capacity to optimize serial and concurrent requests.

Learn the four essential performance metrics: latency, throughput, errors, and resource saturation, and how tail latency and 99 percentile measurements guide capacity planning in large-scale systems.

Explore serial request latency as a key driver of system performance, distinguishing it from parallel latency, and begin studying efficiency in this context.

Contrast internet and intranet latency, noting internet paths add more hops and less reliability while intranets are faster; identify latency sources as wire transfer, TCP connection creation, and TLS/SSL handshakes.

Minimize memory access latency by reducing memory bloat, optimizing heap usage, choosing proper garbage collectors for batch versus live processes, and applying normalization and compute over storage.

Explains how cpu latency arises from context switching between processes on a single cpu, showing how inevitable i/o events cause evictions and wasted time, and explores minimization strategies.

Improve CPU latency by reducing contact switching through batch and async IO, using a single-threaded model for high-IO loads, and tuning thread pools and virtual environments to prevent CPU hogging.

Analyze latency costs across CPU, memory, disk, and network with concrete numbers, from 0.5 to 7 nanoseconds for CPU to about 100 nanoseconds for memory and 10 milliseconds for disk.

Explore how shared resource contention drives latency in large-scale systems, and examine queues, threads, connection pools, disks, and locks as the main bottlenecks to reduce.

Analyze strategies to minimize contention on a single machine by optimizing CPU, memory, disk, and network utilization, tuning thread and connection pools, and managing listen queues before vertical scaling.

Reduce lock contention by shortening the lock duration and applying lock splitting or lock striping. Explore read-write locks, optimistic versus pessimistic locking, and compare-and-swap as alternatives.

In a high-contention e-commerce scenario, pessimistic locking fetches and logs shared records, acquires a lock for the transaction, updates inventory, and releases it on commit.

Compare optimistic locking to pessimistic locking for handling shared resource contention, showing how fetches occur without locks, followed by verification, back-off retries, and eventual commits.

Explore the compare and swap mechanism as an optimistic locking technique supported by hardware and software, enabling non-exclusive updates via atomic operations in Java and NoSQL databases.

Explore deadlocks in concurrent systems: ordering related deadlocks resolved by a global order, and load related deadlocks from high request or database connection pressure, with prevention strategies.

Understand how coherence delays arise from shared data in multi-threaded systems, and compare synchronized and volatile approaches to achieve visibility while weighing main memory and cache costs.

Explore caching to reduce latency by serving requests from cache instead of executing them, improving efficiency for single and concurrent requests.

Explore caching's role in system architecture to boost performance, including persistent connections, response compression, and efficient encoding, plus thread pools, DB pools, and session caching.

Explore how caching boosts performance across a large-scale system, from backend service layer object cache to web app session cache, CDN, and browser cache, while distinguishing dynamic versus static data.

Explore dynamic data caching for services and web apps, comparing exclusive caches with local per-node storage and shared caches with external solutions like Memcached or Redis.

Learn how cache hit ratio measures effectiveness, and how limited space and stale data shape strategies to cache read-only, frequently accessed, small objects while using invalidation or TTL.

Explore strategies for scaling large systems, including vertical and horizontal scalability, replication, caching, and partitioning. Delve into load balancing, microservices, saga pattern, and NoSQL databases for high scalability.

Differentiate performance from scalability by defining performance as low latency and high throughput under a fixed load, and scalability as increasing throughput under variable load via horizontal scaling.

Explore a typical web system with a web browser as its client, a business application, a service, and a database, and learn concepts to scale it to millions of users.

Explore scalability principles for large-scale systems, focusing on decentralization and independence to enable horizontal scaling, avoid monoliths, and minimize coordination bottlenecks among many specialized workers.

Explore replication for horizontal scalability by multiplying web app, service, and database instances, distributing read loads and managing stateless versus stateful replication across layers.

Leverage stateless replication in the web layer by using a shared cache like Memcached or Redis to store session data, boosting scalability and reducing latency.

Stateless replication in the service layer mirrors web layer replication, using a shared cache to reduce latency and a database lock table with row locks to coordinate concurrent writes.

Scale your relational database management system (RDBMS) by using replication to create a master with read replicas to handle read traffic and a backup for high availability.

Scale the system by breaking the monolith into specialized services for modules like user, catalog, order, inventory, and notification. It reduces resource waste and enables module-specific scaling.

Break the monolith into independent services with a gateway to simplify client interactions. Use rest or soap for external clients and internal protocols like gRPC, while managing a shared database.

Use message queues to decouple interfaces, reliably deliver writes, and offload database load, enabling asynchronous processing and scalable order processing with inventory checks and notifications.

Learn how asynchronous processing boosts scalability by decoupling the order service from the database with a message queue, creating a backlog that drains later.

Explore database partitioning to achieve extreme scalability, from vertical designs to horizontal range and hash partitioning, including node distribution, NoSQL vs RDBMS tradeoffs, and the impact on asset transactions.

Range partitioning suits range-based queries, distributing data by value ranges across nodes. Hash partitioning targets key-based fetches, delivering faster equality lookups but costly range queries in distributed databases.

Explore routing in horizontally partitioned databases. Compare three data access methods: cluster-aware client libraries that hash data to a node, a router that directs queries, and any-node forwarding.

Learn methods for horizontal scalability, including breaking a monolith into scalable services and choosing stateful versus stateless replication. Explore partitioning, asynchronous processing, caching, and database partitioning as needed.

Discover how load balancers give a single IP for multi-instance components, route requests to catalog services via round robin or random strategies, and simplify client access.

Discover how external load balancers are found via DNS resolving to an external IP, while internal load balancers are discovered through IP addresses, registries, or local configuration.

Layer-7 load balancers act as reverse proxies, terminate ssl, and route requests by content to static content servers or dynamic web applications, while applying round-robin or least-connections load balancing.

Learn how DNS uses A records and multiple IP addresses for load balancing, with health checks and TTL-driven caching, while noting DNS cannot replace active load balancers.

Implement active-active global data replication with master-master, peer-to-peer replication across regions, using synchronous within a region and asynchronous cross-region writes, plus conflict resolution and DNS failover.

Automates scaling of applications through an autoscaler that responds to load, using monitoring and health checks to launch or deregister new instances behind a load balancer, reducing human error.

Explore the rationale for micro-services in large-scale systems, examining why to adopt micro-services and why not to adopt them.

Explains service oriented architecture, its benefits, and how microservices improve it through decentralization and independent scaling. Highlights dependencies from common interface and database schemas and the resulting bottleneck.

Explore how microservices coordinate an order booking across inventory, order, and shipment services using local acid transactions and the saga pattern for compensating transactions.

Learn how micro-services communicate through synchronous and asynchronous models, and why an event-based transaction approach enhances scalability, reliability, and availability in systems.

Explore event based transactions in microservices, using an order orchestrator to implement compensating transaction steps and coordinate create order, order created, reserve inventory, and shipment events.

Scale across nodes by leveraging compensating transactions for eventual consistency, and enable asset transactions in NoSQL through an aggregate schema, then distribute load with horizontal partitioning and Kafka.

Explore failures in large-scale distributed systems as independent, partial events and their impact on availability, and learn architecture principles to boost reliability.

Analyze how partial failures can affect networks, machines, software, and data centers in distributed systems, and compare prevention and recovery strategies, highlighting recoverability and redundancy as key design goals.

Learn reliability engineering to safeguard distributed systems from partial failures by mastering reliability, availability, and fault tolerance, and architect these properties into your software.

Define reliability as a system property that ensures operation and availability despite partial faults. Use an airplane example where a remaining engine keeps the flight, illustrating reliability during time interval.

Define availability as the probability a system works correctly and is ready. Minimize downtime, recover quickly, and measure uptime over total time or successful requests.

Explore fault tolerant design by provisioning redundancy, enabling automated fault detection, and recovering from faults using the configured redundancy.

Understand redundancy as spare capacity in a three-tier system, and see how primary components, web, business, and database, are backed by secondary instances to maintain availability.

Explore three redundancy types: active (hot spare), passive (warm spare), and cold (backup), and how each affects availability, load distribution, and response time in large-scale systems.

Replicate stateless components to create redundancy and high availability, using extra replicas beyond scalability needs so load balancers enable active-active redundancy among healthy instances.

Explore infrastructural components and ensure load balancer redundancy to avoid single points of failure, using primary and secondary load balancers and a recovery process.

After provisioning redundancy, automatically detect faults to recover the system. This lecture focuses on fault detection.

Identify and classify faults in client-server systems by examining response failures, timeout failures, incorrect responses, crash failures, and the limited scope of arbitrary response failures.

Explore health checks for multi-instance services, including external ping monitoring and internal cluster heartbeat monitoring. See how heartbeat exchanges reveal unhealthy nodes and help determine appropriate responses.

Explain how internal cluster monitoring uses heartbeats between primary and secondary load balancers to detect failures and enable failover without external monitoring.

Learn to detect faults via external monitoring and self-monitoring heartbeats across stateless components, RAC nodes, and databases, enabling high availability and scalable load management.

Discover how to recover a large-scale system from faults using the redundancies you provisioned to detect issues and restore operation.

Examine stateful failover strategies, including virtual IP and registry-based approaches, with floating IPs, heartbeat monitoring, and DNS TTL or router-based routing to promote standby when the primary fails.

Ensure high availability by using two load balancer instances with a floating IP; heartbeats detect failure, and standby transfers the IP to become the new active, recovering the load balancer.

Examine hot standby in a master-slave database, using synchronous replication to prevent downtime and data loss. Understand primary and secondary roles, failover, and how network lag affects latency.

Discover cold database recovery, using backups and redo log files to restore a corrupted database, remove faulty log statements, and apply remaining changes for disaster recovery.

This lecture explains high availability in large-scale systems using multiple data centers, with synchronous replication between Mumbai sites and asynchronous replication to Singapore for disaster recovery and failover.

Explore how retries stabilize large-scale systems by handling transient and system errors, manage race conditions, and apply exponential back-off with jitter for robust, idempotent requests.

Explore how to design highly available and reliable systems with redundancy, stateless versus stateful components, automated fault detection and failover, and resilience patterns like timeouts, retries, circuit breakers, shed load.

Examine network security for remote and intranet access with public key cryptography, certificates, signatures, HTTPS, OAuth 2.0, and JWT; guard against SQL injection, CSS attacks, and cross-site resource forgery.

Enforce access control, verify identities, and authorize users to limit system access. Encrypt data in transit and at rest, manage tokens, and enable single sign-on, while addressing common vulnerabilities.

Explore network security as the fundamental part of security for any software system and its role in protecting software.

Explore public key encryption, also called asymmetric encryption, using public and private keys to achieve confidentiality, authentication, and non-repudiation when exchanging messages.

Explains how ssl/tls secures client-server communication over https by exchanging a server public key, then establishing a symmetric key for fast encrypted messaging, using certificates for verification.

Learn how SSL and TLS secure TCP communication, enable HTTPS and app protocols like HTTP, POP, IMAP, LDAP, FTP over SSL, and how certificates and public keys prove server identity.

Explain how digital certificates deliver and verify public keys through certificate authorities, with certificates carrying owner identity and validity data, digitally signed to enable trusted SSL/TLS connections.

Understand how a chain of trust forms from root ca to tier 2, tier 3, and tier 4 end-entity certificates, and how browsers verify signatures with public keys during ssl/tls.

Initiate a secure TLS/SSL handshake by exchanging client hello and certificate. Verify the certificate, encrypt a generated symmetric key with the server’s public key, and establish a shared encrypted channel.

Configure and apply firewalls to filter ingress and egress traffic, allowing or denying access based on source and destination IP ranges, ports, and protocols.

Configure firewalls to protect a four-subnet network, exposing external access to demilitarized zones only on port 443, and opening port 8080 for internal services while db uses port 1522.

Explore the essential concepts of authentication and authorization, distinguishing identity proof from access rights and explaining how large-scale systems enforce who can access which functions, services, and data.

Explore stateful and stateless authentication, how client credentials transfer and are verified, and server-side and client-side storage options for credentials and tokens.

Explore how credentials transfer from client to server using html forms, http basic, digest, and certificate based authentication, with https and tls, authorization headers, and human versus programmatic access.

Store user authorization and profile data in databases or LDAP depending on scale and environment. LDAP suits enterprise, centralized authentication and high-read workloads, while RDBMS/NoSQL handle external user systems.

Stateless authentication issues tokens from an authorization service, containing user details, signed for verification by a private/public key pair, enabling decentralized, scalable access via cookies and authorization headers.

Learn access management and authorization, modeling access to your system and implementing controls to regulate who can access resources.

explains role-based access control by organizing users into groups, assigning business roles with permissions to access catalog, inventory, and order services, while system resources rely on service credentials.

Explain OAuth2 authorization for large scale systems using stateless authentication, with bearer tokens (JWT or SAML) and API key options, and clarify the roles of resource and authorization servers.

Explore how OAuth2 tokens are granted by walking through client, resource owner, authorization server, and resource server roles, and see how an authorization code exchanges for a token.

Explore OAuth2's two grants: authorization code flow for internet apps and password flow for intranet apps. The authorization code flow uses redirects to exchange a short-lived code for access tokens.

Explore the OAuth2 password flow, where the resource owner provides credentials to the client, which forwards them to the authorization server to obtain an access token.

Choose password flow as the suitable OAuth2 variant; the web app sends user credentials to the auth service, obtains a token, and accesses catalog or other services via the gateway.

Explore OAuth2 token types—bearer tokens and MAC tokens—and compare their TLS requirements, data integrity, and origin protection, highlighting when bearer or MAC tokens are appropriate.

Compare JSON web tokens with SAML tokens, and learn how header, payload, and signature, plus hs256 and rs256 signing, enable compact, url-safe, browser-friendly authorization.

Store tokens in HTTP-only cookies to shield from JavaScript, and avoid local storage due to XSS risks; single-page apps keep tokens in memory, mobile uses key chain or key store.

Secure data at rest by hashing passwords and using transparent data encryption with a data encryption key; optionally add client data encryption for sensitive columns with a key storage system.

Secure a large-scale system by terminating tls at load balancers for https and using certificates for intranet tls, while applying stateless jwt authentication with oauth2 and rbac for access control.

Identify common vulnerabilities found in many software systems and explore practical steps architects and developers can take to mitigate them.

Explore how SQL injection exploits unsafe queries via category parameters, and how prepared statements ensure only values substitute into queries, preventing data leakage and table drops.

Explore cross site scripting by showing how a malicious script in a user comment stored in a database gets broadcast and executed in browsers, underscoring input validation.

Explore deployment challenges in large-scale systems, including replication and partitioning of databases, queues, caches, and services. Learn automated, continuous deployment strategies to manage complexity across RDBMS and NoSQL.

Explore modern ways of deploying an application and compare deployment approaches for learning purposes today.

Learn how to deploy a web application by breaking down deployment from a host machine to the JVM and web container, copying JARs, and configuring the environment to reduce errors.

Automate component deployment by scripting the installation of JDK, Jetty, and war files from staging, achieving idempotent, declarative deployment with tools like Chef, Ansible, or Puppet.

Explore how virtual machines and hypervisors allocate fixed cpu, ram, and disk quotas to apps on shared hardware. This isolation boosts utilization and prevents hogging.

Compare container deployment with virtual machines, highlighting lightweight images and no guest OS. See how a container runtime translates system calls to the host, enabling fast start and image movement.

Learn how Docker containers are created with Dockerfiles, using a base Ubuntu image, layering commands to install JDK and Jetty, and reuse cached layers for efficient builds.

Explore how to make infrastructure deployment simpler for large-scale systems, focusing on practical techniques that streamline deployment processes within modern software architectures.

Learn how Kubernetes enables lifecycle management for containerized applications, including starting, stopping, monitoring, and automatic restart of faulty containers, through automation and frameworks.

Learn how to assign names and IP addresses to application component instances, and resolve them with DNS and internal DNS, enabling load balancers and clients to locate services.

Scale stateless web and services on demand by issuing commands to create many instances, while Kubernetes enables auto scaling and manages multiple database instances for distributed systems.

Discover how a load balancer provides a single IP contact point for web apps, balances load among instances, and how Kubernetes can automatically create external and internal load balancers.

Kubernetes, an open-source and popular deployment system, provides service naming via internal DNS, health checks, automatic rollouts and rollbacks, container restart and replacement, load balancing, resource limits, and storage mounting.

Explore how Kubernetes maps a web application, a restful service, and a database into scalable workloads using pods, docker images, and internal or external load balancer services.

Explore Kubernetes architecture and how a four-VM cluster deploys master and proxy, sets up a network, and uses API server and scheduler to manage scalable deployments.

Explore the different strategies used to deploy upgrades or fixes to a live production environment within large-scale systems.

Explore canary deployment as a production testing strategy that releases a feature to a small subset of nodes, monitors impact with limited user requests, and gradually rolls out if safe.

Review deployment of large-scale systems, provisioning infrastructure, deploying applications with containers and cloud, and managing Kubernetes operations, while applying upgrade strategies: recreate, rolling update, canary, blue-green, and A/B testing.

Select platforms, tools, and technologies for each layer of a large-scale system, and evaluate databases and web servers against functional and non-functional requirements for an end-to-end e-commerce solution.

Explore the reference system architecture, including web, services, database, and analytics layers, and learn to select platform options by weighing functionality and non-functional requirements such as scalability and security.

Explore architectures for static and dynamic content, including Apache and Nginx servers. Compare Jetty and Tomcat, Node.js, and Spring Boot, with caching, Varnish, and CDN distribution via reverse proxies.

Explore Apache webserver scalability in two modes: as a web server behind a load balancer and as a reverse proxy. Learn how CPU, memory, and thread pools shape scaling.

Nginx architecture uses asynchronous I/O with a single worker thread to handle numerous connections efficiently, performing nonblocking requests to databases, services, or disk, ideal for reverse proxy workloads.

Explore how Nginx acts as a reverse proxy and cache, fronting Apache servers, caching static resources, and load balancing requests across multiple backends in large-scale web architectures.

Explore web containers hosting dynamic content with a servlet engine like Tomcat or Jetty. Spring Boot embeds Tomcat and enables NVC with inversion of control, model view controller, JDBC templates.

Jetty serves as a web container for hosting web apps and http services, while Spring streamlines Java development; use NGINX to cache restful json/xml responses and serve static content.

Explore how Node.js brings JavaScript to the server, handling asynchronous IO with a single-threaded event-driven model to support many IO-bound connections, using JSON across browser, server, and databases.

Explore how the node.js event loop uses a single thread to process requests, execute synchronous code, and offload asynchronous I/O to the operating system while managing callbacks in a queue.

Explore how cloud storage supports static data files from product images to CSS and PDFs through unlimited space, pay-for-what-you-use pricing, versioning, access control, low latency, and high-availability replication.

Explore building services with restful and microservices patterns, focusing on caching and asynchronous processing through messaging queues, and hosting on containers with scalable load balancing via Engine X.

Explore how to design scalable services using web containers, object caching with memcache and Redis, and asynchronous messaging with Redis, RabbitMQ, and Kafka, plus deployment via Kubernetes.

Memcached serves as a centralized key-value cache across nodes, storing objects as blobs with TTL and using expired data first, then LRU eviction to maximize hits for frequently accessed data.

Memcache operates as a cluster of nodes with a client library using DNS to route keys by hash, following a cache-aside pattern with sub-millisecond latency and horizontal scalability.

Explore cloud caching solutions that mirror on-premises options for Memcached and Redis, including infrastructure as a service and fully managed services like AWS Elastic Cache and Google Memorystore.

Explore Rabbit MQ architecture as a general purpose message broker with vertical scalability. Learn about persistent and transient messaging, exchange routing, and master-slave replication for high availability.

Kafka architecture as a distributed log with sequential writes and page cache optimization, explores partitioning for horizontal scalability, offsets, and the pull-based consumer model for high throughput streaming.

Redis pub/sub provides a fast, non-persistent in-memory messaging queue with publishers, subscribers, and channels, delivering messages to connected clients in real time for use cases like live leaderboards.

Compare open source RabbitMQ and Kafka with AWS and Google Cloud MQ solutions; learn managed services like SQS, Kinesis, and Pub/Sub for high throughput, IoT, and big data.

Understand how datastores handle read and write workloads on a shared data set, addressing disc latency and concurrent access, and explore datastores that improve scalability for large-scale systems.

Explore distributed databases as alternatives to RDBMS, comparing key-value, column family, and document stores—DynamoDB, Big Table, Cassandra, HBase, and MongoDB—and learn their functionality, performance, scalability, reliability, and use cases.

Explore how RDBMS scale through vertical upgrades, vertical partitioning across services, and read replicas, then compare with NoSQL solutions to address consistency and scalability limits.

Explore the design objectives and trade-offs of NoSQL databases, including horizontal scalability on commodity hardware, availability, eventual consistency, flexible schema, and limited secondary indexing.

Examine DynamoDB's peer-to-peer architecture with no master, using consistent hashing and gossip protocols for extreme high availability and scalable reads and writes of small data chunks.

Google Big Table, a NoSQL store akin to Apache HBase, uses column families, row keys, and timestamps to provide sparse, sorted, multi-version data with atomic reads and writes.

Bigtable provides schema-less, column-family tables on GFS with sorted data, every column indexed, enabling fast range queries and high-throughput writes through in-memory tablet servers, strongly consistent.

HBase mirrors Bigtable’s architecture with similar functionality and range-partitioned keys, using Hadoop/HDFS as underlying storage, and APIs differ with Zookeeper as the lock service.

Explore how Cassandra blends Bigtable and DynamoDB features, leveraging a column-family data model, partition and sort keys, and a decentralized cluster to deliver high write throughput and fault tolerance.

Explore Cassandra features: a column family data model, schemaless and persistent. It uses hash-partitioned, horizontally scalable storage with eventual consistency, optional range partitioning, and peer-to-peer replication.

MongoDB stores nested JSON-like documents and is horizontally scalable and eventually consistent. It updates fields, indexes top-level and nested fields, and enables fast queries by ID.

Explore MongoDB architecture, including indexing for fast searches, the write overhead trade-off, horizontal sharding, and master-slave replication, with Jason-based document storage for node.js applications.

Explore platform products for building analytics layer, move log data with Logstash and Fluentd, store in Hadoop HDFS or Elastic Search, and review stream processing with Kafka as a buffer.

Explore Logstash architecture to collect, transform, and move log events from services to storage; understand input, filter, and output plugins, queues, and backpressure for reliable delivery to destinations like Elasticsearch.

Explore how Elasticsearch uses inverted indexes to power full-text search on JSON documents, offering field-specific and global queries, mappings, and scalable search architecture.

Explore Hadoop HDFS, a distributed file system that stores large unstructured data by breaking files into chunks across multiple nodes, enabling parallel reads, replication, and petabyte-scale processing with MapReduce.

Explore Map-Reduce on a Hadoop cluster to process huge log files in parallel, using map and reduce phases to generate key-value counts and shuffle data across nodes.

Discover stream processing for real-time analytics on streaming data, using technologies like Kafka, Storm, Flink, Spark Streaming, and discuss fault tolerance, latency, throughput, and handling out-of-order or missing data.

Explore the architecture of large-scale systems, covering front-end layers, caching with Redis or Memcached, content delivery networks, service options, databases, and analytics tools as discussed in the lecture.