System Design Masterclass: Scalable Financial Systems

Why a Simple Duplicate Payment Problem Traps Even Experienced Engineers

A user taps "Pay $50." The request succeeds, but the response is lost. The user taps the button again.

Simple problem, right?

In this simulation, you'll step into a Staff Engineer interview and tackle one of the most common challenges in distributed systems: preventing duplicate payments. We'll examine the solution most engineers propose, walk through it step-by-step, and uncover a subtle race condition that can result in charging a customer twice.

By the end of this lesson, you'll understand why seemingly correct code can fail under concurrency and why "check-then-act" is one of the most dangerous patterns in system design.

Key Concepts Covered

• Duplicate payment scenarios

• Retry and double-click behavior

• The Check-Then-Act pattern

• Race conditions

• Concurrency timelines

• False positives in duplicate detection

Why Check-Then-Act Fails in Distributed Systems

In the previous lesson, we saw how a seemingly reasonable solution could charge a customer twice. In this lesson, we take a step back and uncover the real problem behind the bug.

You'll learn why the popular "check-then-act" pattern breaks down under concurrency, how race conditions emerge in distributed systems, and why solutions that work perfectly on a single machine often fail once multiple servers and networks are involved.

Using visual diagrams and real-world examples, we'll connect race conditions to common production issues such as duplicate payments, overbooked tickets, negative inventory, duplicate orders, and lost updates.

By the end of this lesson, you'll understand the root cause behind an entire class of distributed systems failures and the tools engineers use to prevent them.

Key Concepts Covered

• Race conditions

• Check-Then-Act pattern

• Race windows

• Concurrency and distributed systems

• Single-machine vs distributed-system thinking

• Common manifestations of race conditions

• Overview of techniques used to prevent concurrency issues

From race conditions to atomic guarantees using Redis locks and state machines.

Description

In this lesson, we finally build the Staff Engineer solution to the duplicate payment problem. You'll learn how a Redis-backed idempotency layer works, why atomic operations matter, and how a simple three-state model (STARTED, PROCESSING, COMPLETED) prevents duplicate charges under concurrent requests.

By the end of this lesson, you'll understand how production payment systems make retries safe without sacrificing performance.

Topics covered:

• Redis atomic operations

• Idempotency state machines

• Request lifecycle management

• Response caching strategies

• Production-grade payment protection

What happens when your lock store disappears?

Description

Redis is fast, but it isn't always the right source of truth. In this lesson, we explore how to implement idempotency directly inside a relational database using unique constraints and transactional guarantees.

This approach is common in high-value financial systems where losing an idempotency record is unacceptable.

Topics covered:

• Database-backed idempotency

• Unique key constraints

• Transactional guarantees

• Redis vs database trade-offs

• Choosing the right persistence model

How to recover when a server dies in the middle of a payment.

Description

A perfect idempotency system still has one dangerous edge case: server crashes. What happens if a request acquires a lock and then disappears?

In this lesson, you'll learn a practical recovery strategy that detects abandoned requests, safely retries work, and prevents customers from being stuck forever.

Topics covered:

• Crash recovery patterns

• Lock expiration and TTLs

• Recovery decision trees

• Handling stranded requests

• Self-healing architectures

Why synchronized retries can bring down healthy systems.

Description

Even when your service is healthy, thousands of clients retrying at the same moment can create a traffic avalanche. This phenomenon is known as the Thundering Herd problem.

In this lesson, you'll learn how exponential backoff and jitter spread traffic safely across time and prevent retry storms from overwhelming your infrastructure.

Topics covered:

• Exponential backoff

• Jitter strategies

• Retry storm prevention

• Traffic smoothing

• Large-scale reliability patterns

Everything you need to answer idempotency interview questions with confidence.

Description

Let's bring everything together. In this recap session, we'll review every failure mode, every architectural decision, and every production pattern we covered throughout Simulation 1.

You'll leave with a complete mental model for designing payment APIs that never double-charge users and a framework you can confidently use in Staff Engineer interviews.

Topics covered:

• Duplicate payment prevention

• Redis idempotency patterns

• Database-backed alternatives

• Crash recovery strategies

• Retry handling and jitter

• Interview-ready summary and flashcards

Understand race conditions, Read–Modify–Write hazards, and why concurrent requests can silently corrupt financial data.

Description

Two customers pay the same merchant at exactly the same time.

Both payments are valid.

Both requests succeed.

Yet the merchant ends up with $150 instead of $200.

Where did the missing $50 go?

In this lesson, you'll explore one of the most important concurrency bugs in distributed systems—the Lost Update Problem. Using a realistic payment processing scenario, you'll see how a seemingly correct implementation fails under concurrent execution, even when there are no crashes, network failures, or database outages.

You'll learn:

• What concurrency really means in distributed systems

• Why the Read → Modify → Write pattern is dangerous

• How race conditions silently overwrite valid updates

• The concept of the race window (or danger zone)

• Why simple application logic breaks under concurrent requests

• How Lost Updates relate to double payments, overbooking, negative inventory, and TOCTOU (Time of Check to Time of Use)

• Why financial systems require specialized concurrency control mechanisms

By the end of this lecture, you'll understand that many production failures aren't caused by incorrect business logic—they're caused by timing. This foundation prepares you for the next lessons, where you'll learn how Optimistic Locking and Pessimistic Locking eliminate these concurrency issues.

Optimistic Locking – Preventing Lost Updates Without Database Locks

Learn how version-based concurrency control detects conflicts, prevents lost updates, and scales modern financial systems.

Description

In the previous lesson, you saw how a simple Read → Modify → Write workflow can cause a Lost Update, allowing money to disappear even when every request is valid.

In this lecture, you'll learn one of the most widely used concurrency control techniques in modern distributed systems: Optimistic Locking.

Rather than locking database rows and blocking other requests, Optimistic Locking assumes conflicts are uncommon. It allows concurrent updates to proceed and detects conflicts using a version number, ensuring that stale data never overwrites the latest changes.

In this lesson, you'll learn:

• Why Optimistic Locking is preferred in high-scale payment systems

• How version numbers detect concurrent modifications

• How conditional updates prevent Lost Updates

• What happens when two requests modify the same record simultaneously

• Why retrying is safer than silently overwriting data

• The advantages and trade-offs of Optimistic Locking

• When Optimistic Locking is the right choice in production systems

By the end of this lecture, you'll understand how modern financial platforms safely process thousands of concurrent requests while maintaining data integrity, high throughput, and scalability—without relying on long-running database locks.

Pessimistic Locking – Preventing Conflicts with Row-Level Locks

Learn how SELECT FOR UPDATE works, why row-level locking guarantees consistency, and the trade-offs between correctness and scalability.

Description

Optimistic Locking isn't the only way to solve concurrency problems.

In this lesson, you'll be introduced to the second major concurrency control strategy used in relational databases: Pessimistic Locking.

Unlike Optimistic Locking, which detects conflicts after they occur, Pessimistic Locking assumes conflicts are likely and prevents them from happening by acquiring a lock before any updates are made.

We'll explore the core idea behind row-level locking, understand how SELECT ... FOR UPDATE works at a high level, and discuss why many relational databases rely on this approach to guarantee data consistency.

In this lecture, you'll learn:

• What Pessimistic Locking is

• How it differs from Optimistic Locking

• The purpose of SELECT ... FOR UPDATE

• Why row-level locks prevent concurrent modifications

• Why Pessimistic Locking guarantees strong consistency

• The fundamental trade-off between correctness and concurrency

By the end of this lecture, you'll have a solid understanding of the principles behind Pessimistic Locking and be ready to explore its internal workflow, implementation, and performance implications in the next lesson.

Pessimistic Locking in Action – How Row Locks Impact Performance at Scale

Subtitle

Explore how SELECT FOR UPDATE works, why it guarantees consistency, and why long-held locks become a bottleneck in high-scale payment systems.

Description

In the previous lesson, you learned the concept behind Pessimistic Locking. In this lecture, we'll dive deeper and see exactly how it works in practice.

You'll follow the complete lifecycle of a database row lock—from acquiring the lock, executing business logic, updating the record, and finally releasing the lock. Through code examples and execution timelines, you'll understand how relational databases serialize concurrent updates to guarantee correctness.

We'll also analyze what happens during a high-traffic flash sale and discover why holding database locks while waiting for external services can dramatically reduce throughput and increase customer wait times.

In this lecture, you'll learn:

• How row-level locking works internally

• How SELECT ... FOR UPDATE (and SQL Server update locks) serialize concurrent transactions

• What happens when multiple transactions compete for the same row

• How databases automatically manage lock acquisition and release

• Why Pessimistic Locking guarantees strong consistency

• The performance impact of lock contention during flash sales and high-volume workloads

• Why Staff Engineers evaluate both correctness and scalability when choosing a concurrency strategy

By the end of this lecture, you'll understand why Pessimistic Locking is one of the safest concurrency control mechanisms—and why, despite its strong consistency guarantees, modern payment systems often avoid long-running database locks under heavy load.

Optimistic vs. Pessimistic Locking – Choosing the Right Concurrency Strategy

Subtitle

Compare both locking strategies, understand their trade-offs, and learn how architects choose the right approach for real-world payment systems.

Description

Knowing how a concurrency control mechanism works is only half the battle.

The real architectural challenge is knowing when to use it.

In this lesson, you'll compare Optimistic Locking and Pessimistic Locking side by side, analyzing how each strategy behaves under different workloads—from low-concurrency business applications to high-scale payment platforms processing thousands of concurrent requests.

Using practical benchmarks and real-world scenarios, you'll learn why there is no universal winner and how experienced architects make concurrency decisions based on workload characteristics rather than personal preference.

In this lecture, you'll learn:

• How Optimistic and Pessimistic Locking compare under different workloads

• The trade-offs between throughput, latency, retries, and blocking

• When Pessimistic Locking is still the right choice

• Why Optimistic Locking is preferred for most high-scale payment systems

• How workload characteristics influence architectural decisions

• The Staff Engineer mindset for selecting concurrency control strategies

By the end of this lecture, you'll be able to evaluate concurrency requirements, understand the strengths and weaknesses of both approaches, and confidently choose the locking strategy that best fits your system's consistency, performance, and scalability requirements.

The McDonald's Problem – When Optimistic Locking Stops Scaling

Subtitle

Understand why extreme write contention breaks traditional concurrency control and creates a hotspot in modern payment systems.

Description

Optimistic Locking works well when conflicts are rare.

But what happens when 500 payments per second all try to update the same merchant balance?

In this lesson, you'll explore one of the most common scalability challenges in high-volume payment systems: the hotspot problem. Using the example of a large merchant like McDonald's, you'll discover why a single database row can become the bottleneck—even when your concurrency control strategy is working exactly as designed.

In this lecture, you'll learn:

• Why high-contention workloads create database hotspots

• How constant write conflicts impact system performance

• Why Optimistic Locking begins to struggle under extreme contention

• Why simply switching to Pessimistic Locking isn't enough

• Why adding more application servers doesn't solve the problem

• The architectural challenge that sets up the next solution

By the end of this lecture, you'll understand why solving concurrency problems isn't always about choosing a better locking strategy. Sometimes, you need to rethink the architecture itself.

Balance Sharding – Eliminating Database Hotspots at Scale

Subtitle

Learn how modern payment systems distribute writes across balance shards to reduce contention, improve throughput, and scale under extreme concurrency.

Description

In the previous lesson, you discovered why a single merchant balance becomes a bottleneck under heavy write traffic. When hundreds of concurrent payments compete for the same database row, retries increase, throughput drops, and traditional concurrency control reaches its limits.

In this lesson, you'll learn how large-scale payment systems solve this problem using Balance Sharding.

Instead of storing a merchant's balance in a single row, you'll see how distributing it across multiple balance shards eliminates hotspots, dramatically reduces write contention, and enables the system to process thousands of concurrent updates while preserving correctness.

In this lecture, you'll learn:

• Why Balance Sharding eliminates write hotspots

• How merchant balances are split across multiple shards

• How deterministic hashing distributes incoming writes

• How the write path continues using Optimistic Locking

• How the read path reconstructs the total balance

• The trade-offs between write scalability and read complexity

• When Balance Sharding should—and shouldn't—be used

• How experienced architects choose between a simple balance model and a sharded design based on workload characteristics

By the end of this lecture, you'll understand one of the most effective scaling techniques used in modern financial systems and know when to apply Balance Sharding to eliminate contention without sacrificing correctness or maintainability.

Simulation 2 Recap – From Concurrency Bugs to Scalable Architecture

Subtitle

Connect everything you've learned about Lost Updates, locking strategies, trade-offs, and Balance Sharding into one complete architectural story.

Description

Congratulations on completing Simulation 2!

In this recap, we step back from the individual concepts and connect the entire journey—from the first Lost Update bug to designing a highly scalable payment system capable of handling thousands of concurrent transactions.

Rather than reviewing isolated topics, we'll revisit the architectural decisions that shaped our solution and, more importantly, the mindset behind those decisions.

In this recap, you'll revisit:

• How the Lost Update problem exposed the challenges of concurrent data modifications

• Why Optimistic and Pessimistic Locking solve the same problem using different philosophies

• How to think about architectural trade-offs instead of searching for perfect solutions

• Why Optimistic Locking breaks down under high contention

• How Balance Sharding eliminates hotspots by redesigning the data model

• The progression from Junior Engineer to Staff Engineer thinking

• The key architectural lesson: remove bottlenecks instead of endlessly optimizing them

By the end of this recap, you'll have a clear understanding of how every concept in this simulation fits together into a coherent architectural solution. You'll also be ready for the next simulation, where we'll answer a new question:

Even if the balance is correct, how do we prove it's correct?

That's where we'll build the Double-Entry Ledger—the foundation of every modern financial system.

The Audit Problem: Why Account Balances Alone Are Not Enough

Subtitle

Discover why storing only account balances fails during audits, why mutable data destroys history, and how this leads to the need for immutable financial ledgers.

Description

In this lesson, we'll begin Simulation 3 by exploring one of the biggest challenges in financial system design—proving where money went.

Imagine an auditor asking you to explain exactly how a customer's balance was calculated. If your system only stores the latest balance, can you reconstruct the complete history with confidence?

Through a real-world production scenario, you'll discover why mutable balance columns destroy historical evidence, why logs and backups are not the source of truth, and why financial systems require traceability instead of simply storing the current state.

By the end of this lesson, you'll understand the mindset shift that separates traditional CRUD applications from production-grade financial architectures and why modern payment platforms rely on immutable ledgers instead of mutable balances.

In this lesson, you'll learn:

• Why balances alone are insufficient for financial systems

• The limitations of logs, monitoring, and database backups

• How mutable updates destroy historical evidence

• What traceability means in financial architectures

• The architectural mindset required to build auditable systems

In the next lesson, we'll solve this problem by building an immutable Double-Entry Ledger, the foundation of every modern financial platform.

Building the Double-Entry Ledger | The Foundation of Financial Systems

Learn how banks and payment systems preserve financial history using immutable ledger entries, debit/credit records, replay, and double-entry bookkeeping.

Description

In the previous lesson, we discovered why storing only account balances makes financial systems difficult to audit and debug.

In this lesson, you'll build the architecture that solves that problem: the Double-Entry Ledger.

Instead of overwriting balances, you'll learn how modern financial systems preserve every financial event as an immutable ledger entry. Through step-by-step whiteboard diagrams and practical examples, you'll see why every transaction creates both a debit and a credit, how replaying ledger history reconstructs account balances, and why the ledger becomes the ultimate source of truth.

This lesson focuses on the architectural principles behind financial systems—not accounting theory—so you'll understand why banks, payment processors, and fintech companies rely on this design.

In this lecture, you'll learn:

• Why updating balances destroys valuable financial history

• How Double-Entry Bookkeeping solves the audit problem

• Why every transaction creates both a debit and a credit

• How immutable ledger entries preserve permanent financial evidence

• How replaying the ledger reconstructs account balances

• Why the ledger becomes the source of truth in financial systems

• The core architectural vocabulary: Ledger, Ledger Entry, Debit, Credit, Immutable, Replay, and Source of Truth

By the end of this lesson, you'll understand why modern financial systems are designed around immutable ledgers and how this architectural pattern makes transactions traceable, auditable, and mathematically verifiable.

The Performance Problem (Part 1) – Why You Can't Replay the Ledger for Every Request

Subtitle

Discover why immutable ledgers provide perfect correctness but create performance challenges when serving millions of balance requests.

Description

In the previous lesson, we established that the Double-Entry Ledger is the ultimate source of truth for financial systems. Every transaction is preserved forever, making balances fully traceable, auditable, and mathematically verifiable.

But this introduces a new architectural challenge.

If the ledger contains years of transaction history, should the application replay every ledger entry every time a customer checks their account balance?

In this lesson, you'll explore the performance implications of treating the ledger as the source of truth and understand why correctness alone isn't enough when designing high-scale financial systems.

In this lecture, you'll learn:

• Why replaying the entire ledger guarantees correct balances

• How immutable ledgers preserve complete financial history

• Why reading directly from the ledger becomes expensive as transaction history grows

• The trade-off between correctness and performance

• Why large financial systems cannot replay millions of ledger entries for every balance request

• The architectural question that leads to a separate read model

By the end of this lecture, you'll understand why financial systems face a fundamental trade-off between preserving complete history and delivering fast user experiences, setting the stage for the production-grade architecture you'll build in the next lesson.

CQRS Lite in Financial Systems — From Ledger to Balance Cache

Subtitle

Learn how production financial systems separate the Write Model from the Read Model, maintain both atomically, and rebuild balances from an immutable ledger.

Description

In this lesson, we evolve the immutable ledger into a production-grade financial architecture using CQRS Lite. You'll learn why the ledger remains the single source of truth while a separate balance cache delivers fast reads. Through whiteboard diagrams and real-world examples, we'll see how ACID transactions keep both models synchronized, how replay enables self-healing after failures, and why separating responsibilities is one of the defining characteristics of great software architecture. By the end of this lesson, you'll understand not just how CQRS Lite works, but why nearly every modern financial system follows this architectural pattern.

The System Account — Where Does Money Come From?

Subtitle

Discover why every deposit must have a matching debit and uncover the architectural question that every financial system must answer.

Description

In this lesson, we explore one of the most fundamental questions in financial system design.

When Alice loads $100 into her wallet, we know her account receives a credit—but where does the matching debit come from?

Using a real system design interview scenario, we'll investigate why money can never simply appear inside a double-entry ledger. Instead of jumping straight to the answer, we'll develop the architect's mindset by asking a deeper question: What account exists on the other side of every transaction?

By the end of this lesson, you'll understand why this seemingly simple question forms the foundation of modern payment systems and prepares us to introduce the System Account in the next lesson.

The System Account — Modeling the Outside World

Subtitle

Learn how production financial systems represent banks, payment processors, and external networks while keeping every transaction perfectly balanced.

Description

In this lesson, we complete one of the most important concepts in financial system design: the System Account.

You'll learn how banks, payment processors, and external financial networks are represented inside a double-entry ledger without creating special cases. Through whiteboard diagrams and practical examples, we'll model deposits, purchases, and withdrawals using the same accounting principles that power modern payment platforms.

We'll also explore how ledger entries are stored, why every transaction creates matching debit and credit records, how the accounting equation verifies ledger integrity, and why production financial systems never create money—they simply transfer ownership between accounts.

By the end of this lesson, you'll understand:

• Why System Accounts exist

• How deposits are represented in a double-entry ledger

• Why every transaction follows the same debit/credit pattern

• How ledger rows model real financial events

• How the accounting equation detects corruption

• The architectural mindset behind trustworthy financial systems

Simulation 3 Recap (Part 1): Building a Production Financial System

Subtitle

Review the complete journey—from mutable balances to immutable ledgers, CQRS, replay, system accounts, and a full end-to-end transaction flow.

Description

In this first recap of Simulation 3, we revisit the complete architecture we've built throughout the ledger deep dive. Instead of introducing new concepts, we'll connect everything into one coherent mental model using a series of hand-drawn notebook pages.

You'll review:

• Why mutable balances fail

• How immutable ledgers solve the audit problem

• The role of CQRS and replay

• The five fundamental rules of financial systems

• How a real deposit flows through a production-ready architecture

By the end of this recap, you'll be able to explain the complete transaction lifecycle and understand how all the major building blocks fit together before moving on to the architect's mindset.

Simulation 3 Recap (Part 2): Thinking Like an Architect

Subtitle

Master the architectural mindset behind financial systems through common mistakes, production vocabulary, interview thinking, and one final architecture whiteboard.

Description

In the final recap of Simulation 3, we move beyond implementation details and focus on how experienced architects think about financial systems.

Using a collection of hand-drawn notebook pages, we'll reinforce the principles that separate production-ready designs from beginner implementations.

You'll review:

• The three most common design mistakes in financial systems

• Essential production vocabulary used in interviews

• The architectural principles behind immutable ledgers

• Why truth must always be preserved and views can always be rebuilt

• The complete financial system drawn as a single interview whiteboard

By the end of this session, you'll have a complete mental model of modern financial system architecture and a production-ready whiteboard you can confidently reproduce in system design interviews or real-world architecture discussions.

Part 1: Breaking the Monolith

Subtitle

Why a single database transaction disappears when systems become distributed

Description

Monolithic applications rely on a single ACID transaction to keep data consistent. But when Order, Payment, and Inventory become independent services, that safety net disappears. In this lesson, you'll discover why distributed systems introduce a fundamentally different consistency problem and why every modern architect must rethink transactions.

Part 2: The Promise—and Failure—of Two-Phase Commit

Subtitle

Understanding distributed transactions, why they look perfect on paper, and why they fail in production

Description

Explore Two-Phase Commit (2PC) from first principles. Learn how distributed transactions coordinate multiple databases, why they appear to solve consistency problems, and why latency, blocking, coordinator failures, and network partitions make them unsuitable for modern cloud-native systems.

Part 3: CAP Theorem and the Architect's Trade-Off

Subtitle

Why modern payment systems choose availability over blocking transactions

Description

Learn the CAP Theorem from an architect's perspective and see how it shapes real-world payment architectures. Understand the trade-offs between consistency and availability, why high-scale systems embrace eventual consistency, and how these ideas naturally lead to the Saga Pattern.

The Saga Pattern — Part 1: Why Distributed Transactions Need a New Approach

Subtitle

From Two-Phase Commit to Sagas: Understanding why local transactions replace global transactions in modern distributed systems.

Description

In this first part, you'll discover why traditional distributed transactions don't scale in modern microservice architectures and how the Saga Pattern solves this challenge.

We'll explore:

• Why Two-Phase Commit becomes impractical in production

• The shift from global transactions to local transactions

• How a Saga coordinates distributed business operations

• The fundamental constraints every Saga must satisfy

• The architectural mindset behind modern distributed consistency

By the end of this lesson, you'll understand not only what a Saga is, but also why it has become the standard approach for distributed transactions.

The Saga Pattern — Part 2: Choreography vs. Orchestration

Subtitle

Explore the two most common Saga coordination models and learn how services collaborate without distributed locks.

Description

Once you've understood what a Saga is, the next question becomes: how do services actually coordinate?

In this lesson, you'll learn:

• Event-driven Saga Choreography

• Workflow-driven Saga Orchestration

• How both approaches work in production

• Sample implementation using events and commands

• The strengths and trade-offs of each coordination model

By the end of this lesson, you'll understand the two dominant ways distributed systems implement Sagas and when each approach makes sense.

The Saga Pattern — Part 3: Architect's Guide to Choosing the Right Design

Subtitle

Compare Choreography and Orchestration, master the essential Saga vocabulary, and learn how architects make production design decisions.

Description

In this final part, we move beyond implementation and focus on architectural decision-making.

You'll learn:

• Side-by-side comparison of Choreography and Orchestration

• Essential Saga terminology every architect should know

• When to choose each approach in production systems

• Operational considerations, trade-offs, and best practices

• The architectural mindset behind resilient distributed systems

Finally, we'll set the stage for the next major concept in distributed transactions: Compensating Transactions, where we'll learn how modern systems safely recover from partial failures.

Compensating Transactions (Part 1): Recovering Without Rollback

Subtitle

Learn how distributed systems recover after local transactions commit, why rollback no longer works, and how architects design business-level compensating actions.

Description

In this lesson, you'll learn why traditional database rollbacks break down in distributed systems and how modern applications recover using Compensating Transactions.

We'll explore why every forward action should have a corresponding compensating action, why some business operations cannot truly be undone, and how architects minimize recovery complexity by designing workflows around reversibility.

By the end of this lesson, you'll understand the mindset shift from database rollback to business recovery, a foundational concept for building resilient distributed systems.

Compensating Transactions (Part 2): Detecting & Recovering Zombie Sagas

Subtitle

Discover why some Sagas become permanently stuck, how architects detect forgotten workflows, and the recovery strategies used in production systems.

Description

Compensation only works if the system knows that recovery is required.

But what happens when an event is lost, a service crashes, or a workflow silently stops progressing?

In this lesson, you'll explore Zombie Sagas, one of the most challenging failure modes in distributed systems. You'll learn how production systems detect stalled workflows using orchestration timeouts and background sweep jobs, compare the strengths and trade-offs of each approach, and understand the architectural mindset behind designing systems that automatically detect, recover, and keep the business moving.

Simulation 4 Recap: The Saga Pattern

Subtitle
Connecting Distributed Transactions, Sagas, Compensation, and Recovery

Description
In this recap, we bring together everything you've learned throughout Simulation 4. We'll connect distributed transactions, the limitations of Two-Phase Commit, the CAP Theorem, the Saga Pattern, compensating transactions, and Zombie Saga recovery into one complete architectural picture. Rather than introducing new concepts, this session reinforces the mental models, design trade-offs, and architect's mindset you'll need for interviews and real-world distributed systems.

The Dual-Write Problem & the Transactional Outbox Pattern

Subtitle

Why saving to the database and publishing to Kafka atomically is impossible—and the architectural pattern used to solve it.

Description

In this lesson, you'll discover one of the most important reliability challenges in distributed systems: the Dual-Write Problem.

You'll begin with a common interview question:

"How do you save an Order to PostgreSQL and publish an event to Kafka atomically?"

At first, the solution appears simple—write to both systems one after another. But through real production failure scenarios, you'll see why this approach inevitably creates inconsistent states.

You'll then learn the architectural insight behind the Transactional Outbox Pattern. Instead of trying to coordinate two independent systems, you'll redesign the problem so that both the business data and the event are committed together within a single local database transaction.

By the end of this lesson, you'll understand:

• Why the Dual-Write Problem exists

• Why changing the order of writes never solves it

• The architectural thinking behind the Transactional Outbox Pattern

• How architects reduce distributed guarantees to local guarantees

• The high-level workflow of the Outbox Pattern before exploring implementation details

This lesson builds the mental model you'll use throughout the rest of this section, where you'll implement the pattern using production-grade message relay techniques such as Polling and Change Data Capture (CDC).

The Message Relay

Subtitle

Polling Publisher vs. CDC — Delivering Events from the Outbox to Kafka

Description

In this lesson, we'll build the missing piece of the Transactional Outbox Pattern: the Message Relay.

You'll learn why a relay is needed, why At-Least-Once Delivery is the only realistic guarantee, and compare the two most common production implementations—Polling Publisher and Change Data Capture (CDC). We'll finish with a practical decision framework to help you choose the right approach for your own systems.

Idempotent Consumers

Subtitle

Building Consumers That Stay Correct Even When Messages Are Delivered More Than Once

Description

Reliable delivery isn't enough—your consumers must also handle duplicate events safely.

In this lesson, you'll learn why duplicate delivery is unavoidable in distributed systems, how to build idempotent consumers, and how the Outbox Pattern, Message Relay, Kafka, and Sagas work together as one complete reliability architecture.

Simulation 5 Recap: Building Reliable Event-Driven Systems

Subtitle

From Dual Writes to Production-Grade Reliability

Description

In this final recap of Simulation 5, we bring together every concept you've learned throughout the series into one complete architectural picture.

We'll revisit the complete reliability pipeline—from the Dual-Write Problem and the Transactional Outbox Pattern to Message Relays, Polling vs CDC, At-Least-Once Delivery, and Idempotent Consumers. More importantly, we'll connect these patterns through the mindset and design principles that distinguish production-grade distributed systems.

By the end of this lesson, you'll understand not just how these patterns work individually, but how they combine to build reliable, scalable, and recoverable event-driven architectures.

What you'll revisit

• The complete reliability pipeline

• Architectural design principles

• Common anti-patterns

• Architect's mindset

• Reliability maturity

• Production-ready mental models

The Poison Message Problem

Subtitle

Why One Bad Message Can Stop an Entire System

Description

A single permanently failing message can block an entire event-driven system. In this lesson, you'll understand poison messages, retry storms, common implementation traps, and why simply skipping failed messages leads to silent business data loss.

Dead Letter Queues

Subtitle

Quarantining Poison Messages Without Losing Business Events

Description

Learn the production-grade solution for handling poison messages using Dead Letter Queues (DLQs). You'll explore the architecture, understand where DLQs are used across modern messaging platforms, evaluate their benefits and trade-offs, and implement the complete workflow in code.

Operating Dead Letter Queues - Part 1

Subtitle

Failure Classification and Recovery Strategies

Description

Building a Dead Letter Queue is only the beginning. In this lesson, you'll learn how architects classify failures into transient, permanent, and systemic categories, choose the appropriate recovery strategy, understand key resilience concepts, and develop the mindset needed to design reliable distributed systems.

Operating Dead Letter Queues - Part 2

Subtitle

Failure Classification and Recovery Strategies

Description

Building a Dead Letter Queue is only the beginning. In this lesson, you'll learn how architects classify failures into transient, permanent, and systemic categories, choose the appropriate recovery strategy, understand key resilience concepts, and develop the mindset needed to design reliable distributed systems.

Circuit Breaker Pattern: Protecting Distributed Systems from Cascading Failures - Part 1

Subtitle

Learn how production systems detect unhealthy dependencies, stop cascading failures, recover automatically, and build resilient event-driven architectures.

Description

When a downstream dependency starts failing, blindly retrying requests only makes the situation worse. Instead of helping the system recover, continuous retries increase load, exhaust resources, and can trigger cascading failures across the platform.

In this lesson, you'll learn how the Circuit Breaker Pattern prevents small failures from becoming large-scale outages.

We'll cover:

• Why cascading failures happen

• The domino effect in distributed systems

• The three Circuit Breaker states: Closed, Open, and Half-Open

By the end of this lesson, you'll understand why Circuit Breakers are one of the most important resilience patterns used in modern distributed systems and how to design systems that fail fast, recover safely, and protect the entire platform.

Circuit Breaker Pattern: Protecting Distributed Systems from Cascading Failures - Part 2

Subtitle

Learn how production systems detect unhealthy dependencies, stop cascading failures, recover automatically, and build resilient event-driven architectures.

Description

When a downstream dependency starts failing, blindly retrying requests only makes the situation worse. Instead of helping the system recover, continuous retries increase load, exhaust resources, and can trigger cascading failures across the platform.

In this lesson, you'll learn how the Circuit Breaker Pattern prevents small failures from becoming large-scale outages.

We'll cover:

• How Circuit Breakers detect unhealthy dependencies

• Production configuration and tuning strategies

• How to choose thresholds using real production metrics

• Simplified Circuit Breaker implementation in code

• How Circuit Breakers work with retries, timeouts, and fallback patterns

• Architect's vocabulary and production design mindset

By the end of this lesson, you'll understand why Circuit Breakers are one of the most important resilience patterns used in modern distributed systems and how to design systems that fail fast, recover safely, and protect the entire platform.

Simulation 6 Recap (Part 1): The Recovery Journey

Subtitle

From Failure Detection to Reliable Recovery

Description

In this first recap of Simulation 6, we revisit the complete technical journey of building resilient distributed systems.

Rather than introducing new concepts, this session connects everything you've learned into a single architectural flow—from identifying failures to recovering safely in production.

We'll review:

• The complete resilience journey

• The end-to-end architecture for handling failures

• The Failure Decision Tree

• How architects choose the right recovery strategy

• The Recovery Pipeline used in production systems

By the end of this recap, you'll have a clear understanding of how modern distributed systems detect failures, classify them correctly, and execute structured recovery strategies instead of relying on guesswork or manual intervention.

Key takeaway: Resilient systems don't recover by accident—they recover through deliberate architectural decisions.

Simulation 6 Recap (Part 2): The Architect's Mindset

Subtitle

Rules, Patterns, and Principles for Building Resilient Systems

Description

In the second recap of Simulation 6, we move beyond implementation details and focus on the architectural principles behind resilient distributed systems.

This session reinforces the rules, mindset, and design philosophy that experienced architects use to build systems capable of surviving real-world production failures.

We'll revisit:

• Architect's Rules for resilient system design

• Common production mistakes and how to avoid them

• Essential resilience vocabulary

• How resilience patterns work together

• The Architect's Mindset

• Final lessons from Simulation 6

By the end of this recap, you'll understand that resilience isn't created by a single pattern like retries or Circuit Breakers. It emerges from making thoughtful architectural decisions, combining multiple patterns effectively, and continuously learning from production incidents.

Key takeaway: Failure is inevitable. Cascading failure is optional. Architecture decides the difference.

Simulation 7: The Ghost Payment Problem

Subtitle

Why every financial system needs a reconciliation strategy.

This is simple and sets the expectation clearly.

Alternative:

When distributed systems disagree about what really happened.

Description

In this simulation, you'll discover one of the most challenging problems in distributed financial systems: uncertainty.

Through a realistic payment timeout scenario, you'll see how a payment can succeed at the payment gateway while your own database records it as a failure. You'll learn why retries, longer timeouts, and manual log investigations are not reliable solutions, and why modern financial platforms need an entirely different approach.

By the end of this video, you'll understand why preventing failures isn't enough—and why continuously verifying financial correctness is an essential architectural capability.

The Golden Source: Identifying the Real Source of Truth

Subtitle

Why your database isn't always correct—and how architects determine which system owns reality.

Description

In distributed systems, multiple services maintain copies of the same data. But when those copies disagree, which one should you trust?

In this lesson, you'll learn one of the most important architectural concepts in financial systems: the Golden Source. We'll explore why your database is often just a replica of reality, how architects identify the true owner of a business fact, and why copying data is essential for scalability—even though it inevitably introduces consistency challenges.

By the end of this lesson, you'll understand why reconciliation always starts by answering a single question:

Who owns the truth?

This mindset forms the foundation for designing reliable financial systems and prepares you for the next lesson, where we'll examine why distributed systems drift apart in production.

Why Systems Drift (Part 1): Understanding How Distributed Systems Fall Out of Sync

Subtitle

Discover what System Drift is, why distributed systems naturally diverge, and the first three production incidents that silently create inconsistent data across services.

Udemy Description

Every distributed system eventually faces the same problem:

Different services begin believing different versions of reality.

In this lecture, you'll learn what architects call System Drift—the moment when multiple copies of the same business data stop agreeing with each other.

You'll discover why this isn't caused by poor engineering, but by the unavoidable realities of distributed communication.

In this lesson, we'll cover:

• What System Drift actually is

• Why multiple copies naturally diverge

• Why unreliable communication is the root cause

• Incident #1 — Server crashes after a successful payment

• Incident #2 — Database commit fails after success

• Incident #3 — Lost events and missing updates

• The architectural realization behind all three incidents

By the end of this lecture, you'll understand why keeping multiple systems synchronized is one of the hardest problems in distributed systems.

Why Systems Drift (Part 2): Production Failures Every Architect Must Design For

Subtitle

Explore the remaining production incidents, discover the common pattern behind every failure, and learn why eventual consistency alone cannot eliminate System Drift.

Udemy Description

In Part 1, we introduced System Drift and explored the first three incidents that cause distributed systems to diverge.

In this lecture, we continue with even more realistic production scenarios before bringing everything together into a single architectural mental model.

You'll learn:

• Incident #4 — Consumer crashes during processing

• Incident #5 — Network partitions

• Incident #6 — Partial processing across services

• The common pattern behind every System Drift incident

• Why eventual consistency is not enough

• The Architect's Mission for designing resilient distributed systems

• The mindset required to build systems that recover instead of silently diverging

Rather than treating these as isolated failures, you'll learn to recognize them as different manifestations of the same underlying problem.

This prepares you for the next simulation, where we'll design a Reconciliation Engine capable of detecting drift, comparing multiple copies against the source of truth, and automatically repairing inconsistencies.

Designing the Reconciliation Engine — Part 1

Subtitle

Why Copies Drift and How Great Systems Find the Truth

In this lesson, you'll learn why reconciliation is one of the most important reliability mechanisms in distributed systems. We'll explore why data drift is inevitable, introduce the concept of the Golden Source, define reconciliation, and design the high-level architecture of a production reconciliation engine. This lesson lays the conceptual foundation for the implementation we'll build next.

Designing the Reconciliation Engine — Part 2

Subtitle

Reading Reality, Detecting Drift, and Repairing Copies

In this lesson, we implement the core reconciliation workflow used in production systems. You'll learn how reconciliation engines read from the Golden Source, compare millions of records efficiently, classify different types of drift, choose appropriate repair strategies, and safely restore consistency using idempotent repairs and verification.

Designing the Reconciliation Engine — Part 3

Subtitle

Scaling, Scheduling, and Operating Reconciliation at Production Scale

In this lesson, you'll learn how production reconciliation systems are operated at scale. We'll compare automatic and manual reconciliation, explore scheduling strategies, design scalable reconciliation pipelines, discuss real-world production challenges, review key architectural vocabulary, and conclude with the architectural mindset required to build resilient financial systems.

Simulation 7 Recap: Designing Systems That Continuously Prove Their Correctness

Subtitle

A complete review of the Reconciliation Engine—from drift detection and repair to production-ready architectural principles.

Description

In this recap, we'll revisit the complete journey of designing a production-grade Reconciliation Engine for distributed financial systems.

Rather than introducing new concepts, this session reinforces the key architectural ideas from the simulation and connects them into a single mental model.

During this recap, you'll review:

• Why data drift is inevitable in distributed systems

• The role of the Golden Source and authoritative data

• The complete reconciliation lifecycle—from detection to convergence

• Core architectural principles every reconciliation system should follow

• Common production mistakes and how to avoid them

• A practical Architect's Checklist for designing reconciliation engines

• The vocabulary and mindset used by experienced software architects

By the end of this recap, you'll have a concise framework that you can use to review the entire simulation before interviews, system design discussions, or real-world implementation.

Key takeaway:

Great systems don't remain correct—they continuously prove they're correct.

You Made It — Now Think Like an Architect

Subtitle:
The Architecture Changes. The Reasoning Remains.

Description:
Congratulations on completing the course. Step back from the architecture we built and reflect on the mindset behind it: questioning assumptions, understanding constraints, evaluating trade-offs, and continuously evolving your designs.