Change Data Capture (CDC) using Debezium for MySQL

Name: Change Data Capture (CDC) using Debezium for MySQL
Rating: 4.3 (297 reviews)

Complete, In-depth & HANDS-ON practical course on Debezium with MySQL Database

Created byShruti Mantri

Last updated 1/2024

English

English [Auto],Italian [Auto],

What you'll learn

Understand what is meant by CDC
Ability to capture data changes from MySQL DB
Have their own fault-tolerant setup of Debezium connectors
Learn in-depth about the Debezium capabilities

Course content

5 sections • 24 lectures • 2h 13m total length

Introduction to Debezium4:34
Learn Debezium, an open source change data capture platform, that streams row-level database changes in order to Kafka, enabling real-time data sync across MySQL, PostgreSQL, and more.
Introduction to Kafka Connect3:37
Explore Kafka Connect, an open source framework that streams data between Apache Kafka and other systems via source and sink connectors, scalable in standalone or distributed modes.
Change Data Capture2:26
Capture database changes in real time with change data capture, replacing batch snapshots, and feed them into a destination via a Debezium-driven Kafka Connect pipeline.
Introduction to Schema Registry3:06
Explore schema registry as a centralized, rest-based store for Avro, JSON, and protocol schemas, offering version history, compatibility checks, and seamless schema evolution across Kafka producers and consumers.
Debezium for MySQL Database2:39
Explore the Debezium for MySQL division connector and its topologies. Learn how binlog monitoring and position tracking enable reliable changes across standalone, primary and replica, and high availability clusters.
Quiz on Introduction

Installing Zookeeper and Kafka7:31
Install zookeeper and kafka using Confluent Kafka, configure server properties, set the path, and start the Confluent services to enable CDC setup with Debezium for MySQL.
Installing Schema Registry1:59
Install and start the confluent schema registry, verify the service status, and confirm a response from http://localhost:8081.
Choose Debezium Version 1.9 and below0:15
Installing Kafka Connect8:46
Install kafka connect in distributed mode, add the mysql connector plugin, and verify availability via curl and jq on port 8083.
Installing MySQL Server4:36
Install MySQL 5.7 on Docker by pulling the image and running a container with port 3306. Retrieve the root password, log in, and grant full root privileges.
Enabling Binlog for MySQL5:49
Enable binary logging in MySQL by editing /etc/my.cnf, configuring server_id, log_bin, log_bin_format, log_row_image, and expire_logs_days, then restart the container and verify binary logging is on and logs appear.
Quiz on Installation

Setting up MySQL2:04
Set up a MySQL environment by logging in, creating a database named division_demo, creating a user table, inserting a row, and verifying the data with describe and select queries.
Creating Debezium Connector7:09
Create the Debezium MySQL connector via a POST to the Kafka Connect REST API, configuring a unique server name, single task, database include list, and Avro with schema registry.
KafkaConnect APIs3:42
Learn how to use Kafka Connect APIs to create, delete, and manage connectors, and fetch connector details, status, and configuration through curl requests.
Debezium in Action7:11
Demonstrate Debezium in action by using Kafka commands to inspect CDC data from a MySQL table, reading Avro formats via the schema registry, and observing creates, updates, and deletes.
Schema Registry APIs5:49
Explore the schema registry APIs to list schemas, view supported schema types (JSON, protocol, buffer, AVRO), and retrieve subject versions and schemas by id or latest version.
Quiz on Using Debezium

Database / Table Selection8:02
Configure Debezium CDC for MySQL to capture changes with database include lists, table include lists, and column lists, publishing to Kafka topics.
Schema Topics9:47
Compare schema history and schema change topics in Debezium for MySQL, detailing how they store binlog statements, reconstruct schemas, and publish changes to Kafka, with optional schema registry integration.
Topic Routing7:45
Learn topic routing in Debezium for MySQL to consolidate shard topics into one users all shards topic by configuring transforms, ODBC transforms, topic regex, and dollar one replacement.
Topic Routing with Additional Key6:47
Add a shard id key (sha_id) to the cdc key for cross-shard routing, using transforms and regex to derive values, enabling a shard-aware key and a new key-based topic.
Sending Signals in Debezium12:14
Learn how to send signals to division in Debezium, including log and snapshot signals, configure a signal table and data collection, and understand incremental snapshot behavior.

Debezium in Production: Architecture3:09
Explore Debezium in production architecture by comparing standalone versus cluster mode, illustrating fault-tolerant Kafka Connect deployments and automatic task recovery across multiple servers.
Debezium Installation in Cluster Mode7:50
Learn to install Debezium in cluster mode by configuring a two-server MySQL and Kafka Connect distributed setup with Zookeeper, Schema Registry, and proper properties.
Debezium in Cluster Mode in Action7:12
Demonstrate change data capture with Debezium in cluster mode, showing fault-tolerant, distributed kafka connect connectors across cluster hosts. Create ecommerce, inventory, and complaints databases with key tables, illustrating balanced deployment.

Requirements

Basic MySQL commands
Basic knowledge of Kafka

Description

Debezium is an open source distributed platform for change data capture. It is a well-known tool for capturing the data change in your databases, and bring it to your apps as soon as they take place.

Debezium is used in production by a wide range of companies and organizations. Debezium has the capability to capture data change across different databases like MySQL, PostgresSQL, SQL Server, Oracle DB, MongoDB, Cassandra, DB2 and Vitess.

In this course, we will learn in detail how to use Debezium with MySQL database.

What's included in the course?

Complete course on Debezium capabilities explained from Scratch.
Explored in detail how you can integrate Debezium with MySQL server.
Each and Every Debezium concept is explained with a HANDS-ON.
Detailed steps on Debezium setup in both standalone and production scenarios.
Includes even those concepts, the explanation to which is not very clear even in Debezium official documentation.
Related Commands and Datasets used in lectures are attached in the course for your convenience.
Also contains an additional module on how to use Debezium in production.

Who should take this course?

Data engineers who are specializing in Data Ingestion.
Computer graduate interested in mastering Data Engineering.
Data architects who want to understand in detail the concept of change data capture.

Who this course is for:

Data Engineers who want to advance in their careers
Data Engineers who want to understand Data Ingestion process
Data Architects who want to get detailed understanding of Change Data Capture
Engineers who want to learn about Debezium

Change Data Capture (CDC) using Debezium for MySQL

What you'll learn

Explore related topics

Course content

Introduction5 lectures • 16min

Installation6 lectures • 29min

Using Debezium5 lectures • 26min

Features of Debezium5 lectures • 45min

Debezium in Production3 lectures • 18min

Requirements

Description

Who this course is for: