Demystifying CDC: Understanding Change Data Capture in Plain Words
Demystifying CDC: Understanding Change Data Capture in Plain Words
Your essential guide to Change Data Capture
In my work experiences (in the field of Big Data analysis and Data Engineering), the projects are always different, but they always follow a consolidated schema: the goal is to create a data platform that collects data from different sources, performs a series of elaborations, and exposes the consolidated data to those who will then use it.
The schema just described is often summarized in the concepts of Data Lake/Data Lakehouse and ETL (Extract-Transform-Load) flows. The different ways of extracting data from source systems fall into two categories:
- batch: the entire data set is extracted from the source in a single operation
- streaming: the extraction is performed continuously, monitoring the source for any changes. Data is extracted as soon as it is modified
New technologies, new architectures and new approaches emerge every year, but one method that continues to be used frequently is Change Data Capture.
What is Change Data Capture (CDC)?
