Previously posted on link at Nov 3, 2020. Traditionally, MySQL has been used to power most of the backend services at Bolt. We’ve designed our schemas in a way that they’re sharded into different MySQL clusters. Each MySQL cluster contains a subset of data and consists of one primary and multiple replication nodes. Once data is persisted to the database, we use the Debezium MySQL Connector to capture data change events and send them to Kafka.
In my previous blog post, I wrote about analyzing MySQL Slow Query Logs with ClickHouse and ClickTail. One of the follow-up questions I got is when do you want to do that compared to just using tools like Percona Monitoring and Management or VividCortex, which provide a beautiful interface for detailed analyses (rather than spartan SQL interface).
MySQL Logs…[Read more]
In row based replication, the row data generated by DML is logged into binary log with some metadata. For example column type, type length etc. In the new release MySQL-8.0.1, more table metadata are written into binary log. All metadata together brings users and us below benefits:
- Allows us to build robuster replication and convert row data between types smoothly.
What is Change Data Capture and why do we need it?
Change Data Capture (CDC) tracks data changes (usually close to realtime). In MySQL, the easiest and probably most efficient way to track data changes is to use binary logs. However, other approaches exist. For example:
- General log or Audit Log Plugin (which logs all queries, not just the changes)
- MySQL triggers (not recommended, as it can slow down the application — more below)
One of the first implementations of CDC for …[Read more]