The latest from the MySQL community

A few thousand gcache.page.* files in a Percona XtraDB Cluster (PXC) data directory is not something you see every day. We came across a case where these files had been accumulating over time and slowly consuming disk space. So, let’s dig into what happened.

At first glance, it looked like GCache had simply stopped cleaning itself up. The investigation started by answering two simple questions: when did the files start appearing, and what changed in the cluster at that time?

Finding the starting point

The oldest files showed the issue started on July 9.

[hostx] percona@hostx: ~ $ ls -lh /var/lib/mysql/mysql-data/gcache.page.*
-rw-r----- 1 mysql mysql 128M Jul 9 21:42 /var/lib/mysql/mysql-data/gcache.page.000000
-rw-r----- 1 mysql mysql 128M Jul 9 21:42 /var/lib/mysql/mysql-data/gcache.page.000001
-rw-r----- 1 mysql mysql 128M Jul 9 21:42 /var/lib/mysql/mysql-data/gcache.page.000002
... …

What if we tell you that there is a sure way to get IST not SST for node rejoins? You can guarantee that a new node rejoins using IST. Sound interesting? Keep reading.

Normally when a node is taken out of the cluster for a short period of time (for maintenance or shutdown), gcache on other nodes of the cluster help donate the missing write-set(s) when the node rejoins. This approach works if you have configured a larger gcache, or the downtime is short enough. Both approaches aren’t good, especially for a production cluster. Also, a larger gcache for the server lifetime means blocking larger disk-space when the same job can be done with relative smaller disk-space.

Re-configuring gcache, on a potential DONOR node before downtime requires a node …