The latest from the MySQL community

Percona Toolkit’s pt-online-schema-change (pt-osc) has long been the preferred solution for performing online schema changes with minimal downtime. Its chunk-based copy algorithm is designed to adapt dynamically to the workload, making it suitable for very large tables in production environments.

However, under specific conditions, one of its optimization mechanisms can become counterproductive.

During a customer engagement involving Percona XtraDB Cluster (PXC), we investigated a case where running pt-online-schema-change with a selective –where clause caused severe Galera Flow Control and general overhead, effectively stalling the cluster for an extended period. The issue was not caused by the schema change itself, but by the interaction between:

• adaptive chunk auto-resizing
• highly selective copying of the newest rows
• clustered replication in Galera
• InnoDB …

A few thousand gcache.page.* files in a Percona XtraDB Cluster (PXC) data directory is not something you see every day. We came across a case where these files had been accumulating over time and slowly consuming disk space. So, let’s dig into what happened.

At first glance, it looked like GCache had simply stopped cleaning itself up. The investigation started by answering two simple questions: when did the files start appearing, and what changed in the cluster at that time?

Finding the starting point

The oldest files showed the issue started on July 9.

[hostx] percona@hostx: ~ $ ls -lh /var/lib/mysql/mysql-data/gcache.page.*
-rw-r----- 1 mysql mysql 128M Jul 9 21:42 /var/lib/mysql/mysql-data/gcache.page.000000
-rw-r----- 1 mysql mysql 128M Jul 9 21:42 /var/lib/mysql/mysql-data/gcache.page.000001
-rw-r----- 1 mysql mysql 128M Jul 9 21:42 /var/lib/mysql/mysql-data/gcache.page.000002
... …

Starting with Percona XtraDB Cluster (PXC) 8.0, replication traffic encryption is enabled by default. That said, it’s common to find clusters running without TLS that suddenly need it: a new compliance requirement, an audit finding, a network segment that is no longer considered trusted.

PXC has a variable for exactly that case, pxc-encrypt-cluster-traffic, which handles SSL encryption for inter-node traffic, including State Snapshot Transfer (SST), Incremental State Transfer (IST), and the group communication the nodes use for replication.

The variable is not dynamic, and turning it on normally costs a full cluster restart since the node that encrypts traffic listens on ssl:// while its peers are still on tcp://. If a node joins the cluster with TLS enabled while remaining nodes don’t, the restarting node fails to reach out to the other peers with the following error:

2026-09-02T02:16:02.359992Z 0 [Note] [MY-000000] …

The common practice is to size the Galera Cache based on write volume measured during peak load, but often it is more of a guesswork. The writeset cache capacity planning is crucial to shorten the maintenance time and avoid long state transfers while the cluster runs with reduced compute power. Now, if you could understand what’s exactly inside the cache, wouldn’t the planning be more aware as compared to only calculating the best size based on wsrep_received/replicated_bytes variables?

Similarly, while dealing with various incidents occurring in Percona XtraDB Cluster or MariaDB Galera Cluster environments, how many times did you stumble upon the fact that the GCache file (galera.cache) is a black box and you can’t inspect it in a meaningful way? 

In some scenarios, having the opportunity to see what exactly ended up in the cache file(s) could help us understand the write workload impact or what happened …