We’re always testing the latest versions of MySQL with most of the environments to make sure that we can find the critical issues before it goes to production. This wasn’t different with the 5.6 MySQL neither. We already started to play with this version in the summer. The first news were very promising. The performance gain is significant. However we have run into couple of errors. One of them was the

mysql_install_db problem with replicating environment

[root@hostname ~]# mysql_install_db --user=mysql --datadir=/mysql/data/

------------------------------------------------------------------------------------
Installing MySQL system tables...
121217 10:02:20 InnoDB: The InnoDB memory heap is disabled
121217 10:02:20 InnoDB: Mutexes and rw_locks use GCC atomic builtins
121217 10:02:20 InnoDB: Compressed tables use zlib 1.2.3
121217 10:02:20 InnoDB: Using Linux native AIO
121217 10:02:20 InnoDB: CPU supports crc32 instructions
121217 10:02:20 …

New versions of MySQL are always interesting to try out. Often they have features which I may have asked for myself so it’s satisfying to see them eventually appear on a system I use. Often other new features make life easier for the DBA. Finally we hope overall performance will improve and managing the server(s) will be come easier.

So I had a system which needs to make heavy writes, and performance was a problem, even when writing to SSDs. Checkpointing seemed to be the big issue and the ib_logfile size in MySQL 5.5 is limited to 4 GB. That seems a lot, but once MySQL starts to fill these files (and this happens at ~70% of the total I believe),  checkpointing kicks in heavily, and slows things down.  So the main reason for trying out MySQL 5.6 was to see how things performed with larger ib_logfiles. (Yes, MariaDB 5.5 can do this too.)

Things improved a lot for my specific workload which was great news, but one thing …

Warning... what follows is a murky mess.
It's filed as MySQL bug#43398 (verified!) but it's triggered by cPanel doing evil.

Scenario....Start a mysql server, as normal
Then run mysql_install_db (as root, like you would when you first install MySQL)
See a new binlog file get created, with ownership/group root!Of course you generally wouldn't run mysql_install_db while a server is running, but there's nothing to prevent you (or something else) from doing so!
--bootstrap just shouldn't initialise binlog, then there wouldn't be a issue.

cPanel runs mysql_install_db in its automatic upgrade scripts (dangerous already, automatically upgrading MySQL Server on a system!), it's run every night on cPanel systems even if no upgrade is done, and it behaves exactly as described above. It then chowns the binlog files to mysql:mysql which is of course …