While testing in an orchestrator lab I saw that none of my Orchestrator on-raft nodes were coming online after a reboot.

This is the status report from SystemD.

$ sudo systemctl status orchestrator
* orchestrator.service - orchestrator: MySQL replication management and visualization
   Loaded: loaded (/etc/systemd/system/orchestrator.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2020-04-03 09:30:05 UTC; 30s ago
     Docs: https://github.com/github/orchestrator
 Main PID: 957 (code=exited, status=1/FAILURE)

Apr 03 09:30:05 orchestrator-1 systemd[1]: Started orchestrator: MySQL replication management and visualization.
Apr 03 09:30:05 orchestrator-1 orchestrator[957]: 2020-04-03 09:30:05 ERROR dial tcp 127.0.0.1:3306: connect: connection refused
Apr 03 09:30:05 orchestrator-1 orchestrator[957]: 2020-04-03 09:30:05 FATAL dial tcp 127.0.0.1:3306: connect: connection refused
Apr 03 09:30:05 orchestrator-1 systemd[1]: …

Recently we had been bitten by a Systemd limitation at the “Tasks” created per-unit ie., process. This includes both the kernel threads and user-space threads, with each thread counting individually.

Am writing this blog as a reference for someone who might come across this limitation.

We have been actively working on migration DB instances, from one DC to the newly built DC .The instances on the newer DC were provisioned with the latest hardware and latest Debian OS. Below is the detailed spec of the system.

RAM             : 244G
Core             : 44Core
HardDisk.   : SSD
IOPS             : 120K
OS         …

I am currently running tests with sysbench and dbdeployer on a Google Cloud Platform Debian 9 instance.  I ran into an interesting limit and lifting it was not straightforward.  I hope that by sharing this, you can avoid losing too much time.

The benchmark I am using is the insert benchmark.  I am able to run it with 2048 threads but I am not able to run it with 4096 threads (and yes, it makes

This blog will go into how to manage multiple MySQL binary installations with SYSTEMD using the systemctl command.  With package installations of MySQL using YUM or APT, it’s quick and easy to manage your server’s state by executing systemctl commands to stop, start, restart, and status.  But what do you do when you want to install MySQL using the binary installation with a single or with multiple MySQL instances? You can still use SYSTEMD to easily manage the MySQL instances. All commands and testing have been done on Debian, and some details may change in other distro’s.

MySQL preparation

These are the steps to set up MySQL with multiple instances. If you currently have a MySQL server package installation using YUM or APT, it will need to be removed first. Make sure you keep your client. I also had to install some base packages for MySQL on Debian

apt install libaio1 libaio-dev numactl

…

Ever since AMD released their EPYC CPU for servers I wanted to test it, but I did not have the opportunity until recently, when Packet.net started offering bare metal servers for a reasonable price. So I started a couple of instances to test Percona Server for MySQL under this CPU. In this benchmark, I discovered some interesting discrepancies in performance between  AMD and Intel CPUs when running under systemd .

The set up

To test CPU performance, I used a read-only in-memory sysbench OLTP benchmark, as it burns CPU cycles and no IO is performed by Percona Server.

For this benchmark I used Packet.net c2.medium.x86 instances powered by AMD EPYC …

In one of the projects I have been working on, one requirement was the ability to stop traffic from reaching a MySQL host which has been lagging behind its master for longer than a specific amount of time and then bring it back online once the lag has gone away. Of course, this is all automated and no human intervention is required.

In this scenario, we are using HAProxy as the load balancer, and I will walk you through how to configure an agent so we can use HAProxy httpchk to flag the host as up or down, via systemd socket and then automatically set the host as being down/up when applicable, in HAProxy.

I will be setting up a systemd service (I’m running centos7 hosts) and creating a listening socket in the MySQL host we want to monitor so haproxy can have access to replication status.

Scenario:

master: po-mysql1
slaves: po-mysql2, po-mysql3, po-mysql4
secondary slaves: …

CentOS 7 Systemd issue with Percona Cluster is that SST fails to sync the nodes, unable to join cluster group and giving the misleading broken pipe 32 SIG errors.

The post How to Resolve Systemd Issue with Percona XtraDB Cluster on CentOS 7 appeared first on Datavail.

New versions of systemd (like in Ubuntu 16.04) are able to configure pid Cgroups (ulimit -u via Cgroups) using TasksMax.

Check the commit implementing TaksMax.

Author: Lennart Poettering <lennart@poettering.net>
Date:   Fri Nov 13 19:28:32 2015 +0100

    core: enable TasksMax= for all services by default, and set it to 512

    Also, enable TasksAccounting= for all services by default, too.

    See:

    http://lists.freedesktop.org/archives/systemd-devel/2015-November/035006.html

But not we’ve got silly defaults:

$ systemctl show -p TasksMax docker
TasksMax=512

$ systemctl show -p TasksMax mysql
TasksMax=512

So if you use Docker or MySQL most likely you are going to have trouble without really putting load on your server.

Just set TasksMax for your service.

TasksMax=infinity

Feel free to configure that setting fitting …

After some recent work with systemd I’ve realized it’s power and I can come clean that I am a fan. I realize that there are multitudes of posts out there with arguments both for and against systemd but let’s look at some nice ways to have systemd provide us with (but not limited to) pt-kill-as-a-service.

This brief post introduces you to a systemd unit file and how we can leverage them to enable pt-kill at startup, have it start after mysqld and ensure that MySQL is running by using the mysql service as a dependency of pt-kill. By using systemd to handle this we don’t have to complicate matters by ‘monitoring the monitor’ using hacky shell scripts, cron or utilities like monit.

So then, a quick primer on systemd, because lets face it, we’ve all been avoiding it. Systemd is not new but it made recent headlines in …

As the systemd integration keeps on getting forward. It is hardly to be ignored by us MySQL folks :)

Lets have a look into a simple problem, you are not going to solve like you used to solve it. (At least on Centos7 installing the MariaDB package)

Increasing table_open_cache was only a configuration issue. As mysqld was started as root and then switching to the unix user mysql. On Centos7 this was not working anymore. As MariaDB/MySQL ist startet with a service file starting the process as user mysql:

[Service]
Type=simple
User=mysql
Group=mysql
..

As not root (having the right capability) it will be not able to change the open files limit. In the error log you are going to find something like:

150303 11:57:02 [Warning] Changed limits: max_open_files: 1024  
 max_connections: 214  table_cache: 400

Reading the …