This week in Support was busy enough for me. Among other things I had to study all possible reasons (other than obvious query cache impact) for queries hanging in "query end" status and noted Bug #80652 related to binlog group commit and fixed in MySQL 5.7.17+ and 8.0.1+ only. The case I had to review was related to Galera though, and I suggest you to note that "query end" may be related to Galera replication stall. Studying this path further soon brought lp:1197771 - "Cluster stalls while distributing transaction" to my attention again, so I asked about proper status for it on Facebook. As it happens way too often recently, I've got few 'Likes" but no further comments, neither …[Read more]
3 Older Entries »
I recently worked with a customer who had a weird issue: when
their MySQL server was started (Percona Server 5.5), if they try
service mysql start a second time, the init
script was not able to detect that an instance was already
running. As a result, it tried to start a second instance with
the same settings as the first one. Of course this fails and this
creates a mess. What was the issue? A missing rule in SELinux. At
least it looks like
If SELinux is set to enforcing and if you are using Percona
Server on CentOS/RHEL 6 (other versions could be affected),
service mysql start doesn’t work properly and a fix
is simple to run:
# grep mysqld_safe /var/log/audit/audit.log | audit2allow -M mysqld_safe # semodule -i mysqld_safe.pp # service mysql restart
Other options are:
- Set SELinux to permissive
- Use the CentOS/RHEL standard MySQL init script …
If you're setting up MySQL Cluster on Oracle Linux or another Linux such as Red Hat, CentOS or Fedora, you might have come across some problems getting the nodes to communicate. There are two ways you might bump into problems with network connectivity: The iptables firewall, and SELinux. These security mechanisms might prevent your various nodes—management, data, and API—from communicating with each other in various ways and with various symptoms.
Let's have a look at what you're likely to see.
Data nodes stuck at "starting"
The first thing you might notice is that your data nodes get stuck in the "starting" state. Running show in the management client gives something like this:
[ndbd(NDB)] 2 node(s) id=2 @192.168.56.211 (mysql-5.6.11 ndb-7.3.2, Nodegroup: 0, Master) id=3 @192.168.56.212 (mysql-5.6.11 ndb-7.3.2, starting, Nodegroup: 0)
...and it never moves away from starting.…[Read more]
I've previously written about AppArmor and MySQL, and how to change MySQL's default file locations on systems with AppArmor enabled. Ubuntu and SUSE ship with AppArmor enabled, but some other distributions such as Oracle Linux don't, along with related distrubutions such as Red Hat, CentOS and Fedora. Rather, these other distributions use another mandatory access control system called SELinux.
Here's some technical detail that might come in handy later.
SELinux uses concepts such as types and domains. Types belong to resources such as files and ports; these are the "objects" in SELinux. Domains contain the "subjects" (processes) and object types that are associated with each other in some way, for example because they are all related to …[Read more]
There was one MySQL server with a Adaptec Raid controller and 4
disks. One of the disks was having media errors and caused the
whole SCSI bus to become unavailable.
This resulted in a corrupted InnoDB table.
Luckily we did have backups. A full backup and incrementals.
So to restore the backups I installed XtraBackup and MySQL 5.5 on another server.
Then the first step was to 'prepare' the backup. This worked okay for the full backup (redo only).
The second step to add the incremantals failed for the first incremental. This was easily resolved by specifying the full paths instead of relative paths.
Then the backup was fully prepared using the redo logs and undo logs.
As XtraBackup doesn't backup your my.cnf we copied the my.cnf from another server and adjusted it for this server. The my.cnf in your backup only contains everything needed for a …
I don't know why SELinux problems seem so frustrating. The problem almost certainly is related to the fact that there is frequently no error message. This is exactly the problem I ran into while turning up a new Apache web server on Red Hat Enterprise Linux 6 (RHEL6) with SELinux enabled.
I feel a sense of pride when I think that I was involved in the development and maintenance of what was probably the first piece of software accepted into Debian which then had and still has direct up-stream support from Microsoft. The world is a better place for having Microsoft in it. The first operating system I ever ran on an 08086-based CPU was MS-DOS 2.x. I remember how thrilled I was when we got to see how my friend’s 80286 system ran BBS software that would cause a modem to dial a local system and display the application as if it were running on a local machine. Totally sweet.
When we were living at 6162 NE Middle in the nine-eight 292, we got an 80386 which ran Doom. Yeah, the original one, not the fancy new one with the double barrel shotgun, but it would probably run that one, too. It was also …[Read more]
The brief outage was due to a scheduled move of the servers to a separate rack and subnet dedicated to our work with the Center for Information Assurance & Cybersecurity (ciac) at the University of Washington Bothell (uwb), and a11y.com
I am currently exercising the new (to us) equipment and hope to winnow the less than awesome equipment over the next quarter. I spent the last six months finding the best in breed of the surplussed DL385 and DL380 chassis we (work) were going to have recycled. The team and I were able to find enough equipment to bring up one of each with eight and six gigs of memory, respectively. These will make excellent hypervisors for provisioning embedded instances of Slackware, Fedora, RHEL, CentOS, Debian, FreeBSD, OpenSolaris, OpenIndiana, FreeDOS, etc.
When I initially configured this xen paravirt environment, I failed to plan for integration with libvirt, so I am now re-jiggering the software bridges so …[Read more]
I see a lot of people coming by #centos and similar channels asking for help when they’re experiencing a problem with their Linux system. It amazes me how many people describe their problem, and then say something along the lines of, “and I disabled SELinux...”. Most of the time SELinux has nothing to do with the problem, and if SELinux is the cause of the problem, why would you throw out the extra security by disabling it completely rather than configuring it to work with your application?
If you're using a non-standard MySQL data directory on your Red Hat Enterprise Linux (RHEL) server, you may have seen an error like /usr/libexec/mysqld: Can't change dir to '/mysql_data/' (Errcode: 13). The key to fixing this problem is to ensure the new MySQL data directory has the proper SELinux security context. In my case: # […]
3 Older Entries »