InnoDB and the Thread Sanitizer do not work well together in MySQL 5.7.  There are hundreds of possible data races reported by the Thread Sanitizer in the InnoDB code for the simplest case of initializing the MySQL server and shutting it down.  Luckily, most of these data races are benign.  The problem of finding an interesting data race in the InnoDB software is similar to finding evidence of a sub-atomic particle in a particle accelerator; many Thread Sanitizer events must be analyzed before something interesting is found.

One example of a benign data race is InnoDB's implementation of fuzzy counters.  A …

This blog continues the investigation into possible data races in the MySQL server reported by the Thread Sanitizer.  The previous blog investigated several possible data races that were detected when the MySQL server is initialized.  This blog investigates three data races in the performance schema that are reported when the MySQL server is shut down.  The first data race is a racy increment of thread statistics.  The second data race is a problem that may leave the state of the performance schema memory allocator incorrect.  The final data race is in the lock …

The MySQL server is a large multi-threaded program primarily written in the C++ programming language.  Unfortunately,  data races and deadlocks in concurrent C++ programs are common.  Fortunately, the Thread Sanitizer can be used to find data races and deadlocks.  MySQL 5.7 does not support the Thread Sanitizer, but is is easy to add to the MySQL configuration using a simple build procedure.  When MySQL 5.7 is built with the Thread Sanitizer and database …

The LZMA algorithm implemented by the xz compression software package is one of the compression algorithms used by TokuDB for MySQL and TokuMX for MongoDB.  Unfortunately, valgrind's memcheck reports an uninitialized variable problem in the …

MySQL 5.7 supports the Address Sanitizer, which checks for several memory related software errors including memory leaks.  It is really nice to see support for the Address Sanitizer built into MySQL.  Unfortunately, when running the mysql tests included in MySQL 5.7.9, the Address Sanitizer reports several memory leaks, which causes some of  the tests to fail.  Here are some of the memory leaks in the MySQL 5.7.9 software found by the Address Sanitizer.
Memory leaks in 'mysqlbinlog'Several of the 'mysqlbinlog' tests fail due to memory leaks in the 'mysqlbinlog' client program.  Here are the details from the ' …

We remember when we first started auditing MySQL servers, there were very few tools available.  In one of our early big gigs, we were battling serious performance issues for a client.  At the time, tuning-primer.sh was about the only tool available that could be used to diagnose performance bottlenecks.  Fortunately, with a lot of manual interpolation of the raw data it presented, we were able to find the issue with the server and suggest how to resolve them.  For that we are very thankful.  It was a first step in analyzing MySQL status variables, minimizing the number of formulas to learn and calculate by hand.  Obviously doing it by hand takes forever!

Now fast-forward to today.  Unfortunately, not much has changed.  Many DBAs and developers are still using open source tools such as tuning-primer, mysqltuner.pl, mysqlreport, and so on.  Don’t get the wrong; those tools have …

Percona recently open sourced TokuBackup, a library that may be used to take a snapshot of a set of files while these files are being changed by a running application like MySQL or MongoDB.  Making TokuBackup open source allows a larger set of users to improve and extend the software.  This blog describes how the Address Sanitizer and Thread Sanitizer found bugs in the TokuBackup library.

The TokuBackup library is used by Percona Server for MySQL and …

With Galera (Percona Cluster or MariaDB Cluster), it is sometimes advisable to not route traffic to a node during a backup due to the node already being under a heavier load.  In these situations, it may be wise to not route traffic there until the backup is complete.

Since the default /usr/bin/clustercheck script did not have the option of doing this, we created a modified version.  The below script looks for the presence of the xtrabackup tool running in the process list.  If it is found, the clustercheck script returns the appropriate exit code (502) which signals the load balancer to not route traffic its way.  The script could easily be modified to look for the presence of programs/tools in the process list.

We hope others will find it useful.

#!/bin/bash
# clustercheck.sh
#
# Script to make a proxy (ie HAProxy) capable of monitoring Percona XtraDB Cluster nodes properly
#
# Modified by Itchy …

Last week, 21-23 September, it took place the European MySQL Conference, or “Data performance Conference” as this year’s subtitle was “MySQL. NoSQL. Data in the cloud.”. This year, it changed its location from London to Amsterdam and, as most people I talked to agreed, the change was for good. As every year, Percona was the company organizing it, but it had the participation of all the major players in the open source MySQL/MongoDB/Cloud data world. Special mention goes to Booking.com, which had more …

Ever get called out for a MySQL issue only to realize that there was no issue?  It was a false alarm from the monitor.  We sure have and it’s frustrating, especially at 3:00 or 4:00 in the morning!

Many DBAs work in an environment where there is some sort of first level support that gets assigned tickets first.  Unfortunately, many of the times these groups are, shall we say, less than skilled in MySQL.  As a result, they quickly escalate the ticket onto the primary on-call DBA, even when there is really nothing wrong.

Much of the time, there are multiple types of MySQL topology in these environments: standalone, galera cluster, replication, etc.  Writing large runbooks with detailed test cases can be a daunting process and one that will cause many first-level support engineers to give up and simply escalate the issue anyway.

In an effort to avoid undue call outs, we developed a simple bash …