Dealing with MySQL you might need to deal with RAID recovery every so often. Sometimes because of client lacking the proper backup or sometimes because recovering RAID might improve recovery, for example you might get point in time recovery while backup setup only takes you to the point where last binary log was backed up. I wanted for a chance to write instructions for recovery for long time
and finally I had gotten the problems with my ReadyNAS Pro 6 which I was setting up/testing at home for use for backups. I got it doing initial sync while it spotted the problem with one other drive and as such RAID volume failed. ReadyNAS has Debian inside and as you can get root login via SSH it can be recovered as any generic Linux server.

When you restart the system RAID5 volume …

Core file can be quite helpful to troubleshoot MySQL Crashes yet it is not always easy to get, especially with recent Linux distributions which have security features to prevent core files to be dumped by
setuid processes (and MySQL Server is most commonly ran changing user from “root” to “mysql”). Before you embark on enabling core file you should consider two things – disk space and restart time. The core file will dump all MySQL Server memory content including buffer pool which can be tens on even hundreds GB of disk space. It can also take very long time to write this amount of data to the disk. If you are using “pid” with core files, which you probably should, as getting different samples often help developers to find what is wrong easier, you may be looking at many times the amount of memory MySQL consumes worth of disk space.

You have to do couple of changes to enable core files. First you need …

It’s Friday again, and time for another TGIF give-away of a Percona Live London ticket! But first, what’s new with the MySQL query cache? You may know that it still has the same fundamental architecture that it’s always had, and that this can cause scalability problems and locking, but there have been some important changes recently. Let’s take a look at those.

The first important change is that both Percona and Oracle actually built some code improvements into the query cache and the interface between it and MySQL. It’s now possible to completely disable it, for example. This used to be possible only by eliminating it at compile time. If you didn’t do that, then there was still a query-cache single choke-point in the server. Now that’s gone. As of MySQL 5.5, the query cache mutex isn’t hit at all if …

One of the typical problems I see setting up ext2/3/4 file system is sticking to defaults when it comes to behavior on errors. By default these filesystems are configured to Continue when error (such as IO error or meta data inconsistency) is discovered which can continue spreading corruption. This manifests itself in a worst way when device have some “flapping” problems returning errors every so often as this would cause some random pieces of data and meta data to be lost. Not good for system running mySQL Server. As far as I understand this problem is limited to EXT2/3/4 while over systems like XFS will not continue if consistency problems are discovered.

So how can you check what error behavior mode your file system has ? Run dumpe2fs /dev/sda1 and you will get something like this:

dumpe2fs 1.41.14 (22-Dec-2010)
Filesystem volume name:
Last mounted on: /mnt/data
Filesystem UUID: …

I see this message on our forums, and I think it’s a great question: “Which version of Percona Server is currently recommended?” It’s really the same question as “Which version of MySQL is currently recommended?” I’ll respond here and then post a link in the forum as a reply.

In my opinion, it’s important to qualify this question by understanding whether we’re talking about an existing MySQL installation, or a new one. The answer is different for each case. (There are other qualifying questions I’d ask too, but this is the biggest distinguisher).

For an existing MySQL database server, I’d encourage not jumping on a new version immediately when it comes out. Let some early adopters try it out first, and when it gets more broad deployment, then consider it. The reason I say this is that the …

One common theme in the questions our MySQL Support customers ask is Replication Lag. The story is typically along the lines everything is same as before and for some unknown reason the slave is started to lag and not catching up any more. I always smile at “nothing has changed” claim as it usually wrong, and one should claim I’m not aware of any change. Digging deeper we would find some change, though often subtle as computers are state machines and with no external and internal changes they behave the same way. First let me start pointing out common causes of replication lag

Hardware Faults Hardware faults may cause reduced performance causing replication to lag. Failed hard drive getting RAID in degraded mode could be one common example. Sometimes it is not fault per say but operating mode change may cause the problem. Many RAID controllers …

Even though multiple fixes have been implemented in Percona Server and MySQL 5.5, there are still workloads in which case mutex (or rw-lock) contention is a performance limiting factor, helped by ever growing number of cores available in the systems. It is interesting though the contention may manifest itself in the different form from the system monitoring standpoint. In many cases as heavy contention happens user CPU will be very high, and the context switches will be somewhere reasonable. In others you would see the CPU usage being low with a lot of CPU being idle, increased compared to normal workload portion of system CPU and high number of context switches. These correspond to different contention situations which can be handled differently.

First situation often corresponds to Innodb spending a lot of CPU time running “loops” as part of spinlock implementation. In many cases busy wait is indeed more efficient than doing …

I got a serendipitous call (thanks!) yesterday asking what would be needed to tune[1] a database for better performance. It is a question that I hear often, but I never thought about answering it in public. Here’s a consolidated version of what I explained during our conversation.

Have realistic expectations about configuration

The first thing to know is that server configuration itself really isn’t something you should expect to deliver huge wins. If MySQL is actually badly configured, you can hurt its performance significantly. Correcting these mistakes can correspondingly improve performance. But such mistakes are relatively few and/or non-obvious to make. A few of the common ones I see are not configuring the InnoDB buffer pool size or log file size, and not using InnoDB. If your server really hasn’t been configured — that is, it’s running with a default configuration — then it’s quite possible you have …

A while back, I wrote a two part post on how you can extract an amazing amount of information about a system’s performance, scalability, queueing, and more by just measuring request arrivals and completions, and the timestamps thereof.

I promised to develop this into a more complete description of how to analyze MySQL’s performance and scalability through nothing more than TCP/IP packet headers. I presented the results of that research at Percona Live in New York City, and the slides and video are online. The video editors didn’t synch the slides …

If you need to change MySQL’s innodb_log_file_size parameter (see How to calculate a good InnoDB log file size), you can’t just change the parameter in the my.cnf file and restart the server. If you do, InnoDB will refuse to start because the existing log files don’t match the configured size.

You need to shut the server down cleanly and normally, and move away (don’t delete) the log files, which are named ib_logfile0, ib_logfile1, and so on. Check the error log to ensure there was no problem shutting down. Then restart the server and watch the error log output carefully. You should see InnoDB print messages saying that the log files don’t exist. It will create new ones and then start. At this point you can verify that InnoDB is working, and then you can delete the old log files.

The typical error …