The aim of this post is to enumerate real-world usage of Percona XtraDB Cluster (PXC), and also to solicit use cases from the readers. One of the prominent usages in the production environment that we have come across (and our Percona consultants have assisted) is that of HP Cloud. There is a post about it here by Patrick Galbraith of HP. The post focuses on their deployment of PXC for HP Cloud DNS. The post focuses on the key aspects of

Galera/Percona XtraDB Cluster (PXC) for MySQL is a hot thing right now and some users jump right in without enough testing. Consequently, they’re more likely to either suffer failure or issues that prevent them from moving forward. If you are thinking of migrating your workload to Percona XtraDB Cluster, make sure to go through these basics.

log_slave_updates is REQUIRED

You need to have log_slave_updates enabled on the cluster node acting as async slave for replicated events from the async master to be applied to the other

Percona is glad to announce the release of Percona XtraDB Cluster 5.5.30-23.7.4 for MySQL on April 17, 2013. Binaries are available from the downloads area or from our software repositories.

New Features:

• Percona XtraDB Cluster has implemented initial implementation of weighted quorum. Weight for node can be assigned via pc.weight option in the wsrep_provider_options variable. Accepted values are in the range [0, 255] (inclusive). Quorum is computed using weighted

The Percona Live MySQL Conference and Expo begins next Monday and runs April 22-25, 2013. Attendees will see great keynotes from leaders in the industry including representatives from Oracle, Amazon Web Services, HP, Continuent, and Percona. They can also participate in thought provoking Birds of a Feather sessions on Tuesday night and the Wednesday night Community Networking Reception will be fun and entertaining with the presentation of the Community Awards and the Lightning Talks.

If you

Percona CEO Peter Zaitsev leads a track at Percona MySQL University in Raleigh, N.C. on Jan. 29, 2013.

Percona MySQL University, Toronto is taking place this Friday and I’m very excited about this event because it is a special opportunity to fit a phenomenal number of specific and focused MySQL technical talks all into one day, for free.

Over the course of the day we will cover some of the hottest topics in the MySQL space. There will be talks covering topics like MySQL 5.6, MySQL in the Cloud and High Availability for MySQL, as well as

I’m pleased to announce that Oracle is sending some of their top technical people to speak at the Percona Live MySQL Conference and Expo. The conference takes place April 22-25, 2013 at the Santa Clara Convention Center and Hyatt Santa Clara.

Tomas Ulin, VP, MySQL Engineering for Oracle, will present an invited keynote talk on “Driving MySQL Innovation” during the Tuesday morning opening keynotes. With the recent release of MySQL 5.6, conference attendees will hear about the latest developments of this

Accessing Percona XtraDB Cluster nodes in parallel with MySQL asynchronous calls

This post is followup to Peter’s recent post, “Investigating MySQL Replication Latency in Percona XtraDB Cluster,” in which a question was raised as to whether we can measure latency to all nodes at the same time. It is an interesting question: If we have N nodes, can we send queries to nodes to be executed in parallel?

To answer it, I decided to try a new asynchronous call to send a

Investigating MySQL Replication Latency in Percona XtraDB Cluster

I was curious to check how Percona XtraDB Cluster behaves when it comes to MySQL replication latency — or better yet, call it data propagation latency. It was interesting to see whenever I can get stale data reads from other cluster nodes after write performed to some specific node. To test it I wrote quite a simple script (you can find it in the end of the post) which connects to one node in the cluster, performs an update and then immediately does the read from second node. If the data has been

Percona is glad to announce the release of Percona XtraDB Cluster on January 30th, 2013. Binaries are available from downloads area or from our software repositories.

Bugs fixed:

• In some cases when node is recovered variable threads_running would become huge. Bug fixed #1040108 (Teemu Ollakka).
• Variable wsrep_defaults_file would be set up to the value in the last configuration file read. Bug fixed by keeping the value found in the top configuration file. Bug fixed

If you are using SElinux, you should know that it’s advised to disable it to avoid issue with PXC. Generally the communication between your nodes doesn’t work properly and a node having SElinux enabled won’t be able to join the cluster.

So when a node doesn’t join the cluster where it should, my first reflex is to have a look at audit.log. But recently I faced another problem: the node joined the cluster but SST failed (whatever which method was used, discarding skip).

I checked SElinux and it was of course disabled, then I add some debug information in the SST script but it seemed that the script was never launched. And this time the culprit is called : AppArmor !

Percona doesn’t provide any AppArmor profile for PXC, but it seems that on this server (Ubuntu TLS), a previous version of MySQL was

How easy is it to identify and debug Percona XtraDB Cluster replication problem ?

If you are using PXC, you may have already seen in your datadirectory several log files starting with GRA_

Those files correspond to a replication failure. That means the slave thread was not able to apply one transaction. For each of those file, a corresponding warning or error message is present in the mysql error log file.

Those error can also be false positive like a bad DDL statement (DROP a table that doesn’t exists for example) and therefore nothing to worry about. However it’s always recommended to understand what’s is happening.

As the GRA files contain binlog events in ROW format representing the failed transaction this post explains how to proceed.

The first step to be able to analyze your GRA files is to

The Web And PHP Magazine just published an article describing Percona XtraDB Cluster written by Liz van Dijk and Kenny Gryp. It is a pretty high-level introduction to the technology, so definitely give it a read if you’ve been wanting to give it a closer look.

Go to their website http://www.webandphp.com and download Issue number 9 for free!

On Monday I will deliver with Seppo Jaakola from Codership a tutorial on Percona XtraDB Cluster (Percona Server + Galera).

Already in London and you don’t know how to consume the hotel’s bandwidth ? If you attend this tutorial, you can download the Virtual Box appliance we will use during the session.

Two versions are available:

• PLUK 2012 Appliance (32-bit)
• PLUK 2012 Appliance (64-bit)

How to test them:

I learn more and more about Galera every day.  As I learn more, I try to keep my myq_gadgets toolkit up to date with what I consider is important to keep any eye on on a PXC node.  In that spirit, I just today pushed some changes to the ‘wsrep’ report, and I thought I’d go over some of the status variables and metrics being tracked there with the aim to show folks what they should be watching (at least from my opinion, this is subject to change!).

First, let’s take a look at the output:

[root@node3 ~]# myq_status -t 1 wsrep
Wsrep    Cluster        Node           Queue   Ops     Bytes     Flow        Conflct
    time  name P cnf

If you are interested by a great and revolutionary HA solution for MySQL or even if you already have some experience with PXC and/or Galera, I invite you to attend the tutorial I will deliver in Percona Live MySQL Conference London 2012 (PLUK2012), December 3rd : Percona XtraDB Cluster in a nutshell.

Small amount of theory and a lot of exercises and examples to really tame the beast that PXC is !

A Codership engineer will also be present and happy to help me answering your questions.

Note to attendees: please bring your

I have addressed previously how multi-node writing causes unexpected deadlocks in PXC, at least, it is unexpected unless you know how Galera replication works.  This is a complicated topic and I personally feel like I’m only just starting to wrap my head around it.

The short of it is that Galera replication is not a doing a full 2-phase commit to replicate (where all nodes commit synchronously with the client), but it is instead something Codership calls “virtually synchronous” replication.  This could either be clever marketing or clever engineering, at least at face value.   However, I believe it really is clever engineering

The Percona Toolkit team is happy to announce the release of Percona Toolkit version 2.1.6. This is the sixth stable release in the 2.1 series, and primarily a bug-fix release; We suggest that users upgrade to the latest version of the tools.

The complete list of changes is on the Launchpad milestone for 2.1.6, but here are some highlights the release:

• pt-online-schema-change can now handle column renames
• pt-online-schema-change’s DELETE trigger no longer fails when altering a primary key
• pt-stalk got a plugin hook interface
• pt-show-grant support for column-level grants
• pt-table-checksum will warn if slaves have a differing binlog_format

Remember that if you need help with Percona Toolkit, we provide flat-fee, unlimited

Thanks for all, who attended my webinar, I got many questions and I wanted to take this opportunity to answer them.

Q: Even ntp has a delay of 0.3-0.4 between servers does that mean a 0.25 as from logs can be an issue ?
A: My demo vms were running for a few hours before the webinar in my local virtualbox instance, and I tried to show the minimal installation required for XtraDB cluster. Unfortunately enough, I didn’t include NTP, which caused SSTs to fail with the tar stream being from the future. The 0.3 – 0.4 second delay seems too much to me. According to the NTP standard, if the delay between the hosts is greater

The Percona Toolkit team is happy to announce the release of Percona Toolkit version 2.1.4. This is the fifth stable release in the 2.1 series, and primarily a bug-fix release; We suggest that users upgrade to the latest version of the tools.

The complete list of changes is on the Launchpad milestone for 2.1.4, but here are some highlights the release:

• pt-table-checksum now works with Percona XtraDB Cluster
• The “Version Check” feature, explained at length here.
• –defaults-file is now used when connecting to discovered slaves in pt-table-checksum

All in all, a solid bug-fix release, with the addition of some new

I’ve mentioned this before, but I’m delivering a tutorial on Percona XtraDB Cluster / Galera at Percona Live NY 2012 right in midtown-Manhattan on October 1st.

I really want this tutorial to be hands-on, not just a 3 hour lecture, so you can actually see how a cluster operates and interact with it.  So, I have come up with a set of walkthroughs for different features and areas of learning PXC using a common set of Virtual Machines setup with Vagrant and Puppet (which works on Linux, OSX, or Windows).  This sounds complicated, but it basically means we can create a test environment

Some people want to use a backup to prepare a new Percona XtraDB Cluster node. They want this to avoid State Snapshot Transfer that could slow down the donor (depending of the SST method you are using, the donor can be blocked. I will cover this in a future blog post). As backup are generally performed during non-peak time, the effect should be reduced, and this avoid the need of performing 2 backups: the usual backup and the SST).

So to be able to use a backup for this purpose, we have 3 prerequisites:

• use XtraBackup >= 2.0.1
• the backup needs to be performed with –galera-info (option for innobackupex)
• have a gcache big enough to store all the changes from the time of the backup until the restore to be able to perform the

During the design period of a new cluster, it is always advised to have at least 3 nodes (this is the case with PXC but it’s also the same with PRM). But why and what are the risks ?

The goal of having more than 2 nodes, in fact an odd number is recommended in that kind of clusters, is to avoid split-brain situation. This can occur when quorum (that can be simplified as “majority vote”) is not honoured. A split-brain is a state in which the nodes lose contact with one another and then both try to take control of shared resources or provide simultaneously the cluster service.
On PRM the problem is obvious, both nodes will try to run the master and slave IPS and will accept writes. But what could happen with Galera replication on PXC ?

Ok first let’s have a look with a standard PXC setup (no special

This post is a step-by-step guide to set up Percona XtraDB Cluster (PXC) in a virtualized test sandbox. I used Amazon EC2 micro instances, but the content here is applicable for any kind of virtualization technology (for example VirtualBox). The goal is to give step by step instructions, so the setup process is understandable and it could serve as a reference architecture.

You will need 4 virtual machines. 3 for PXC and 1 for the client, which will have HaProxy. I used CentOS 6 as the operating system, the instructions are similar for any Linux distribution.

The client node will have HaProxy installed and it will redirect requests to PXC nodes. This approach works well in real-world scenarios too. Running HaProxy on the application servers instead of having them as dedicated entities gives you benefits like you don’t need

I have a customer who is considering Percona XtraDB Cluster (PXC) in a two colo WAN environment.  They wanted me to do a test comparing PXC against semi-synchronous replication to see how they stack up against each other.

The test environment included AWS EC2 nodes in US-East and US-West (Oregon).  The ping RTT latency between these nodes was right around 100ms.

All environments used Percona Server or Percona XtraDB Cluster server 5.5.24.  Innodb durability was disabled for all tests (innodb_flush_log_at_trx_commit=2).  All other settings were the same kept the same unless otherwise noted.

I tested against the following

Many of you heard of this nasty security vulnerability in MySQL, and as we are getting a lot of inquiries how does it affect Percona Server, I decided to address it in this post.

• The issue exists in the source code of MySQL 5.5.23 or earlier and MySQL 5.1.62 or earlier. The same is true for Percona Server, as we share the same code base.
• However binaries provided by Percona do not have this problem, as in our build process we do not use sse-optimized glibc memcmp. This is true for any version of Percona and all tar.gz, RPM and DEB packages. Once again, if you use binary builds, provided by Percona from our official download area, or from our repositories,

I am happy to announce next version Percona XtraDB Cluster 5.5.24, which is mostly jump to the current version of Percona Server.

It includes an important bug fix:

• table with no PK but still having UK, can cause crash in slave using PA

Binaries are available from downloads area or from our repositories.

For this release we provide binaries for Ubuntu 12.04.

If you want to know more how to migrate to XtraDB Cluster, you can check our recent webinar Migrating To Percona XtraDB Cluster.

This is an General Availability release. We did our best to eliminate bugs and problems

Thanks to all who attended my webinar today. The session was recorded and will be available to watch for free soon here.

There were a lot of great questions asked during the session, so I’d like to take this opportunity to try to answer a few of them:

Q: Is there an easy way to leverage the xtrabackup SST and IST in an xtradb cluster to take your full and incremental backups of the cluster’s databases?

Well, if you noticed, all the current SST methods are just commonly used backup tools, and the SST is, in reality, just doing a full backup. So I’m not sure there’s much to leverage with SST.

The straight-forward to way to backup XtraDB Cluster (IMHO) would

Whenever I learn about a new technology, I typically want to know the major administrative touch points more than I want to know exhaustive detail about every configuration option.  Give me the gist, show me enough to get started, and give me a link to the manual.

XtraDB cluster (and Galera, the technology on which it is based) has been attracting a lot of interest in the community and we want to start presenting information about both what we know is essential, but also what we suspect will become essential as more and more production experience is logged with these important new technologies.

As such, I put together a baseline talk for getting you into a working knowledge of XtraDB

Our previous GA release of Percona XtraDB Cluster caused a lot of interest and feedback. I am happy to announce next version Percona XtraDB Cluster 5.5.23, which comes with bug fixes and improvements.

List of changes:

• Fixes merged from upstream (Codership-mysql)
• Support for MyISAM, now changes to MyISAM tables are replicated to other nodes
• Improvements to XtraBackup SST methods, better error handling
• New SST wsrep_sst_method=skip, useful when you start all nodes from the same sources (i.e. backup)
• Ability to pass list of IP addresses for a new node, it will connect to the first available

Binaries are available from downloads area or from