8PM. One of the servers found a critical bug. Hop online and discuss, log bug. 10PM. Patch ready. 10:30PM. New build ready. 10:45PM. New RQG run initiated. This was by no means an uncommon sight during the months of testing that went into Percona Server 5.6, in fact it was commonplace.

At a certain point, we had 3 very high end servers (modern cpu’s, heaps of cores and memory), all equipped with either fast SSD’s or Fusion-io flash storage, executing thousands of trials, 8 in parallel per server, each executing 1 to 25 mysql threads per running mysqld instance.

And that was just the final months of testing. Before that much work was done on finding “every last bug out there”. We discovered many bugs in both upstream (Oracle’s MySQL 5.6) and in Percona Server 5.6. I personally logged around 100 bugs, but the total count would be much higher still.

My colleague Laurynas (lead of Percona Server) stated at some point during the testing that Percona Server 5.6 is the most qualitative release we have ever made. I agree wholeheartedly with him, and would add to it that we have also included a long list of upstream bugs present in Oracle’s MySQL 5.6.

During the many months of testing, we would have executed around 15000-20000 individual trial runs in RQG (start mysqld, test, stop mysqld), if not more, and each trial lasting around 5 minutes.

So what is the world premiere mentioned in the title all about you may wonder? During the testing, it became clear (and paramount) that testing the ever growing number of options in both upstream and Percona Server could not proceed as it had in the past.

With 35+ options, each of those to be tested in 2-way and 3-way combinations, and each of those being having multiple valid value settings, the number of combinations quickly became dazzling. “2 Years for testing” is just not an option.

The solution came by starting to use advanced option combinatorics, also called “pairwise testing”. For example, if you test abc (where a,b,c are options), and you are testing 2-way combinations, any other combinations with “bc” or “ab” or “ac” could potentially be excluded from the stream.

For the full article and information on how to get into combinatorics, see here. As far as I know, this was the first time this technique was used (worldwide) for mysqld option testing! It was exhilarating to see that instead of thousands of trials, we eventually only needed 133 trials across 3 build types.

Sidenote: When testing using the RQG, we usually focus on 3 types of Percona Server builds: optimized (the “production” binary), debug (with debug instrumentation activated, enabling us to check more developer asserts etc.), and finally Valgrind (with valgrind instrumentation, enabling us to see if developers made mistakes like forgetting to release memory, etc.)

Besides this extensive RQG testing, Percona relies on a huge Jenkins farm to do much of it’s automated regression and performance check testing. For each code push, we execute thousands of tests across a myriad of OS platforms. Each day, we also do an automated performance sanity check to ensure that server performance did not suddenly drop due to an inadvertent change. Finally, patches to the server are reviewed by two developers.

As you can see, when you use Percona Server 5.6, you can sleep at night, knowing that your database is running on software tested by one of the most advanced testing techniques in use today, as well as having been evaluated by top-notch testing frameworks, and people who care about the quality of their product!

The post How Percona tested Percona Server 5.6: A world premiere in advanced testing appeared first on MySQL Performance Blog.