8PM. One of the servers found a critical bug. Hop online and
discuss, log bug. 10PM. Patch ready. 10:30PM. New build ready.
10:45PM. New RQG run initiated. This was by no means an uncommon
sight during the months of testing that went into Percona Server
5.6, in fact it was commonplace.
At a certain point, we had 3 very high end servers (modern cpu’s,
heaps of cores and memory), all equipped with either fast SSD’s
or Fusion-io flash storage, executing thousands of trials, 8 in
parallel per server, each executing 1 to 25 mysql threads per
running mysqld instance.
And that was just the final months of testing. Before that much
work was done on finding “every last bug out there”. We
discovered many bugs in both upstream (Oracle’s MySQL 5.6) and in
Percona Server 5.6. I personally logged around 100 bugs, but the
total count would be much higher still.
My colleague …
[Read more]