8PM. One of the servers found a critical bug. Hop online and discuss, log bug. 10PM. Patch ready. 10:30PM. New build ready. 10:45PM. New RQG run initiated. This was by no means an uncommon sight during the months of testing that went into Percona Server 5.6, in fact it was commonplace.
At a certain point, we had 3 very high end servers (modern cpu’s, heaps of cores and memory), all equipped with either fast SSD’s or Fusion-io flash storage, executing thousands of trials, 8 in parallel per server, each executing 1 to 25 mysql threads per running mysqld instance.
And that was just the final months of testing. Before that much work was done on finding “every last bug out there”. We discovered many bugs in both upstream (Oracle’s MySQL 5.6) and in Percona Server 5.6. I personally logged around 100 [Read more...]