Planet MySQL

Holland Backup Manager

MySQLBoy — Fri, 24 May 2013 23:07:00 +0000

Part 1 - Installing Holland Backup Manager

I spoke at Percona Live Conference and Expo 2013 about backups. Part of the talk focussed on the backup products in the ecosystem that will help you make a backup of your MySQL data. This follow-up article touches on one of the frameworks from my talk, the Holland Backup Manager. I was able to have a chat with some of the guys on the Rackspace booth about Holland and had some questions regarding features answered.

Holland is a backup framework focussing mostly on MySQL backups but it is pluggable so you can write add backup providers to extend it to your own needs. Using the framework you're able to configure and deploy backup jobs of varying scope to multiple machines. The framework which was originally developed at Rackspace, currently supports mysqldump, lvm, xtrabackup and pgdump (Postgres) also sqllite. The latest version is number 1.0.8 which brings a few nice new features with it. I'll talk about the most significant of these in the third of the series.
I want to show some installation, configuration and then use of the framework. I'm a fan of this project and if you've got the need to standardise whilst being flexible then Holland could be the solution you've been looking for.

Firstly, head over to github to grab the files. https://github.com/holland-backup/holland
I've been using vagrant to provide a quick VM environment. This OS I'm using here is a CentOS 6.4 64bit host running in Virtualbox. All dependencies for Holland should be available on the popular package managers but if you're using something other then debian (apt-get) or redhat based (yum) then you might require some research elsewhere to meet the requirements of Holland.

Download the zip archive from github (the extension was lost in wget but it is a .zip file)

[vagrant@node1 ~]$ wget https://github.com/holland-backup/holland/archive/master.zip ... Saving to: `master'  100%[==================================================>] 602,186      572K/s   in 1.0s (572 KB/s) - `master' saved [602186/602186]

So then proceed to unzip your archive

[vagrant@node1 ~]$ unzip ./master 
Archive:  ./master 
832eaa6a0fff56a3009cdde22eda4746203c97f6 
   creating: holland-master/
  inflating: holland-master/CHANGES.txt
  inflating: holland-master/INSTALL
  inflating: holland-master/LICENSE
  inflating: holland-master/README
   creating: holland-master/config/
  inflating: holland-master/config/README
...

ProTip: Take some time to go through the various Holland directories. Throughout the directories there README and INSTALL files that will explain more about the accompanying files.

[vagrant@node1 holland-master]$ ls -l
total 56
-rw-rw-r--  1 vagrant vagrant 7064 May 13 17:45 CHANGES.txt
drwxrwxr-x  4 vagrant vagrant 4096 May 13 17:45 config
drwxrwxr-x  3 vagrant vagrant 4096 May 13 17:45 contrib
drwxrwxr-x  4 vagrant vagrant 4096 May 13 17:45 docs
drwxrwxr-x  7 vagrant vagrant 4096 May 13 17:45 holland
-rw-rw-r--  1 vagrant vagrant 1782 May 13 17:45 INSTALL
-rw-rw-r--  1 vagrant vagrant 3290 May 13 17:45 LICENSE
drwxrwxr-x 13 vagrant vagrant 4096 May 13 17:45 plugins
-rw-rw-r--  1 vagrant vagrant  514 May 13 17:45 README
-rw-rw-r--  1 vagrant vagrant   62 May 13 17:45 setup.cfg
-rw-rw-r--  1 vagrant vagrant 1441 May 13 17:45 setup.py
drwxrwxr-x  2 vagrant vagrant 4096 May 13 17:45 tests
-rwxr-xr-x  1 vagrant vagrant 1782 May 13 17:45 tomsay.py

Before going any further lets take care of the dependencies. They're easily met using yum/apt or if you want to compile them yourself then go right ahead;
- python-setuptools
- mysqldb (python's mysql connector)

Now, lets take care of some directories that need to exists. These are configurable later, but for simplicity sake we'll roll forward with the Holland defaults. Your environment is likely to be different especially if you're mounting remote storage to store the backup files you make.

[vagrant@node1 config]$ sudo mkdir -p /etc/holland [vagrant@node1 plugins]$ sudo mkdir -p /var/log/holland [vagrant@node1 plugins]$ sudo mkdir -p /var/spool/holland

Now the directories we've created will only be writable to root since we needed elevated privileges to make them. You might choose to chmod the directories if you're going to use a specific backup user to run your backup jobs. Now we're ready to run the python setup file from the root of the unarchived directory

[vagrant@node1 holland-master]$ sudo python ./setup.py install
running install
running bdist_egg
running egg_info
creating holland.egg-info
writing holland.egg-info/PKG-INFO
...
Installing holland script to /usr/bin

Installed /usr/lib/python2.6/site-packages/holland-1.0.8-py2.6.egg
Processing dependencies for holland==1.0.8
Finished processing dependencies for holland==1.0.8

Without any errors in the previous output it's safe to assume we're installed.

[vagrant@node1 config]$ holland --version
Holland Backup v1.0.8
Copyright (c) 2008-2010 Rackspace US, Inc.
More info available at http://hollandbackup.org

Lets review what is in the plugin directory. These are the providers. Some are core libraries for the use of particular providers. There's also an example and random directory which we will reference in the next article on config.


[vagrant@node1 plugins]$ ls -w1
ACTIVE
holland.backup.example
holland.backup.mysqldump
holland.backup.mysqlhotcopy
holland.backup.mysql_lvm
holland.backup.pgdump
holland.backup.random
holland.backup.sqlite
holland.backup.xtrabackup
holland.lib.common
holland.lib.lvm
holland.lib.mysql
README

There are a couple of essential installs we need to take care of before we're ready to tackle the plugins for the backup tools. These are the holland.lib.common & holland.lib.mysql. For each you need to run the setup.py file from within the directories. The output should look something like the following.


[vagrant@node1]$ cd holland.lib.common/
[vagrant@node1 holland.lib.common]$ sudo python setup.py install
running install
...
Adding holland.lib.common 1.0.8dev to easy-install.pth file

Installed /usr/lib/python2.6/site-packages/holland.lib.common-1.0.8dev-py2.6.egg
Processing dependencies for holland.lib.common==1.0.8dev
Finished processing dependencies for holland.lib.common==1.0.8dev

[vagrant@node1 holland.lib.common]$ cd ../holland.lib.mysql/
[vagrant@node1 holland.lib.mysql]$ sudo python setup.py install
running install
...
Adding holland.lib.mysql 1.0.8dev to easy-install.pth file

Installed /usr/lib/python2.6/site-packages/holland.lib.mysql-1.0.8dev-py2.6.egg
Processing dependencies for holland.lib.mysql==1.0.8dev
Finished processing dependencies for holland.lib.mysql==1.0.8dev

Now, I'm mainly interested in mysqldump and xtrabackup but you might want to use the mysql_lvm. I don't think there are many takers for mysqlhotcopy these days. The same steps apply for each provider you want to install on each machine.

[vagrant@node1 holland.backup.mysqldump]$ sudo python setup.py install running install ... Adding holland.backup.mysqldump 1.0.8dev to easy-install.pth file  Installed /usr/lib/python2.6/site-packages/holland.backup.mysqldump-1.0.8dev-py2.6.egg Processing dependencies for holland.backup.mysqldump==1.0.8dev Finished processing dependencies for holland.backup.mysqldump==1.0.8dev

and then finally from the xtrabackup directory;

[vagrant@node1 plugins]$ cd holland.backup.xtrabackup/ [vagrant@node1 holland.backup.xtrabackup]$ sudo python setup.py install running install ... Adding holland.backup.xtrabackup 1.0.8dev to easy-install.pth file  Installed /usr/lib/python2.6/site-packages/holland.backup.xtrabackup-1.0.8dev-py2.6.egg Processing dependencies for holland.backup.xtrabackup==1.0.8dev Finished processing dependencies for holland.backup.xtrabackup==1.0.8dev

So in summary it's the library files for the plugins;
- holland.lib.common
- holland.lib.mysql
- holland.lib.lvm (depending if you're going to use it)

In part two I'll be giving an overview of the config files and then part three will be some cookbook-like config for the various providers.
PlanetMySQL Voting: Vote UP / Vote DOWN

Percona Server for MySQL 5.5.31-30.3 now available

MySQL Performance Blog — Fri, 24 May 2013 20:24:59 +0000

Percona Server for MySQL version 5.5.31-30.3

Percona is glad to announce the release of Percona Server for MySQL 5.5.31-30.3 on May 24, 2013 (Downloads are available here and from the Percona Software Repositories). Based on MySQL 5.5.31, including all the bug fixes in it, Percona Server 5.5.31-30.3 is now the current stable release in the 5.5 series. All of Percona‘s software is open-source and free, all the details of the release can be found in the 5.5.31-30.3 milestone at Launchpad.

New Features:

Percona Server has ported the Atomic write support for Fusion-io devices patch from MariaDB. This feature adds atomic write support for directFS filesystem on Fusion-io devices. This feature implementation is considered BETA quality.
Percona Server has introduced innodb_read_views_memory and innodb_descriptors_memory status variables in the Extended Show Engine InnoDB Status to improve InnoDB memory usage diagnostics.

Bugs Fixed:

Fix for bug #1131187 introduced a regression that could cause a memory leak if query cache was used together with InnoDB. Bug fixed #1170103.
Fixed the RPM packaging regression that was introduced with the fix for bug #710799. This regression caused mysql schema to be missing after the clean RPM installation. Bug fixed #1174426.
Fixed the Percona-Server-shared-55 and Percona-XtraDB-Cluster-shared RPM package dependences. Bug fixed #1050654.
Fixed the upstream bug #68999 which caused compiling Percona Server to fail on CentOS 5 and Debian squeeze due to older OpenSSL version. Bug fixed #1183610.
If a slave was running with its binary log enabled and then restarted with the binary log disabled, Crash-Resistant Replication could overwrite the relay log info log with an incorrect position. Bug fixed #1092593.
Fixed the CVE-2012-5615 vulnerability. This vulnerability would allow remote attacker to detect what user accounts exist on the server. This bug fix comes originally from MariaDB (see MDEV-3909). Bug fixed #1171941.
Fixed the CVE-2012-5627 vulnerability, where an unprivileged MySQL account owner could perform brute-force password guessing attack on other accounts efficiently. This bug fix comes originally from MariaDB (see MDEV-3915). Bug fixed #1172090.
mysql_set_permission was failing on Debian due to missing libdbd-mysql-perl package. Fixed by adding the package dependency. Bug fixed #1003776.
Rebuilding Debian source package would fail because dpatch and automake were missing from build-dep. Bug fixed #1023575 (Stephan Adig).
Backported the fix for the upstream bug #65077 from the MySQL 5.6 version, which removed MyISAM internal temporary table mutex contention. Bug fixed #1179978.

Release notes for Percona Server for MySQL 5.5.31-30.3 are available in our online documentation. Bugs can be reported on the launchpad bug tracker.

The post Percona Server for MySQL 5.5.31-30.3 now available appeared first on MySQL Performance Blog.

PlanetMySQL Voting: Vote UP / Vote DOWN

Tungsten University: Load a Vertica data warehouse with MySQL data

Petri Virsunen — Fri, 24 May 2013 19:32:00 +0000

Continuent Tungsten offers real-time replication from MySQL to a variety of DBMS types including Vertica. In this Tungsten University webcast we will show you the details of setting up MySQL-to-Vertica replication, including the following topics: Introduction to Continuent Tungsten features for data warehouse loading Installation for MySQL to Vertica replication Best practices for applications:
PlanetMySQL Voting: Vote UP / Vote DOWN

Log Buffer #321, A Carnival of the Vanities for DBAs

The Pythian Group — Fri, 24 May 2013 12:34:54 +0000

Can’t find what you want or need? Do you have a tip or trick to share? Do you want to lament over a technical woe? If yes then blog and send it to us for the Log Buffer :), just like this Log Buffer contains tips, tricks and woes.
Oracle:

Fahd Mirza and Tanel Poder throw spotlight on the v$cell_thread_history view with respect to the Exadata.

Mark W. Farnham‘s rightsizing goes on with a roar, as he declares that he is pretty much a green sneaker, tree hugging conservationist.

I would like to use the Microsoft Query tool in Microsoft Excel to extract records from the ERP database. Charles Hooper answers.

Kevin continues with yet another post about Exadata. The time he discusses: Oracle Exadata Database Machine: Proving 160 Xeon E7 Cores Are As “Slow” As 128 Xeon E5 Cores?

Alberto does a post which is a brief discussion about the advantages of activating parallelism by altering the session environment instead of using the alternative ways (hints, DDL).

SQL Server:

Security Questions: Differences between “Grant” and “With Grant?”

Using Scoped Assignments To Show Calendar and Financial Year-To-Date Values In The Same Calculated Measure.

A cool post about Business Objects on Linux and SQL Server.

Is Your Code an English Garden or Ikebana?

The Journey is the Reward: Speaking for SQLPass, where do you start?

Resolving Error Connecting Report Builder to a PowerPivot Data Source.

MySQL:

Stewart Smith is sharing an old note on the Storage Engine API.

A good look at the Non-Deterministic Query in Replication Stream.

The MariaDB project is pleased to announce the immediate availability of the MariaDB 5.5.31.

Fernando had recently had to do some customer work involving the McAfee MySQL Audit Plugin.

There’s a big difference in how plugins are treated in MySQL and how they are treated in Drizzle.

PlanetMySQL Voting: Vote UP / Vote DOWN

ZFS on Linux and MySQL

Yves Trudeau — Fri, 24 May 2013 10:00:12 +0000

I am currently working with a large customer and I am involved with servers located in two data centers, one with Solaris servers and the other one with Linux servers. The Solaris side is cleverly setup using zones and ZFS and this provides a very low virtualization overhead. I learned quite a lot about these technologies while looking at this, thanks to Corey Mosher.

On the Linux side, we recently deployed a pair on servers for backup purpose, boxes with 64 300GB SAS drives, 3 raid controllers and 192GB of RAM. These servers will run a few slave instances each of production database servers and will perform the backups. The write load is not excessive so a single server can easily handle the write load of all the MySQL instances. The original idea was to configure them with raid-10 + LVM, making sure to stripe the LV when we need to and align the partition correctly.

We got decent tpcc performance, nearly 37k NoTPM using 5.6.11 and xfs. Then, since ZFS on Linux is available and there is in house ZFS knowledge, we decided to reconfigure one of the server and give ZFS a try. So I trashed the raid-10 arrays, configure JBODs and gave all those drives to ZFS (30 mirrors + spares + OS partition mirror) and I limited the ARC size to 4GB. I don’t want to start a war but ZFS performance level was less than half of xfs for the tpcc test and that’s maybe just normal. We didn’t try too hard to get better performance because we already had more than enough for our purpose and some ZFS features are just too useful for backups (most apply also for btrfs). Let’s review them.

Snapshots

ZFS does snapshot, like LVM but… since it is a copy on write filesystem, the snapshots are free, no performance penalty. You can easily run a server with hundreds of snapshots. With LVM, your IO performance drops to 33% after the first snapshot so keeping a large number of snapshots running is simply not an option. With ZFS you can easily have:

one snapshot per day for the last 30 days
one snapshot per hour for the last 2 days
one snapshot per 5min for the last 2 hours

and that will be perfectly fine. Since starting a snapshot take less than a second, you could even be more zealous. Pretty interesting to speed up point in time recovery when you dataset is 700GB. If you google a bit with “zfs snapshot script” you’ll many scripts ready for the task. Snapshots work best with InnoDB, with MyISAM you’ll have to start the snapshot while holding a “flush tables with read lock” and the flush operation will take some time to complete.

Compression

ZFS can compress data on the fly and it is surprisingly cheap. In fact the best tpcc results I got were when using compression. I still have to explain this, maybe it is related to better raid controller write cache use. Even the fairly slow gzip-1 mode works well. The tpcc database, which contains a lot of random data that doesn’t compress well showed a compression ration of 1.70 with gzip-1. Real data will compress much more. That gives us much more disk space than we expected so even more snapshots!

Integrity

With ZFS each record on disk has a checksum. If a cosmic ray flip a bit on a drive, instead of crashing InnoDB, it will be caught by ZFS and the data will be read from the other drive in the mirror.

Better availability and disk usage

On purpose, I allocated mirror pairs using drives from different controllers. That way, if a controller dies, the storage will still be working. Also, instead of having 1 or 2 spare drives per controller, I have 2 for the whole setup. A small but yet interesting saving.

All put together, ZFS on Linux is a very interesting solution for MySQL backup servers. All backup solutions have an impact on performance with ZFS the impact is up front and the backups are almost free.

The post ZFS on Linux and MySQL appeared first on MySQL Performance Blog.

PlanetMySQL Voting: Vote UP / Vote DOWN

MySQL Applier For Hadoop: Implementation

Shubhangi Garg — Fri, 24 May 2013 06:47:47 +0000

This is a follow up post, describing the implementation details of Hadoop Applier, and steps to configure and install it. Hadoop Applier integrates MySQL with Hadoop providing the real-time replication of INSERTs to HDFS, and hence can be consumed by the data stores working on top of Hadoop. You can know more about the design rationale and per-requisites in the previous post.

Design and Implementation:

Hadoop Applier replicates rows inserted into a table in MySQL to the Hadoop Distributed File System(HDFS). It uses an API provided by libhdfs, a C library to manipulate files in HDFS.

The library comes pre-compiled with Hadoop distributions.

It connects to the MySQL master (or read a binary log generated by MySQL) and:

fetches the row insert events occurring on the master
decodes these events, extracts data inserted into each field of the row
uses content handlers to get it in the format required and appends it to a text file in HDFS.

Schema equivalence is a simple mapping:

Databases are mapped as separate directories, with tables in them as sub-directories. Data inserted into each table is written into text files (named as datafile1.txt) in HDFS. Data can be in comma separated format; or any other delimiter can be used, that is configurable by command line arguments.

The diagram explains the mapping between MySQL and HDFS schema.

The file in which the data is stored is named datafile1.txt here; you can name is anything you want. The working directory where this datafile goes is base_dir/db_name.db/tb_name.

The timestamp at which the event occurs is included as the first field in each row inserted in the text file.

The implementation follows these steps:

- Connect to the MySQL master using the interfaces to the binary log

#include “binlog_api.h”

Binary_log binlog(create_transport(mysql_uri.c_str()));

binlog.connect();

- Register content handlers

/*
Table_index is a sub class of Content_handler class in the Binlog API
*/

Table_index table_event_hdlr;

Applier replay_hndlr(&table_event_hdlr, &sqltohdfs_obj);

binlog.content_handler_pipeline()->push_back(&table_event_hdlr);

binlog.content_handler_pipeline()->push_back(&replay_hndlr);

- Start an event loop and wait for the events to occur on the master

while (true)

{

Pull events from the master. This is the heart beat of the event listener.

Binary_log_event *event;

binlog.wait_for_next_event(&event);

}

- Decode the event using the content handler interfaces

class Applier : public mysql::Content_handler

{

public:

Applier(Table_index *index, HDFSSchema *mysqltohdfs_obj)

{

m_table_index= index;

m_hdfs_schema= mysqltohdfs_obj;

}

mysql::Binary_log_event *process_event(mysql::Row_event *rev)

{

int table_id= rev->table_id;

typedef std::map Int2event_map;

int2event_map::iterator ti_it= m_table_index->find(table_id);

- Each row event contains multiple rows and fields.
Iterate one row at a time using Row_iterator.

mysql::Row_event_set rows(rev, ti_it->second);

mysql::Row_event_set::iterator it= rows.begin();

{

mysql::Row_of_fields fields= *it;

long int timestamp= rev->header()->timestamp;

if (rev->get_event_type() == mysql::WRITE_ROWS_EVENT)

table_insert(db_name, table_name, fields, timestamp, m_hdfs_schema);

} while (++it != rows.end());

- Get the field data separated by field delimiters and row delimiters.

Each row contains a vector of Value objects. The converter allows us to transform the value into another representation.

mysql::Row_of_fields::const_iterator field_it= fields.begin();

mysql::Converter converter;

std::ostringstream data;

data << timestamp;

{

field_index_counter++;

std::vector::iterator it;

std::string str;

converter.to(str, *field_it);

data << sqltohdfs_obj->hdfs_field_delim;

data << str;

} while (++field_it != fields.end());

data << sqltohdfs_obj->hdfs_row_delim;

- Connect to the HDFS file system.

If not provided, the connection information (user name, password host and port) are read from the XML configuration file, hadoop-site.xml.

HdfsFS m_fs= hdfsConnect(host.c_str(), port);

- Create the directory structure in HDFS.

Set the working directory and open the file in append mode.

hdfsSetWorkingDirectory(m_fs, (stream_dir_path.str()).c_str());

const char* write_path= "datafile1.txt";

hdfsFile writeFile;

- Append data at the end of the file.

writeFile= hdfsOpenFile(m_fs, write_path, O_WRONLY|O_APPEND, 0, 0, 0);

tSize num_written_bytes = hdfsWrite(m_fs, writeFile, (void*)data, strlen(data));

Install and Configure:
Follow these steps to install and run the Applier:

1. Download a Hadoop release (I am using 1.0.4); configure and install (for the purpose of the demo, install it in pseudo distributed mode). Flag 'dfs.support.append'must be set to true while configuring HDFS(hdfs-site.xml). Since append is not supported in Hadoop 1.x, set the flag 'dfs.support.broken.append' to true.

2. Set the environment variable $HADOOP_HOME to point to the Hadoop installation directory.

3. CMake doesn't come with a 'find' module for libhdfs. Ensure that the 'FindHDFS.cmake' is in the CMAKE_MODULE_PATH. You can download a copy here.

4. Edit the file 'FindHDFS.cmake', if necessary, to have HDFS_LIB_PATHS set as a path to libhdfs.so, and HDFS_INCLUDE_DIRS have the path pointing to the location of hdfs.h.

For 1.x versions, library path is $ENV{HADOOP_HOME}/c++/Linux-i386-32/lib , and header files are contained in $ENV{HADOOP_HOME}/src/c++/libhdfs. For 2.x releases, header files and libraries can be found in $ENV{HADOOP_HOME}/lib/native, and $ENV{HADOOP_HOME}/include respectively.

For versions 1.x, this patch will fix the paths:

  --- a/cmake_modules/FindHDFS.cmake
  +++ b/cmake_modules/FindHDFS.cmake
  @@ -11,6 +11,7 @@ exec_program(hadoop ARGS version OUTPUT_VARIABLE
Hadoop_VERSION
   # currently only looking in HADOOP_HOME
   find_path(HDFS_INCLUDE_DIR hdfs.h PATHS
     $ENV{HADOOP_HOME}/include/
  +  $ENV{HADOOP_HOME}/src/c++/libhdfs/
     # make sure we don't accidentally pick up a different version
     NO_DEFAULT_PATH
   )
  @@ -26,9 +27,9 @@ endif()
   message(STATUS "Architecture: ${arch_hint}")
  
   if ("${arch_hint}" STREQUAL "x64")
  -  set(HDFS_LIB_PATHS $ENV{HADOOP_HOME}/lib/native)
  +  set(HDFS_LIB_PATHS $ENV{HADOOP_HOME}/c++/Linux-amd64-64/lib)
   else ()
  -  set(HDFS_LIB_PATHS $ENV{HADOOP_HOME}/lib/native)
  +  set(HDFS_LIB_PATHS $ENV{HADOOP_HOME}/c++/Linux-i386-32/lib)
   endif ()
  
   message(STATUS "HDFS_LIB_PATHS: ${HDFS_LIB_PATHS}")

5.Since libhdfs is JNI based API, it requires JNI header files and libraries to build. If there exists a module FindJNI.cmake in the CMAKE_MODULE_PATH and JAVA_HOME is set; the headers will be included, and the libraries would be linked to. If not, it will be required to include the headers and load the libraries separately (modify LD_LIBRARY_PATH).

6. Build and install the library 'libreplication', to be used by Hadoop Applier,using CMake.

Download a copy of Hadoop Applier from http://labs.mysql.com.
'mysqlclient' library is required to be installed in the default library paths. You can either download and install it (you can get a copy here), or set the environment variable $MYSQL_DIR to point to the parent directory of MySQL source code. Make sure to run cmake on MySQL source directory.
$export MYSQL_DIR=/usr/local/mysql-5.6

Run the 'cmake' command on the parent directory of the Hadoop Applier source. This will generate the necessary Makefiles. Make sure to set cmake option ENABLE_DOWNLOADS=1; which will install Google Test required to run the unit tests.
$cmake . -DENABLE_DOWNLOADS=1
Run 'make' and 'make install' to build and install. This will install the library 'libreplication' which is to be used by Hadoop Applier.

7. Make sure to set the CLASSPATH to all the hadoop jars needed to run Hadoop itself.

$export PATH=$HADOOP_HOME/bin:$PATH

$export CLASSPATH=$(hadoop classpath)

8. The code for Hadoop Applier can be found in /examples/mysql2hdfs, in the Hadoop Applier repository. To compile, you can simply load the libraries (modify LD_LIBRARY_PATH if required), and run the command “make happlier” on your terminal. This will create an executable file in the mysql2hdfs directory.

.. and then you are done!

Now run hadoop dfs (namenode and datanode), start a MySQL server as master with row based replication (you can use mtr rpl suite for testing purposes : $MySQL-5.6/mysql-test$./mtr --start --suite=rpl --mysqld=--binlog_format='ROW' --mysqld=--binlog_checksum=NONE), start hive (optional) and run the executable ./happlier, optionally providing MySQL and HDFS uri's and other available command line options. (./happlier –help for more info).

There are useful filters as command line options to the Hadoop applier.

Options	Use
-r, --field-delimiter=DELIM Use DELIM instead of ctrl-A for field delimiter. DELIM can be a string or an ASCII value in the format '\nnn' . Escape sequences are not allowed.	Provide the string by which the fields in a row will be separated. By default, it is set to ctrl-A
-w, --row-delimiter=DELIM Use DELIM instead of LINE FEED for row delimiter . DELIM can be a string or an ASCII value in the format '\nnn' Escape sequences are not allowed.	Provide the string by which the rows of a table will be separated. By default, it is set to LINE FEED (\n)
-d, --databases=DB_LIST DB-LIST is made up of one database name, or many names separated by commas. Each database name can be optionally followed by table names. The table names must follow the database name, separated by HYPHENS Example: -d=db_name1-table1-table2,dbname2-table1,dbname3	Import entries for some databases, optionally include only specified tables.
-f, --fields=LIST Similar to cut command, LIST is made up of one range, or many ranges separated by commas.Each range is one of: N N'th byte, character or field, counted from 1 N- from N'th byte, character or field, to end of line N-M from N'th to M'th (included) byte, character or field -M from first to M'th (included) byte, character or field	Import entries for some fields only in a table
-h, --help	Display help

Integration with HIVE:

Hiveruns on top of Hadoop. It is sufficient to install Hive only on the Hadoop master node.

Take note of the default data warehouse directory, set as a property in hive-default.xml.template configuration file. This must be the same as the base directory into which Hadoop Applier writes.

Since the Applier does not import DDL statements; you have to create similar schema's on both MySQL and Hive, i.e. set up a similar database in Hive using Hive QL(Hive Query Language). Since timestamps are inserted as first field in HDFS files,you must take this into account while creating tables in Hive.

SQL Query	Hive Query
CREATE TABLE t (i INT);	CREATE TABLE t ( time_stamp INT, i INT) [ROW FORMAT DELIMITED] STORED AS TEXTFILE;

Now, when any row is inserted into table on MySQL databases, a corresponding entry is made in the Hive tables. Watch the demo to get a better understanding.
The demo is non audio, and is meant to be followed in conjunction with the blog.You can also create an external table in hive and load data into the tables; its your choice!

Watch the Hadoop Applier Demo >>

Limitations of the Applier:

In the first version we support WRITE_ROW_EVENTS, i.e. only insert statements are replicated.

We have considered adding support for deletes, updates and DDL's as well, but they are more complicated to handle and we are not sure how much interest there is in this.

We would very much appreciate your feedback on requirements - please use the comments section of this blog to let us know!

The Hadoop Applier is compatible with MySQL 5.6, however it does not import events if binlog checksums are enabled. Make sure to set them to NONE on the master, and the server in row based replication mode.

This innovation includes dedicated contribution from Neha Kumari, Mats Kindahl and Narayanan Venkateswaran. Without them, this project would not be a success.

Give it a try! You can download a copy from http://labs.mysql.com and get started NOW!

PlanetMySQL Voting: Vote UP / Vote DOWN

High Availability for Drupal Part 1 - Investigating the Issues

Adam Donnison — Fri, 24 May 2013 06:11:42 +0000

Drupal is one of the most popular Content Management Systems (CMS) and is used increasingly in high-visibility sites, such as www.whitehouse.gov. This has brought a lot of attention on how to get the most performance out of Drupal and how to improve the availability of such sites. In this blog series I'll take you through the basics and on through to designing your own HA Drupal site.

But first, we need to understand what the challenges are in getting Drupal (or indeed any CMS) working on multiple servers in such a way as to ensure high availability and performance.

PlanetMySQL Voting: Vote UP / Vote DOWN

MySQL Book in Chinese

Michael McLaughlin — Fri, 24 May 2013 03:58:06 +0000

One of my old students and lab assistants stopped by to show his fiancée the BYU-Idaho campus. It was a long trip since he lives in Macao, China.

He kindly brought me a copy of my Oracle Database 11g and MySQL 5.6 Developer Handbook in simplified Chinese. He’s holding it in the photo.

That makes three books translated into Chinese, which made my day. It’ll be interesting to see if the new MySQL Workbench: Data Modeling & Development book gets translated into Chinese too. Oddly, I never hear about this from the publisher first.

The cover emphasized only the Dolphin, not the Oracle logo material. It made me wonder, how many MySQL users there might be in China. If anybody from China catches the post, it would be great to hear about the MySQL Community in China.

Likewise, if anybody in China catches the post and reads the book, please let me know if you liked it. Naturally, let me know if you found any problems with it too. By the way, I keep an errata for the book here.

PlanetMySQL Voting: Vote UP / Vote DOWN

An old note on the Storage Engine API

Stewart Smith — Fri, 24 May 2013 00:00:59 +0000

Whenever I stick my head into the MySQL storage engine API, I’m reminded of a MySQL User Conference from several years ago now.

Specifically, I’m reminded of a slide from an early talk at the MySQL User Conference by Paul McCullagh describing developing PBXT. For “How to write a Storage Engine for MySQL”, it went something like this:

Develop basic INSERT (write_row) support – INSERT INTO t1 VALUES (42)
Develop full table scan (rnd_init, rnd_next, rnd_end) - SELECT * from t1
If you’re sane, stop here.

A lot of people stop at step 3. It’s a really good place to stop too. It avoids most of the tricky parts that are unexpected, undocumented and unlogical (yes, I’m inventing words here).

PlanetMySQL Voting: Vote UP / Vote DOWN

Non-Deterministic Query in Replication Stream

Open Query — Thu, 23 May 2013 22:10:58 +0000

You might find a warning like the below in your error log:

130522 17:54:18 [Warning] Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. Statements writing to a table with an auto-increment column after selecting from another table are unsafe because the order in which rows are retrieved determines what (if any) rows will be written. This order cannot be predicted and may differ on master and the slave. Statement: INSERT INTO tbl2 SELECT * FROM tbl1 WHERE col IN (417,523)

What do MariaDB and MySQL mean with this warning? The server can’t guarantee that this exact query, with STATEMENT based replication, will always yield identical results on the slave.

Does that mean that you have to use ROW based (or MIXED) replication? Possibly, but not necessarily.

For this type of query, it primarily refers to the fact that without ORDER BY, rows have no order and thus a result set may show up in any order the server decides. Sometimes it’s predictable (depending on storage engine and index use), but that’s not something you want to rely on. You don’t have to ponder that, as an ORDER BY is never harmful.

Would ORDER BY col solve the problem? That depends!
If col is unique, yes. If col is not unique, then multiple rows could result and they’d still have a non-deterministic order. So in that case you’d need to ORDER BY col,anothercol to make it absolutely deterministic. The same of course applies if the WHERE clause only referred to a single col value: if multiple rows can match, then it’s not unique and it will require an additional column for the sort.

There are other query constructs where going to row based or mixed replication is the only way. But, just because the server tells you it can’t safely replicate a query with statement based replication, that doesn’t mean you can’t use statement based replication at all… there might be another way.

PlanetMySQL Voting: Vote UP / Vote DOWN

Zend 6 & Timezones

Michael McLaughlin — Thu, 23 May 2013 17:21:53 +0000

Just went through all my PHP testing against a fresh instance of Oracle with Zend Server Community Edition 6, and found these warnings, guess that’s pretty clean for the Oracle part of the installation. I didn’t notice it before because generally I do most of my PHP development against a MySQL database. I should have been configuring the php.ini file routinely, as qualified in this PHP forum discussion.

Warning: oci_set_client_info(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected the timezone 'UTC' for now, but please set date.timezone to select your timezone. in C:\Program Files (x86)\Zend\Apache2\htdocs\Oracle\Db.php on line 47
Warning: oci_set_module_name(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected the timezone 'UTC' for now, but please set date.timezone to select your timezone. in C:\Program Files (x86)\Zend\Apache2\htdocs\Oracle\Db.php on line 48
Warning: oci_set_action(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected the timezone 'UTC' for now, but please set date.timezone to select your timezone. in C:\Program Files (x86)\Zend\Apache2\htdocs\Oracle\Db.php on line 69

Turns out Zend 6 doesn’t automatically set the [Date] elements in the php.ini file, which is required for the oci_set_client_info(), oci_set_module_name(), and oci_set_action() functions of the OCI. You can find the php.ini file in the C:\Program Files (x86)\Zend\ZendServer\etc folder on Windows:

[Date]
; Defines the default timezone used by the date functions
; http://php.net/date.timezone
;date.timezone =
 
; http://php.net/date.default-latitude
;date.default_latitude = 31.7667
 
; http://php.net/date.default-longitude
;date.default_longitude = 35.2333
 
; http://php.net/date.sunrise-zenith
;date.sunrise_zenith = 90.583333
 
; http://php.net/date.sunset-zenith
;date.sunset_zenith = 90.583333

You can find the values for date.timezone here. Update the date.timezone as follows:

date.timezone = America/Denver

Then, reboot the Zend Server, and it fixes the warning messages.

PlanetMySQL Voting: Vote UP / Vote DOWN

MariaDB 5.5.31 Now Available

Monty Program Group Blog — Thu, 23 May 2013 13:15:05 +0000

The MariaDB project is pleased to announce the immediate availability of the MariaDB 5.5.31. This is a Stable (GA) release. See the Release Notes and Changelog for detailed information on this release and the What is MariaDB 5.5? page in the AskMonty Knowledgebase for general information about the MariaDB 5.5 series.

Download MariaDB 5.5.31

Release Notes Changelog What is MariaDB 5.5?

This release is primarily a bug-fix release.

Includes MariaDB 5.3.12 and MySQL 5.5.31

This release includes MariaDB 5.3.12 and MySQL 5.5.31.

Other Notable Updates

Includes XtraDB from Percona Server-5.5.30-rel30.2
Includes an alpha version of the QUERY_CACHE_INFO plugin, that allows to see the content of the query cache via the corresponding INFORMATION_SCHEMA table.
MariaDB MSI packages for Windows include the latest HeidiSQL 8.0.
Support for FusionIO/directFS atomic writes added (MDEV-4338). See the following for more information:

Ubuntu 8.04 LTS “Hardy” and 11.10 “Oneiric” Deprecated

In accordance with the MariaDB Deprecation Policy the following distributions have reached the end of their support periods (EOL):

Ubuntu 8.04 LTS “Hardy”
Ubuntu 11.10 “Oneiric”
Mint 9 LTS “Isadora”
Mint 12 “Lisa”

MariaDB 5.5.31) is therefore the last MariaDB release to feature packages for these distros. The repositories for them will remain online until the 5.5.32 release. At that time they will go away.

Thanks, and enjoy MariaDB!

PlanetMySQL Voting: Vote UP / Vote DOWN

Experiences with the McAfee MySQL Audit Plugin

Fernando Ipar — Thu, 23 May 2013 13:09:30 +0000

I recently had to do some customer work involving the McAfee MySQL Audit Plugin and would like to share my experience in this post.

Auditing user activity in MySQL has traditionally been challenging. Most data can be obtained from the slow or general log, but this involves a lot of data you don’t need too, and isn’t flexible at all. The specific problem of logging failed connection attempts has been discussed on a previous post in our blog.

Starting with 5.1, the new plugin API gives us more flexibility by allowing users to extend the server’s functionality with their own code, and this is what the McAffee plugin does.

Installation and configuration are straightforward following the available instructions. The only extra step I had to take was to extract the offsets for the Percona Server version I was using for the test (5.5.28-29.1). This is needed as the plugin needs the offset to some MySQL data structures that, the plugin authors say, aren’t exposed by a consistent API. If you also need to do this, the details are clearly explained here.

The plugin writes its output in json format, and supports writing it directly to a file, or to a unix socket, which means you can write a script to listen on this socket and process the audit records as you wish.

Performance-wise, I did basic tests on the VM I was working in and didn’t get significant differences between either output option, or between using the plugin or enabling the general log. Bear in mind these were basic tests (just a few mysqlslap runs with increasing levels of concurrency), but initially, I would think the advantage of the plugin is its flexibility, and not its performance, which seems to be on par with having the general log enabled.

The flexibility comes from the three variables that can be set to control what is logged by the plugin:
- audit_record_cmds : This is the list of commands you want written to the log (all the lists in these variables are comma separated). As pointed here, anything that would generate a write to the general log will be sent to the plugin, and you can control if it gets written on not with this list. I tested this with “connect,Quit” to log successful and failed connections. Yes, it had to be a capital Q in Quit for that to work, and no, my code-fu was not enough to understand why that is the case. Maybe someone more knowledgeable in MySQL internals can enlighten me here.
- audit_record_objs : List of database objects (tables, according to the docs) for which you want events written to the log.
- audit_whitelist_users : This one is undocumented on the wiki at the time of writing, and is a list of users for which you do not want events written to the log.

Just for reference, these are the lines I had to add to my config file for the plugin to work (plus one commented line for switching between file and socket for output):

plugin-load=AUDIT=libaudit_plugin.so audit_offsets=6464, 6512, 4072, 4512, 104, 2584 audit_json_file=1 audit_json_socket_name=/tmp/audit.sock #audit_json_socket=1 audit_json_log_file=/var/lib/mysql/audit.log audit_record_cmds=connect,Quit

Notice the audit_offsets that I mentioned had to be extracted due to this Percona Server version not being included in the binary.

And here’s a few sample output lines generated by the plugin with this configuration:

{"msg-type":"activity","date":"1369155747373","thread-id":"6439","query-id":"0","user":"debian-sys-maint","priv_user":"debian-sys-maint","host":"localhost","cmd":"Connect","query":"Connect"} {"msg-type":"activity","date":"1369155747373","thread-id":"6439","query-id":"219309","user":"debian-sys-maint","priv_user":"debian-sys-maint","host":"localhost","cmd":"Quit","query":"Quit"} {"msg-type":"activity","date":"1369155747383","thread-id":"6440","query-id":"0","user":"debian-sys-maint","priv_user":"debian-sys-maint","host":"localhost","cmd":"Connect","query":"Connect"}

In conclusion, the plugin API seems to be opening new possibilities of extending MySQL’s behavior in a way that, once set up, is transparent to users, and the McAfee MySQL Audit Plugin is only one of example of what can be achieved with it. It is a very good one for me, since I think proper audit trail support has been an important missing feature on the server, which has made using MySQL in PCI or SOX compliant environments, to name just two, artificially complicated, as one had to rely on too much info (general log) or external help (snort or similar IDS).

The post Experiences with the McAfee MySQL Audit Plugin appeared first on MySQL Performance Blog.

PlanetMySQL Voting: Vote UP / Vote DOWN

mycheckpoint revision 231 released

Shlomi Noach — Thu, 23 May 2013 12:21:52 +0000

A new release for mycheckpoint: lightweight, SQL oriented MySQL monitoring solution.

If you're unfamiliar with mycheckpoint, well, the one minute sales pitch is: it's a free and open source monitoring tool for MySQL, which is extremely easy to install and execute, and which includes custom queries, alerts (via emails), and out of the box HTTP server and charting.

This is mostly a maintenance release, with some long-time requested features, and of course solved bugs. Here are a few highlights:

Supports MariaDB and MySQL 5.6 (issues with new variables, space padded variables, text-valued variables)
Supports alerts via function invocation on monitored host (so not only checking alerts via aggregated data like 'Seconds_behind_master' but also by SELECT my_sanity_check_function() on monitored instance). See alerts.
Supports single-running-instance via "--single" command line argument
Supports strict sql_mode, including ONLY_FULL_GROUP_BY, overcoming bug #69310.
Supports sending of pending email HTML report
Better re-deployment process
Better recognizing of SIGNED/UNSIGNED values
Some other improvements in charting, etc.

mycheckpoint is released under the BSD license.

Downloads are available from the project's page.

PlanetMySQL Voting: Vote UP / Vote DOWN

MySQL at DrupalCamp, Goteborg - Sweden

Oracle MySQL Group — Thu, 23 May 2013 10:16:07 +0000

DrupalCamp Conference is hold in Göteborg, Sweden on May 25, 2013.

MySQL is part of this show and I would like to invite you to our session on "Scalability and Availability with MySQL Replication" given by Sven Sandberg on Saturday-May 25, 2013 @ 13:45-14:25!

So, if you are close to Göteborg or attending this event, do not forget to come to listen Sven's talk!

PlanetMySQL Voting: Vote UP / Vote DOWN

Percona XtraBackup 2.1.3 for MySQL available for download

MySQL Performance Blog — Thu, 23 May 2013 00:33:16 +0000

Percona is glad to announce the release of Percona XtraBackup 2.1.3 for MySQL on May 22, 2013. Downloads are available from our download site here and Percona Software Repositories.

This release fixes a high priority bug. It’s advised to upgrade your latest 2.1 version to 2.1.3 if you’re using the Percona XtraBackup with Percona XtraDB Cluster. This release is the latest stable release in the 2.1 series.

Bug Fixed:

Percona XtraBackup 2.1.2 would hang when performing State Snapshot Transfer. Bug fixed #1182698.

Release notes with all the bugfixes for Percona XtraBackup 2.1.3 are available in our online documentation. Bugs can be reported on the launchpad bug tracker.

* * *

Percona XtraBackup is the world’s only open-source, free MySQL hot backup software that performs non-blocking backups for InnoDB and XtraDB databases. With Percona XtraBackup, you can achieve the following benefits:

Backups that complete quickly and reliably
Uninterrupted transaction processing during backups
Savings on disk space and network bandwidth
Automatic backup verification
Higher uptime due to faster restore time

XtraBackup makes MySQL hot backups for all versions of Percona Server, MySQL, MariaDB, and Drizzle. It performs streaming, compressed, and incremental MySQL backups.

Percona’s enterprise-grade commercial MySQL Support contracts include support for XtraBackup. We recommend support for critical production deployments.

The post Percona XtraBackup 2.1.3 for MySQL available for download appeared first on MySQL Performance Blog.

PlanetMySQL Voting: Vote UP / Vote DOWN

MySQL vs Drizzle plugin APIs

Stewart Smith — Thu, 23 May 2013 00:11:21 +0000

There’s a big difference in how plugins are treated in MySQL and how they are treated in Drizzle. The MySQL way has been to create a C API in front of the C++-like (I call it C- as it manages to take the worst of both worlds) internal “API”. The Drizzle way is to have plugins be first class citizens and use exactly the same API as if they were inside the server.

This means that MySQL attempts to maintain API stability. This isn’t something worth trying for. Any plugin that isn’t trivial quickly surpasses what is exposed via the C API and has to work around it, or, it’s a storage engine and instead you have this horrible mash of C and C++. The byproduct of this is that no core server features are being re-implemented as plugins. This means the API is being developed in a vacuum devoid of usefulness. At least, this was the case… The authentication plugin API seems to be an exception, and it’s interesting to note that semisync replication is in fact a plugin.

So times may be changing… sort of. Yesterday I noted that some storage engine API features are only available if you’re InnoDB and I’ve voiced my general disappointment in the audit API being unsuitable to implement various forms of query logging already in the server (general query log, slow query log).

One thing to note: when the API is the same for both inside the server and a plugin, it makes initial refactoring very easy, and you quickly see the bits that could be improved.

PlanetMySQL Voting: Vote UP / Vote DOWN

Hint of the day: Warning level in Error Log to see Aborted Connections

Open Query — Wed, 22 May 2013 23:26:09 +0000

log_warnings = 2

Yields useful information in the MariaDB or MySQL error log file (or syslog on Debian/Ubuntu) you don’t want to miss out on.

You will know about aborted connections, which are otherwise only visible through global status as Aborted_connects (lost connection before they completed authentication) and Aborted_clients (cut fully authenticated connection).

It looks like

130523 2:14:05 [Warning] Aborted connection 173629 to db: 'unconnected' user: 'someapp' host: '10.2.0.50' (Unknown error)

You will know when, where from, and if for instance a wrong password was used you’ll see the username. Basically you’ll get as much info as the server has available at that point. Useful.

PlanetMySQL Voting: Vote UP / Vote DOWN

Why you shouldn't hire a devops

Kris Buytaert — Wed, 22 May 2013 21:11:15 +0000

Lately there have been a lot of organisations trying to hire a devops engineer.
I myselve have been asked to fill in devops roles ..

There's a number of issues with that.

The biggest problem is that I always have to ask what exactly the organisation is looking for.

So you want a devops engineer with experience in Linux, MongoDB, MySQL and Java , does that mean you want a Java developer who is familiar with MySQL and Linux and breaths a devops Culture.
Or a Linux expert who understands Java developers and knows how to tune Mongo and MySQL ?

It's absolutely unclear what you want when you are hiring "A devops engineer"

The second problem is that you are trying to hire people who are knowledgeable about devops,

Yet a lot of those people know that you can't do devops on your own , devops is not a jobtitle. devops is not a new devops team you create.

To some of them you are even making a fool out of yourselve, as to them you show that you don't understand devops

On top .. the ones that do apply for this fancy new devops role, are the ones that might not get the fact that the problem isn't about tooling but about people working together and helping eachother , so you end up hiring the wrong people.

Even in todays devops culture a system engineer is still a system engineer, and a developer is still a developer.
You might have developers supporting the build tool chain, or system engineers focussing on infrastructure automation.

But as John said almost 3 years ago they are good at their job.

Devops is not a word you slap onto a tool, a team or a person and expect magic to happen

Let's face it .. devops is hard, you can't do this on your own .. you need to find the right people ..

PlanetMySQL Voting: Vote UP / Vote DOWN

Percona MySQL University @Portland: June 17

MySQL Performance Blog — Wed, 22 May 2013 19:32:23 +0000

Peter Zaitsev leads a track at the inaugural Percona MySQL University event in Raleigh, N.C. on Jan. 29, 2013.

Portland is a well-recognized hub for Open Source technologies in the Northwest, home to conferences such as OSCON and Open Source Bridge as well as hosts of OpenSQL Camp in 2009. As such it is a very natural place for our next Percona MySQL University event scheduled for June 17.

We run this event in partnership with MySQL Meetup at Portland organized by our own Daniel Nichter, who recently moved to the area.

Percona MySQL University is a daylong, free, fast-paced and very technical MySQL educational event for wide range of people interested in MySQL – Developers, System Administrators, DBAs, etc. It will be held at Portland State University’s Smith Memorial Student Union.

We’ll finalize the schedule next week and still have some speaking opportunities available – if you would like to share your MySQL story at this event please email Matthew Dowell by Tuesday, May 28.

If you’re not in Portland and would like Percona MySQL University to come to your city, please fill out the form to let us know. We’ll try to come to the cities showing greatest interest.

As usual space is limited, so Register Now!

The post Percona MySQL University @Portland: June 17 appeared first on MySQL Performance Blog.

PlanetMySQL Voting: Vote UP / Vote DOWN

MySQL and the SSB – Part 2 – MyISAM vs InnoDB low concurrency

Justin Swanhart — Wed, 22 May 2013 13:50:16 +0000

This blog post is part two in what is now a continuing series on the Star Schema Benchmark.

In my previous blog post I compared MySQL 5.5.30 to MySQL 5.6.10, both with default settings using only the InnoDB storage engine. In my testing I discovered that innodb_old_blocks_time had an effect on performance of the benchmark. There was some discussion in the comments and I promised to follow up with more SSB tests at a later date.

I also promised more low concurrency SSB tests when Peter blogged about the importance of performance at low concurrency.

The SSB
The SSB tests a database’s ability to optimize queries for a star schema. A star schema presents some unique challenge to the database optimizer. The SSB benchmark consists of four sets of queries. Each set is known as a “flight”. I have labeled each query as Q{FLIGHT_NUMBER}.{QUERY_NUMBER}. In general, each flight examines different time periods or different regions. The flights represent the type of investigations and drill-downs that are common in OLAP analysis.

Each query in each flight (Q1.1 for example) is tested with a cold buffer pool. Then the query is tested again without restarting the database. The first test is described as the cold test, and the second as the hot test. The database software is restarted after the hot test. All OS caches are dropped at this time as well.

These set of queries were tested on the SSB at SCALE FACTOR: 20. This means there is approximately 12GB of data in the largest table.

You can find the individual SSB query definitions in my previous blog post.

Test environment
These tests were done on a relatively fast machine with a Xeon E5-2680 (8 cores, 16 threads) with fast IO (OCZ R4 1.6TB) and 128GB memory. For the hot test, the data fits in the buffer pool and has been loaded by the cold test already. The buffer pool and adaptive hash index are cold for the cold test. All tests were done with no concurrency. The hardware for this test was provided by Adotomi. I will be blogging about raw performance of the OCZ card in another post.

Also, while it is labeled on the graphs, it is important to note that in all cases, lower times are better.

SSB Flight #1
Here you will see the start of an interesting trend. MyISAM is faster when the data is not cached (the cold run) but is slower in the hot (cached) run. I did some investigation during the testing and found that InnoDB does more IO than MyISAM when the database is cold, but uses less CPU time when the database is hot. I am only speculating (and I can investigate further), but I believe the adaptive hash index is improving performance of InnoDB significantly during the hot run, as hash indexes are faster than a b-tree index. Also accessing pages from the buffer pool should be faster than getting them from the OS cache, which is another advantage of InnoDB.

SSB Flight #2
Flight #2 is similar to Flight #1. MyISAM is faster than InnoDB when the database is cold, but the opposite is true when the database is hot.

SSB Flight #3
Here in some cases MyISAM is substantially faster than InnoDB both cold and hot.

SSB Flight #4
There is one query in this flight, Q4.3, which is faster using MyISAM than InnoDB. Like the queries in Flight #3 that are faster using MyISAM, Q4.3 examines very little data. It seems that InnoDB performs better when a larger number of rows must be joined together (Q4.1, Q4.2) but worse when small amounts of data are examined.

Conclusion

In some cases MyISAM is faster than InnoDB, but usually only when the buffer pool is cold. Please don’t take away that you should be using MyISAM for everything!. MyISAM may be good for raw performance, but there are limitations which MyISAM imposes that are difficult to work with. MyISAM does not maintain checksum consistency during regular operations and is not ACID compliant. MyISAM and InnoDB may perform differently under concurrency, which this benchmark does not cover. I will make a follow-up post about concurrency in another blog post in this series. Regardless, when the working set fits in memory, InnoDB almost always performs better, at least for this workload.

Notes

MySQL version used: 5.6.11, custom compiled to remove performance_schema

For the InnoDB tests, a 64GB buffer pool was used. O_DIRECT was used so, there was no caching of data at the filesystem level. The InnoDB indexes were built using ALTER TABLE fast index creation (merge sort).

For the MyISAM tests I used a 10GB key buffer. I used ALTER TABLE DISABLE KEYS and built the keys with sort via ALTER TABLE ENABLE KEYS.
my.cnf

[mysqld]
datadir=/mnt/mysql56/data
basedir=/usr/local/mysql
socket=/var/lib/mysql/mysql.sock
user=justin
innodb_buffer_pool_size=64G
innodb_log_file_size=4G
innodb_file_per_table
innodb_stats_on_metadata=off
innodb_file_format=barracuda
innodb_log_buffer_size=32M
innodb_buffer_pool_instances=16
metadata_locks_hash_instances=32
table_open_cache_instances=8
sort_buffer_size=128k
read_rnd_buffer_size=8M
join_buffer_size=8M
default_tmp_storage_engine=myisam
tmpdir=/dev/shm
innodb_undo_logs=32
innodb_old_blocks_time=0
table_open_cache=2048
table_definition_cache=16384
innodb_flush_method=O_DIRECT
key_buffer_size=10G
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
innodb_stats_persistent
innodb_stats_auto_update=off
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

The post MySQL and the SSB – Part 2 – MyISAM vs InnoDB low concurrency appeared first on MySQL Performance Blog.

PlanetMySQL Voting: Vote UP / Vote DOWN

MySQL Workbench 6.0 – A Sneak Preview

The Workbench Team — Wed, 22 May 2013 12:30:57 +0000

The MySQL Workbench team has been a little quiet for the past few months, but that’s because we’ve been busy working on an exciting new version. Read more about it in Tomas’ blog

PlanetMySQL Voting: Vote UP / Vote DOWN

MySQL Workbench 6.0 – A Sneak Preview

Tomas Ulin — Wed, 22 May 2013 08:57:54 +0000

The MySQL Developer Tools team is known for their steady release cycle, putting out a new MySQL Workbench release every 4-5 weeks. Now that it has been a bit quiet for a while you may wonder what is going on. Let me share some inside knowledge of what’s happening behind the scenes.

MySQL Workbench is a key component of our MySQL stack and extremely popular as shown by download numbers and interest in our white papers. It is the face of MySQL on the desktop, and we aim to make it even more popular for developers & DBAs than it is today. We have been looking for strong C++ and Python developers to grow the team further, and I’m happy to welcome Marcin Szalowicz from Poland and Miguel Tadeu from Portugal who started this Monday. They are going to work on both, the C++ backend and the native UI frontend of the product.

For some time the focus for the team has been on the next major release of MySQL Workbench which will be called 6.0. If you are familiar with the current version (5.2) you are going to see that we have been listening carefully to the feedback we’ve got. Here are the main areas that have been improved.

Simplified Workflow

The current WB 5.2 UI was designed for three specific workflows, SQL Development, EER Modelling and Administration. This is represented by the 3 main areas of the WB 5.2 Home screen and makes good sense if you have a very specific role in your company and use WB for that specific purpose, e.g. Administration of MySQL servers. What we learned is that e.g. many developers working in the SQL Editor also want to perform administrative tasks on their development machines and therefore also open an Administrator session. The same is true for DBAs using the SQL Editor to run scripts and queries.

For that reason we have unified SQL Development & Administration and made both available in a single session when you connect to your MySQL Server. The connections now take the main area of the new, redesigned WB 6.0 Home screen. We have also adopt a modern looking UI.

Enterprise Feature Support

MySQL Enterprise Backup and the new Audit Log functionality found in MySQL 5.6 Enterprise are very compelling features for our customers. We got many requests to deliver a streamlined GUI for these features that will now be released as part of WB 6.0. MySQL Workbench will perform all tasks that are needed to configure your MySQL installation for the use of the backup feature and setting up an incremental backup is now as easy as clicking a button.

Improved Code Quality

Another area of focus was to improve the quality of the product. WB 6.0 will contain over 100 bug fixes and will be tested by a dedicated QA team. We are soon going to launch the public Beta test phase and with the help of the MySQL community the WB 6.0 will be the best MySQL Workbench release to date.

More Than 50 Improvements

Apart from working on the major UI changes a lot of work has been invested in improving and extending existing features like Server Status overview, Visual Explain, Editors, Schema Inspector, Table Data Search, and much more.

I hope you got interested in the upcoming MySQL Workbench 6.0 release and you will join the Beta test when it is first announced.

PlanetMySQL Voting: Vote UP / Vote DOWN

Some storage engine features you only get if you’re InnoDB

Stewart Smith — Wed, 22 May 2013 03:45:49 +0000

I had reason to look into the extended secondary index code in MariaDB and MySQL recently, and there was one bit that I really didn’t like.

MariaDB:

share->set_use_ext_keys_flag(legacy_db_type == DB_TYPE_INNODB);

MySQL:

use_extended_sk= (legacy_db_type == DB_TYPE_INNODB);

In case you were wondering what “legacy_db_type” actually does, let me tell you: it’s not legacy at all, it’s kind of key to how the whole “metadata” system in MySQL works. For example, to drop a table, this magic number is used to work out what storage engine to call to drop the table.

Now, these code snippets basically kiss goodbye to the idea of a “pluggable storage engine” architecture. If you’re not InnoDB, you don’t get to have certain features. This isn’t exactly MySQL or MariaDB encouraging an open storage engine ecosystem (quite the opposite really).

Having the MySQL server have this incredibly basic, busy and incomplete understanding of metadata has always been a bit of a mess. The code for reading a table definition out of the FRM file really does show its age, and has fingers all through the server.

If somebody was serious about refactoring server code, you’d certainly be looking here, as this code is a major source of arbitrary limitations. However, if you have the server and the engine(s) both having separate views of what is the “correct” state of metadata you end up with a mess (anyone who has had InnoDB be out of sync with FRMs knows this one). I worry that the FRM code will be replaced with something even less understandable by humans, again making the mistake that the server knows the state of the engine better than the engine does.

Implementing asynchronous cascade delete in MySQL

Ike Walker — Wed, 22 May 2013 02:12:00 +0000

A while back one of my foreign keys started causing trouble. The problem was that some parent rows had tens of thousand of child rows, and the foreign key was defined with CASCADE DELETE enabled. When we deleted one of those parent rows on a master database, it took several seconds to execute the delete because of the cascade. This led to latency for the end user, and also led to replication delays.

The immediate solution was make the application tolerant of orphaned rows in the child table and to drop the explicit foreign key constraint.

I didn't really want to leave those orphaned rows hanging around in the child table, so I decided to implement an asynchronous process to delete the orphaned rows on a scheduled basis. Read on for a description of that process.

Using the sakila database as an example, imagine I drop the foreign key between file_category and category, like so:

alter table sakila.film_category drop foreign key fk_film_category_category;

Without the foreign key in place, deletes on the category table lead to orphaned rows in film_category. For example, I will delete the "New" category:

``` mysql> delete from sakila.category

-> where name = 'New';

Query OK, 1 row affected (0.01 sec) ```

There are several ways to count the orphaned rows. Here are two different naive implementations using OUTER JOIN or NOT EXISTS:

``` mysql> select count(*)

-> from sakila.film_category c 
-> left outer join sakila.category p on p.category_id = c.category_id 
-> where p.category_id is null;

+----------+ | count(*) | +----------+ | 63 | +----------+ 1 row in set (0.00 sec)

mysql> select count(*)

-> from sakila.film_category c 
-> where not exists 
-> (
->   select NULL from sakila.category p where p.category_id = c.category_id
-> );

+----------+ | count(*) | +----------+ | 63 | +----------+ 1 row in set (0.01 sec) ```

I can also delete the orphaned rows using the same query approaches. I'll roll the first delete back so I can demonstrate the second query in the same session:

``` mysql> set autocommit = 0; Query OK, 0 rows affected (0.00 sec)

mysql> delete c.*

-> from sakila.film_category c 
-> left outer join sakila.category p on p.category_id = c.category_id 
-> where p.category_id is null;

Query OK, 63 rows affected (0.02 sec)

mysql> rollback; Query OK, 0 rows affected (0.00 sec)

mysql> delete c.*

-> from sakila.film_category c 
-> where not exists (select NULL from sakila.category p where p.category_id = c.category_id);

Query OK, 63 rows affected (0.01 sec)

mysql> commit; Query OK, 0 rows affected (0.00 sec) ```

This approach will be very slow for tables containing millions of rows, so in my real world case I didn't use this approach. Instead I decided it would be a lot easier and faster to delete the orphaned rows if I knew who their parent was. To this end I created a new table to track the deleted rows, and populated it using a trigger. Continuing the example in the sakila database:

``` CREATE TABLE category_deleted ( category_id tinyint(3) unsigned NOT NULL, name varchar(25) NOT NULL, last_update timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, delete_time timestamp NOT NULL, PRIMARY KEY (category_id) ) ENGINE=InnoDB DEFAULT CHARSET=utf8;

DELIMITER $$

DROP TRIGGER IF EXISTS sakila.TR_A_DEL_CATEGORY $$

CREATE TRIGGER sakila.TR_A_DEL_CATEGORY AFTER DELETE ON sakila.category FOR EACH ROW BEGIN

INSERT IGNORE INTO sakila.category_deleted (category_id, name, last_update, delete_time) VALUES (old.category_id, old.name, old.last_update,now());

END $$

DELIMITER ; ```

Now I can delete another category to test the trigger:

``` mysql> -- delete a single category mysql> delete from sakila.category

->     where name = 'Classics';

Query OK, 1 row affected (0.01 sec)

mysql> -- count the orphaned rows using the _deleted table mysql> select count(*)

-> from sakila.film_category c 
->     inner join sakila.category_deleted p on p.category_id = c.category_id;

+----------+ | count(*) | +----------+ | 57 | +----------+ 1 row in set (0.00 sec) ```

I also wanted to execute the deletes on the child table in chunks, so I implemented a stored procedure to delete the orphaned rows by iterating through the rows in the _deleted table, deleting the child rows, and then deleting from the _deleted table. If I had to implement it again, I would probably use common_schema to chunk the deletes so I wouldn't need the stored procedure.

Here's an implementation using common_schema:

``` -- first delete the orphaned rows from the child table set @script := " split(sakila.film_category: delete sakila.film_category.*

from sakila.film_category 
inner join sakila.category_deleted on sakila.category_deleted.category_id = sakila.film_category.category_id

) SELECT $split_total_rowcount AS 'rows deleted so far'; "; call common_schema.run(@script);

-- then delete the rows from the _deleted table (assuming they have no children) delete p.* from sakila.category_deleted p left outer join sakila.film_category c on c.category_id = p.category_id where c.category_id is null; ```

PlanetMySQL Voting: Vote UP / Vote DOWN

Implementing asynchronous cascade delete in MySQL

Ike Walker — Tue, 21 May 2013 23:12:00 +0000

The immediate solution was make the application tolerant of orphaned rows in the child table and to drop the explicit foreign key constraint.

Using the sakila database as an example, imagine I drop the foreign key between file_category and category, like so:

alter table sakila.film_category drop foreign key fk_film_category_category;

Without the foreign key in place, deletes on the category table lead to orphaned rows in film_category. For example, I will delete the "New" category:

``` mysql> delete from sakila.category

-> where name = 'New';

Query OK, 1 row affected (0.01 sec) ```

There are several ways to count the orphaned rows. Here are two different naive implementations using OUTER JOIN or NOT EXISTS:

``` mysql> select count(*)

-> from sakila.film_category c 
-> left outer join sakila.category p on p.category_id = c.category_id 
-> where p.category_id is null;

+----------+ | count(*) | +----------+ | 63 | +----------+ 1 row in set (0.00 sec)

mysql> select count(*)

-> from sakila.film_category c 
-> where not exists 
-> (
->   select NULL from sakila.category p where p.category_id = c.category_id
-> );

+----------+ | count(*) | +----------+ | 63 | +----------+ 1 row in set (0.01 sec) ```

I can also delete the orphaned rows using the same query approaches. I'll roll the first delete back so I can demonstrate the second query in the same session:

``` mysql> set autocommit = 0; Query OK, 0 rows affected (0.00 sec)

mysql> delete c.*

-> from sakila.film_category c 
-> left outer join sakila.category p on p.category_id = c.category_id 
-> where p.category_id is null;

Query OK, 63 rows affected (0.02 sec)

mysql> rollback; Query OK, 0 rows affected (0.00 sec)

mysql> delete c.*

-> from sakila.film_category c 
-> where not exists (select NULL from sakila.category p where p.category_id = c.category_id);

Query OK, 63 rows affected (0.01 sec)

mysql> commit; Query OK, 0 rows affected (0.00 sec) ```

DELIMITER $$

DROP TRIGGER IF EXISTS sakila.TR_A_DEL_CATEGORY $$

CREATE TRIGGER sakila.TR_A_DEL_CATEGORY AFTER DELETE ON sakila.category FOR EACH ROW BEGIN

INSERT IGNORE INTO sakila.category_deleted (category_id, name, last_update, delete_time) VALUES (old.category_id, old.name, old.last_update,now());

END $$

DELIMITER ; ```

Now I can delete another category to test the trigger:

``` mysql> -- delete a single category mysql> delete from sakila.category

->     where name = 'Classics';

Query OK, 1 row affected (0.01 sec)

mysql> -- count the orphaned rows using the _deleted table mysql> select count(*)

-> from sakila.film_category c 
->     inner join sakila.category_deleted p on p.category_id = c.category_id;

+----------+ | count(*) | +----------+ | 57 | +----------+ 1 row in set (0.00 sec) ```

Here's an implementation using common_schema:

``` -- first delete the orphaned rows from the child table set @script := " split(sakila.film_category: delete sakila.film_category.*

from sakila.film_category 
inner join sakila.category_deleted on sakila.category_deleted.category_id = sakila.film_category.category_id

) SELECT $split_total_rowcount AS 'rows deleted so far'; "; call common_schema.run(@script);

PlanetMySQL Voting: Vote UP / Vote DOWN

Shard-Query 2.0 Beta 1 released

Justin Swanhart — Tue, 21 May 2013 20:11:20 +0000

It is finally here. After three years of development, the new version of Shard-Query is finally available for broad testing.

This new version of Shard-Query is vastly improved over previous versions in many ways. This is in large part due to the fact that the previous version of Shard-Query (version 1.1) entered into production at a large company. Their feedback during implementation was invaluable in building the new Shard-Query features. The great thing is that this means that many of the new 2.0 features have already been tested in at least one production environment.

This post is intended to highlight the new features in Shard-Query 2.0. I will be making posts about individual features as well as posting benchmark results.

Configuration Repository
Shard-Query 1.x stored the configuration in a .ini file. In an installation with many nodes, keeping the .ini file in sync was difficult. There was no way to guarantee that all of Shard-Query saw the state of the cluster (in terms of node composition) as an entire unit. Shard-Query 2.0 solves this problem and makes configuration and bootstrapping of new nodes simpler. Shard-Query 2.0 stores the configuration inside of a database schema, rather than in a flat file. Each node is made aware of the configuration repository through the node setup process. The cluster configuration is then read from the repository each time Shard-Query initializes.

REST interface/GUI
Shard-Query 2.0 features a user interface built using the DooPHP MVC framework. This interface was graciously contributed by Alex Hurd as part of the implementation I spoke of earlier. The web interface provides for both query execution and post-installation Shard-Query configuration.

MySQL proxy 0.8.3 interface
Shard-Query now includes a Lua script for MySQL proxy. This allows applications designed to interact with MySQL to work with Shard-Query transparently. This is very useful for tools such as Mondrian, Tableau or other tools that are designed for working with and visualizing big data.

DDL and SHOW support
Shard-Query now supports creating and dropping tables and indexes. The MySQL Proxy interface also supports SHOW commands such as SHOW TABLES and SHOW DATABASES. Please note that cross-shard operations such as creating and dropping tables requires that all shards be online for the operation. Failures may result in inconsistency (tables or indexes on some nodes, but not others) so please use with care. Unfortunately MySQL does not have transactional DDL.

Extended SQL support
Previous versions of Shard-Query only supported limited support for SQL expressions. Expressions such as ROUND(SUM(expr), 2) did not work. Nor did SUM(expr)/COUNT(expr). These issues have been resolved. Shard-Query now supports nearly all MySQL SELECT syntax. This includes complex expressions, ORDER BY, LIMIT, HAVING and WITH ROLLUP. Support for subqueries in the FROM clause has been extended. UNION and UNION ALL queries are now executed slightly differently as well. SQL syntax checking is now performed as well, and malformed queries return familiar error messages.

Partitioning based parallelism
Tables partitioned at the MySQL level are now automatically detected and used for intra-query parallelism. Currently RANGE, LIST, RANGE COLUMNS and LIST COLUMNS partitioning methods are supported for partition parallelism when a single column is used as the partition key. Shard-Query will automatically construct WHERE clauses to examine individual partitions and examine partitions in parallel. The next version of Shard-Query (2.5) will support all types of partitions for any number of partition key columns for MySQL 5.6.10 or greater, through use of the new PARTITION hint.

Improved parallel loading mechanism
The Shard-Query loader was formerly multi-process, but this did not work on Windows. Now the loader has been split into two parts. The first is a quick scan operation with identifies individual “chunks” of the table to load. The second phase involves Gearman workers actually loading those chunks. Currently, the flat file input must be available on all nodes running loader workers via the same path. This likely means that the files must be placed on shared storage or duplicated on all nodes on which the loader workers run. This limitation will probably be lifted in the future.

Asynchronous query execution
Shard-Query now supports executing queries asynchronously. There is a job table which can be used to determine if a background job has completed yet. See the bin/update_job_table script which actually populates the job table. If you don’r run (or cron) the script, you won’t see any jobs completing. See another example in the GUI (it invokes the job table update code too). This feature can be used to execute long running queries over large amounts of data.

Multi-shard (range) shard lookups
Queries which access more than one shard key via IN, BETWEEN or OR clauses will now be sent to only the shards which contain the keys in question. In previous versions range lookups scanned all shards. A bug was also fixed in WHERE clause processing. WHERE clause expressions such as “WHERE (the_shard_key IN(1,2,3,4,5))” were not handled properly due to the enclosing parenthesis, but this has been resolved.

Star Schema Optimization
Star schemata present unique challenges to traditional database optimizers. By turning on star schema optimization Shard-Query can offer vastly improved query performance for queries which utilize a star schema. Often an improvement of an order of magnitude or more can be seen when using this option, but it only works forstar schema.

Custom aggregate functions
The parser now supports the addition of custom aggregate functions to the SQL dialect. A reference function called PERCENTILE(expr, N) is included. Custom non-aggregate functions are not yet supported.

PlanetMySQL Voting: Vote UP / Vote DOWN

PHPTek

Dave Stokes — Tue, 21 May 2013 18:08:01 +0000

The PHPTek Conference bills itself as the premier professional PHP conference with a community flair. Having been to dozens of PHP conferences, I thought that was a pretty bold claim.

MySQL Boogiebot

But as they say in Texas, it ain’t bragging if you can do it.

MySQL plush dolphins and BoogieBots. The wind up dancing robots were popular in Chicago

And the organizers picked perfect weather for this Chicago based show.

The quality of the presenters and presentations was amazing high. Most of these sessions covered intricacies with the PHP language. The talks covered the range from beginning Symfony 2 to advanced security. The presenters were indeed the cream of the crop and the material covered very well developed.

Oracle’s own Ligaya Trumelle started the second day with a MySQL 5.6 new features session specifically tailored to what this elite audience wanted to know about 5.6. Later she teamed with Davey Shafik on Engine Yard on MySQL HA, Recovery, and Load Balancing. The second talks was a comprehensive look at the most popular options for all three areas¹.

Oracle was a sponsor of this show and I also had an opportunity to present Ten Things To Do To Make Your MySQL Databases Stronger and Healthier when not at the booth handing out three hundred Boogie Bots, MySQL Stickers, and a handful of plush Sakila Dolphins. The attendee questions at the booth were also high level (isolation levels for the best insert concurrency, how to use GTIDs, and partitioning) as opposed to more mundane questions at other shows.

So this truly is the premier PHP show and if you are a serious PHP coder, you need to plan to attend this show next year.

You should download their slides on this as they worked hard to provide good benchmark numbers. Sadly some of the software (third party) did not prove to be robust enough for a good comparison. Lig and Davey put a lot of hard work into this sessions and the slides were loaded with vital info and their configurations.

PlanetMySQL Voting: Vote UP / Vote DOWN

Replication in MySQL 5.6: GTIDs benefits and limitations – Part 1

MySQL Performance Blog — Tue, 21 May 2013 10:00:42 +0000

Global Transactions Identifiers are one of the new features regarding replication in MySQL 5.6. They open up a lot of opportunities to make the life of DBAs much easier when having to maintain servers under a specific replication topology. However you should keep in mind some limitations of the current implementation. This post is the first one of a series of articles focused on the implications of enabling GTIDs on a production setup.

The manual describes very nicely how to switch to GTID-based replication, I won’t repeat it.

Basically the steps are:

Make the master read-only so that the slaves can execute all events and be in sync with the master
Change configuration for all servers and restart them
Use CHANGE MASTER TO to instruct all servers to use GTIDs
Disable read-only mode

This procedure will switch all your servers from regular replication to GTID replication. But if you are running a production system, you will probably want to gradually enable GTID replication for an easier rollback in the event of a problem. And some items in the documentation are not so clear.

For instance:

Do we really need to restart all the servers at the same time? Downtime is something we like to avoid!
Is it necessary to make the master read-only?
Can we use regular replication for some slaves and GTID replication for other slaves at the same time?

To find an answer to these questions, let’s create a simple replication configuration with one master and two slaves, all running MySQL 5.6 with GTIDs disabled.

First try: configure only one of the servers with GTIDs

Let’s stop slave #2, change configuration and restart it:

mysql> show slave status\G
[...]
Slave_IO_Running: No
Slave_SQL_Running: Yes
          [...]

The error log tells us why the IO thread has not started:

2013-05-17 13:21:26 3130 [ERROR] Slave I/O: The slave IO thread stops because the master has GTID_MODE OFF and this server has GTID_MODE ON, Error_code: 1593

So unfortunately if you want replication to work correctly, gtid_mode must be ON on all servers or OFF on all servers, but not something in the middle.

What if we try to reconfigure the master? This time, replication on slave #1 will stop:

2013-05-17 13:32:08 2563 [ERROR] Slave I/O: The slave IO thread stops because the master has GTID_MODE ON and this server has GTID_MODE OFF, Error_code: 1593

These simple tests answer the first two questions: replication works only if all servers have the same value for gtid_mode, so you should restart them at the same time, which is best done by making the master read-only. However, “at the same time” means “at the same binlog position”, so you can perfectly restart the servers one by one.

Second try: GTIDs enabled, mixing regular replication and GTID replication

This time, we will enable GTID replication on slave #1, but not on slave #2:

# slave #1
mysql> change master to master_auto_position = 1;
mysql> start slave;

and let’s create a new table on the master:

mysql> create table test.t (id int not null auto_increment primary key);

Executing SHOW TABLES FROM test on both slaves shows that the table has been created everywhere. So once GTIDs are enabled on all servers, you can have some slaves using file-based positioning and some other slaves using GTID-based positioning.

This answers the second question: we can have different replication modes on different servers, but only if all servers have gtid_mode set to ON. Could it be interesting to run file-based replication when gtid_mode is ON? I can’t think of any use case, so in practice, you’ll probably use either file-based replication only (gtid_mode=off for all servers) or GTID-based replication only (gtid_mode=on for all servers).

Additional question: how can you know if a slave is using GTID-based replication by inspecting the output of SHOW SLAVE STATUS? Look at the last field, Auto_Position:

# Slave #1
mysql> show slave status\G
[...]
Auto_Position: 1  -> GTID-based positioning
# Slave #2
mysql> show slave status\G
[...]
Auto_Position: 0  -> File-based positioning

Conclusion

Enabling GTID-based replication can be tricky if your application does not easily tolerate downtime or read-only mode, especially if you have a lot of servers to reconfigure. It would be really nice to be able to mix servers where gtid_mode is ON with servers where gtid_mode is OFF. This would greatly simplify the transition to GTID-based replication and allow easier rollbacks if something goes wrong.

The post Replication in MySQL 5.6: GTIDs benefits and limitations – Part 1 appeared first on MySQL Performance Blog.

PlanetMySQL Voting: Vote UP / Vote DOWN

MyISAM's "table lock" problem, and how InnoDB solves it

Steve Meyers — Mon, 20 May 2013 21:54:00 +0000

Most serious users of MySQL have moved their tables to InnoDB years ago. For those who haven't, let's discuss why InnoDB is a more scalable solution than MyISAM. MyISAM was designed to be very fast for read queries. It does not handle higher loads of writes very well. It also suffers a more serious flaw: it isn't crash-safe. In other words, you better have frequent backups. MyISAM tables
PlanetMySQL Voting: Vote UP / Vote DOWN

Why I do what I do?

Cedric PEINTRE — Mon, 20 May 2013 20:46:37 +0000

I was sincerely affected by this last MySQL post and this other very long post from Jeremy Cole.
Yes, these two guys are MySQL rock stars and they are really impressives, their involvement in the MySQL community is utter!

I don’t want to write a long long speech about my simple life…
I just want to clarify why I do what I do.

Many people have asked or wondered without asking why I do what I do
(Jeremy Cole - 2013)

A few years ago Ashley Unitt asked me what I was most proud of, and now, I can make a complete answer. I’m very proud to take part of a community, MySQL has transformed my job into a passion and an incredible desire to share this passion, as honestly as possible.

What does it mean in real life?

If I can help in decision making or take part of a discussion, without any bad motivations, independently, I do it!

If the opportunity arises to have a good time over a glass of rosé and discussing open source with Stéphane Varoqui, I do it!

If I have suddenly an idea and a friend with me for several nights and weekends to achieve it, oh yes, I do it!

I do it because it’s terribly exciting and fun, even if I’ve just did it for myself… on a whim.

Yes, I’m very proud when we made MYXPLAIN.net with Max, I’m proud to work for an amazing company, I’m proud to advise someone who challenge me from the other side of the world, I’m proud to read an email from Rick James in my inbox…

I’m proud to be honest!

PlanetMySQL Voting: Vote UP / Vote DOWN

Get the Best from Web, Cloud, and Embedded Applications as a MySQL DBA

Oracle MySQL Group — Mon, 20 May 2013 20:03:01 +0000

After taking this MySQL for Database Administrators course, you will be equipped to use all the features of MySQL to get the best out of your Web, Cloud, and embedded applications, whether you work with the command line or graphical tools such as MySQL Workbench and MySQL Enterprise Monitor, whether your application uses complex queries or the NoSQL API, and whether your preferred challenge is replicated servers or highly-tuned transactional systems.

You can take this 5-day live instructor-led course as a:

Live-Virtual Event: Take this course from your own desk, no travel required. You can choose from a wide selection of events on the schedule to suit different timezones.
In-Class Event: Travel to an education center to attend this class. Below is a selection of events already on the schedule.

Location	Date	Delivery Language
Brussels, Belgium	3 June 2013	English
London, England	9 September 2013	English
Aix-en-Provence, France	2 December 2013	French
Bordeaux, France	2 December 2013	French
Nice, France	4 November 2013	French
Puteaux, France	16 September 2013	French
Strasbourg, France	1 July 2013	French
Dresden, Germany	3 June 2013	German
Dusseldorf, Germany	24 June 2013	German
Gummersbach, Germany	1 July 2013	German
Hamburg, Germany	24 June 2013	German
Munchen, Germany	19 August 2013	German
Munster, Germany	9 September 2013	German
Stuttgart, Germany	8 July 2013	German
Budapest, Hungary	4 November 2013	Hungarian
Belfast, Ireland	24 June 2013	English
Milan, Italy	7 October 2013	Italian
Rome, Italy	17 June 2013	Italian
Utrecht, Netherlands	24 June 2013	Dutch
Warsaw, Poland	5 August 2013	Polish
Lisbon, Portugal	16 September 2013	European Portugese
Cairo, Egypt	30 June 2013	English
Nairobi, Kenya	22 July 2013	English
Jakarta, Indonesia	16 September 2013	English
Petaling Jaya, Malaysia	1 July 2013	English
Makati City, Philippines	3 June 2013	English
Melbourne, Australia	3 June 2013	English
Curitiba, Brazil	27 May 2013	Brazilian Portugese
Sao Paolo, Brazil	10 June 2013	Brazilian Portugese
Mexico City, Mexico	24 June 2013	Spanish

To register for this course or to learn more about the courses on the authentic MySQL curriculum, go to http://oracle.com/education/MySQL.

PlanetMySQL Voting: Vote UP / Vote DOWN

More details on "MySQL 5.6 Experiences" coming soon...

Valeriy Kravchuk — Mon, 20 May 2013 18:12:01 +0000

I've already shared my presentation few hours before I made it during PLMCE 2013, back on April 24. Let's just have it here for the reference: http://www.slideshare.net/ValeriyKravchuk/mysql-56experiencesbugssolutions50mins.

During the upcoming weeks I plan to explain every slide in more details (as 50 minutes were not enough for this) here and check status of all the active bugs mentioned in it. I'll also check new bugs for each major feature mentioned (if any). So, stay tuned...
PlanetMySQL Voting: Vote UP / Vote DOWN

Webinar: SQL Query Patterns, Optimized

Bill Karwin — Mon, 20 May 2013 10:00:28 +0000

Next Friday, May 31 at 10 a.m. Pacific, I’ll present Percona’s next webinar, “SQL Query Patterns, Optimized.”

Based on my experiences solving tough SQL problems for Percona training and consulting, I’ll classify several common types of queries with which developers struggle. I’ll test several SQL solutions for each type of query objective, and show how you can use MySQL 5.6 built-in methods to analyze them for optimal query efficiency. The discussion will cover optimizer reports, query profiling, and session status to measure performance.

The query patterns will include:

Exclusion Join
Random Selection
Greatest-Per-Group
Dynamic Pivot
Relational Division

Please register for this webinar and join me next Friday!

The post Webinar: SQL Query Patterns, Optimized appeared first on MySQL Performance Blog.

PlanetMySQL Voting: Vote UP / Vote DOWN

New Article on Performance Schema on Dr.Dobbs

Mark Leith — Mon, 20 May 2013 09:06:58 +0000

Interested in an introduction to using Performance Schema to profile statement activity on your MySQL instances? Head on over to Detailed Profiling of SQL Activity in MySQL 5.6, which was recently published on the Dr.Dobbs site!

PlanetMySQL Voting: Vote UP / Vote DOWN

Easier Overview of Current Performance Schema Setting

Jesper Krogh — Mon, 20 May 2013 04:31:07 +0000

While I prepared for my Hands-On Lab about the Performance Schema at MySQL Connect last year, one of the things that occurred to me was how difficult it was quickly getting an overview of which consumers, instruments, actors, etc. are actually enabled. For the consumers things are made more complicated as the effective setting also depends on parents in the hierarchy. So my thought was: “How difficult can it be to write a stored procedure that outputs a tree of the hierarchies.” Well, simple enough in principle, but trying to be general ended up making it into a lengthy project and as it was a hobby project, it often ended up being put aside for more urgent tasks.

However here around eight months later, it is starting to shape up. While there definitely still is work to be done, e.g. creating the full tree and outputting it in text mode (more on modes later) takes around one minute on my test system – granted I am using a standard laptop and MySQL is running in a VM, so it is nothing sophisticated.

The current routines can be found in ps_tools.sql.gz – it may later be merged into Mark Leith’s ps_helper to try to keep the Performance Schema tools collected in one place.

Note: This far the routines have only been tested in Linux on MySQL 5.6.11. Particularly line endings may give problems on Windows and Mac.

Description of the ps_tools Routines

The current status are two views, four stored procedure, and four functions – not including several functions and procedures that does all the hard work:

Views:
- setup_consumers – Displays whether each consumer is enabled and whether the consumer actually will be collected based on the hierarchy rules described in Pre-Filtering by Consumer in the Reference Manual.
- accounts_enabled – Displays whether each account defined in the mysql.user table has instrumentation enabled based on the rows in performance_schema.setup_actors.
Procedures:
- setup_tree_consumers(format, color) – Create a tree based on setup_consumers displaying whether each consumer is effectively enabled. The arguments are:
  - format is the output format and can be either (see also below).:
    - Text: Left-Right
    - Text: Top-Bottom
    - Dot: Left-Right
    - Dot: Top-Bottom
  - color is whether to add bash color escapes sequences around the consumer names when used a Text format (ignored for Dot outputs).
- setup_tree_instruments(format, color, only_enabled, regex_filter) – Create a tree based on setup_instruments displaying whether each instrument is enabled. The tree is creating by splitting the instrument names at each /. The arguments are:
  - format is the output format and can be either:
    - Text: Left-Right
    - Dot: Left-Right
    - Dot: Top-Bottom
  - color is whether to add bash color escapes sequences around the instrument names when used a Text format (ignored for Dot outputs).
  - type – whether to base the tree on the ENABLED or TIMED column of setup_instruments.
  - only_enabled – if TRUE only the enabled instruments are included.
  - regex_filter – if set to a non-empty string only instruments that match the regex will be included.
- setup_tree_actors_by_host(format, color) – Create a tree of each account defined in mysql.user and whether they are enabled; grouped by host. The arguments are:
  - format is the output format and can be either:
    - Text: Left-Right
    - Dot: Left-Right
    - Dot: Top-Bottom
  - color is whether to add bash color escape sequences around the names when used a Text format (ignored for Dot outputs).
- setup_tree_actors_by_user – Create a tree of each account defined in mysql.user and whether they are enabled; grouped by username. The arguments are:
  - format is the output format and can be either:
    - Text: Left-Right
    - Dot: Left-Right
    - Dot: Top-Bottom
  - color is whether to add bash color escape sequences around the names when used a Text format (ignored for Dot outputs).
Functions:
- is_consumer_enabled(consumer_name) – Returns whether a given consumer is effectively enabled.
- is_account_enabled(host, user) – Returns whether a given account (host, user) is enabled according to setup_actors.
- substr_count(haystack, needle, offset, length) – The number of times a given substring occurs in a string. A port of the PHP function of the same name.
- substr_by_delim(set, delimiter, pos) – Returns the Nth element from a delimiter string.

The two functions substr_count() and substr_by_delim() was also described in an earlier blog.

The formats for the four stored procedures consists of two parts: whether it is Text or Dot and the direction. Text is a tree that can be viewed directly either in the mysql command line client (coloured output not supported) or the shell (colours supported for bash). Dot will output a DOT graph file in the same way as dump_thread_stack() in ps_helper. The direction is as defined in the DOT language, so e.g. Left-Right will have the first level furthest to the left, then add each new level to the right of the parent level.

Examples

As the source code – including comments – is more than 1600 lines, I will not discuss it here, but rather go through some examples.

setup_tree_consumers

Using the coloured output:

or the same using a non-coloured output:

setup_tree_instruments

Here a small part of the tree is selected using a regex.

setup_tree_actors_%

With only root@localhost and root@127.0.0.1 enabled, the outputs of setup_tree_actors_by_host and setup_tree_actors_by_user gives respectively:

DOT Graph of setup_instruments

The full tree of setup_instruments can be created using the following sequence of steps (I am using graphviz to get support for dot files):

MySQL 5.6.11$ echo -e "$(mysql -NBe "CALL ps_tools.setup_tree_instruments('Dot: Left-Right', FALSE, 'Enabled', FALSE, '')")" > /tmp/setup_instruments.dot
MySQL 5.6.11$ dot -Tpng /tmp/setup_instruments.dot -o /tmp/setup_instruments.png

The full output is rather large (6.7M). If you want to see if you can get to it at http://mysql.wisborg.dk/wp-content/uploads/2013/05/setup_tree_instruments_dot_lr.png.

Views

mysql> SELECT * FROM ps_tools.setup_consumers;
+--------------------------------+---------+----------+
| NAME                           | ENABLED | COLLECTS |
+--------------------------------+---------+----------+
| events_stages_current          | NO      | NO       |
| events_stages_history          | NO      | NO       |
| events_stages_history_long     | NO      | NO       |
| events_statements_current      | YES     | YES      |
| events_statements_history      | NO      | NO       |
| events_statements_history_long | NO      | NO       |
| events_waits_current           | NO      | NO       |
| events_waits_history           | NO      | NO       |
| events_waits_history_long      | NO      | NO       |
| global_instrumentation         | YES     | YES      |
| thread_instrumentation         | YES     | YES      |
| statements_digest              | YES     | YES      |
+--------------------------------+---------+----------+
12 rows in set (0.00 sec)

mysql> SELECT * FROM ps_tools.accounts_enabled;
+-------------+-----------+---------+
| User        | Host      | Enabled |
+-------------+-----------+---------+
| replication | 127.0.0.1 | NO      |
| root        | 127.0.0.1 | YES     |
| root        | ::1       | NO      |
| meb         | localhost | NO      |
| memagent    | localhost | NO      |
| root        | localhost | YES     |
+-------------+-----------+---------+
6 rows in set (0.00 sec)

Conclusion

There is definitely more work to do on making the Performance Schema easier to access. ps_helper and ps_tools are a great start to what I am sure will be an extensive library of useful diagnostic queries and tools.

PlanetMySQL Voting: Vote UP / Vote DOWN

Experimenting with MySQL 5.7

Morgan Tocker — Mon, 20 May 2013 04:00:00 +0000

I was playing around with MySQL 5.7 this weekend and before having read the changelog, I managed to spot these two little gems.

Duplicate Indexes

“The server now issues a warning if an index is created that duplicates an existing index, or an error in strict SQL mode.” Bug #37520

Example Testcase:

mysql> SHOW CREATE TABLE city\G
*************************** 1. row ***************************
       Table: city
Create Table: CREATE TABLE `city` (
  `ID` int(11) NOT NULL AUTO_INCREMENT,
  `Name` char(35) NOT NULL DEFAULT '',
  `CountryCode` char(3) NOT NULL DEFAULT '',
  `District` char(20) NOT NULL DEFAULT '',
  `Population` int(11) NOT NULL DEFAULT '0',
  PRIMARY KEY (`ID`),
  KEY `CountryCode` (`CountryCode`),
  CONSTRAINT `city_ibfk_1` FOREIGN KEY (`CountryCode`) REFERENCES `Country` (`Code`)
) ENGINE=InnoDB AUTO_INCREMENT=4080 DEFAULT CHARSET=latin1
1 row in set (0.00 sec)

mysql> ALTER TABLE city add index (countrycode);
ERROR 1831 (HY000): Duplicate index 'CountryCode_2' defined on the table 'world.city'. 
This is deprecated and will be disallowed in a future release.

Pretty cool - I know this previously caught a lot of people.

Control-C support in the client

“Previously, Control+C in mysql interrupted the current statement if there was one, or exited mysql if not. Now Control+C interrupts the current statement if there was one, or cancels any partial input line otherwise, but does not exit.” Bug #66583

Example Testcase:

mysql> this is a test
    -> test
    -> test
    -> ^C

So if I want to quit, I can now control-C then type “quit”. This is much more intuitive.

PlanetMySQL Voting: Vote UP / Vote DOWN

What's the deal with NoSQL?

Anders Karlsson — Sun, 19 May 2013 18:49:00 +0000

Everybody seems to be looking at and debating NoSQL these days, and so am I and I thought I'd say a few words about it. Which is not to say I haven't said stuff before, bit them I was mainly targeting specific attributes of many NoSQL solutions (like "eventual consistency" or, as you might call it, "instant inconsistency", What I was opposing is that "eventual consistency" has anything to do with just that, consistency. Rather, what this means is that at any point in time the system is inconsistent, and even if it might be consistent, you cannot rely on it being so. Which is fine, but don't call it consistency, call it inconsistency. Allowing a database to be somewhat inconsistent doesn't necessarily mean that it's something wrong with it).

All this said, what is going on here, why are we MySQL and MariaDB users seeing so many MongoDB, Cassandra and LevelDB applications pop up? Come on, these are typically less functional implementations of a database than even the most basic MySQL setup? No transactions, no joins, no standards etc. etc. And the answer, if you want to hear what I have to say, is ease of use. So let's explore that a bit.

Following the Object Orientation frenzy of the 1990s, when any application project ended up consisting of endless sessions modeling objects, usually involving expensive consultants, dresses in expensive, blue suits. And when that was done (which took years!) you had a way cool object model, but no money left to do the actual implementation, i.e. do the real programming (shiver), and you went to some other project and the nicely dressed object design consultant left to see another OO sucker.

Now, objects are much more standard, even non-OO languages have a big chunk of OO features, and these are used enhance programmer productivity and better code and design. Which is fine (except that if you were one of those OO consultants, which means you are now out of a job, as such mundane tasks of writing is not something you would ever do, such dirty stuff is better left to "programmers". Oh no, I forgot that you are now an ITIL consultant, that just slipped my mind) but how does this relate to MySQL and MariaDB. The answer is that MySQL, which was once considered real easy to use, no longer is as easy as it used to be. The Relational data model is still brilliant when you look at data as data, and that is how many of us look at it, so we go through the process of mapping data to objects, if that is what it takes. SQL and Java, PHP or whatever merges, and the application now contains a layer mapping objects to real data. Or we use hibernate, which does this automatically for us.

But a new cadre of developers are emerging, and they look at OO as natural and they look at objects as data (it's not. Data, in my mind, should be independent from the application using it, objects on the other hand, are closely tied to the application at hand). With which I do not mean that there is something wrong with building applications using objects, quite the opposite. But if all you know is objects, then using relational technology turns difficult, and SQL, for all the good things with it, seems old-fashioned and arcane, which it is (but it is so widely used you cannot avoid it). So you go with something that looks at objects as all you need, and present that in some object format. Like JSON.

And again, there is nothing wrong with that. But if we who are on the SQL and Relational track just discards these NoSQL technologies, we are not making any friends. We have to accept that MySQL and MariaDB really aren't that easy to use anymore, at least not for newcomers.

And then there is another thing: Some data, like Big Data, has attributes that really doesn't fit well in a relational model. Data where the attribute of a value can't easily be determined once and for all, and you need to reprocess that data (large test objects, images and maps are some examples). In this case, you really need to extend the relational model, somehow.

But SQL-based relational isn't going away. The Relational model is still one of the best ways to look at data, it's just that we also need some other ways of looking at data. And it needs to be easier to access. And we shouldn't really have to push SQL down the throat of every single developer, trying to develop an application using some OO technology. The answer is we need both. And these technologies needs to interoperate. I want to use SQL for my data. But I also want JSON and REST for my data. And there shouldn't be much of a performance overhead. All in all, we SQL folks need to wake up and data easier to use again. We know data better than the Cassandra and MongoDB folks. We know transactions better than them too. But they know how to work with developers who doesn't know who The Beatles were and make Relational easy to use for them, without them having to learn JSON (and now having to listen to a tirade about todays youngsters not knowing what real music is and that it died with John Lennon! What? You don't know who John Lennon was! That's exactly what I mean, you have no taste at all!).

Just my 2 cents...

/Karlsson
PlanetMySQL Voting: Vote UP / Vote DOWN

HeidiSQL 8.0 released

Ansgar Becker — Sun, 19 May 2013 09:11:52 +0000

343 revisions after the 7.0 release follows the new 8.0 release of HeidiSQL now.

Get it from the download page.

Here are the most noticable changes:

- Available in 23 languages now. Thanks to all translators and Transifex hereby!
- Database tree: Introduce optional folders for tables, views, routines etc.
- Introduce session folders in session manager.
- Make routine editor work on MS SQL servers.
- Support search and replace in data and query results.
- Add support for microseconds in temporal datatypes of MariaDB 5.3+ and MySQL 5.6.
- Introduce a query history, available in the right side helpers box. Can be turned off.
- Implement grid export as PHP array.
- Host > Variables: Add "Global" column, and highlight values different to their session pendant
- Add menu item for launching mysql.exe command line with current parameters.

... and more:
- Table editor: Fix handling of BIT default values, and support BIT columns in MS SQL.
- Table editor: Improve selection of ENUM and SET default values
- User manager: Support dots in database and table privileges
- Data grids: Support copying/pasting NULL values
- Fix stripped backslashes in VIEW body editor
- Apply hotkeys to dialog buttons
- Grid export: Remove zero padding to avoid octal => integer conversion in PHP
- Data grid: Propose column names from selected table in filter panel
- Database and new table filter above database tree
- Table editor: Display number of selected columns in status bar
- Database tree: Indicate previously selected tables with a non-ghosted icon in the tree, while leaving never selected ones ghosted
- Display timestamp in very right status bar panel when executing a query
- Table editor: Add missing DATE and TIME datatypes for MS SQL
- Table editor: Support old style "TYPE BTREE" in table index code
- Routine editor: Finally fix ramshackle detection of routine body
- Data grid: Make foreign values drop down optionally
- Dialogs: Introduce "KeepAskingSetting" checkbox
- Session manager: Move startup script and local time zone options together with SSL settings to a new "Advanced" tab
- SQL export: Support filename and dirname patterns in export target combobox
- Database tree: Display overlay icons for some special table engines like federated, csv, aria and performance_schema
- Implement an automatic keep-alive ping, to prevent SSH tunnels from disconnecting
- Add support for renaming tables in MS SQL
- Fix crash on exit when connected to pre-4.1 servers
- Table editor: Enhance MS SQL compatibility in table editor
- Fix and enhance handling of multiple statements and multiple results
- Grid export: Add "Include query" and "Include auto increment column" checkbox options
- Processlist: Add link label "EXPLAIN Analyzer on MariaDB.org"
- Internal: Refactor logic for reading and writing application and session settings
- Session manager: Introduce new columns "Last connect" and "Counter"
- Extend the variable editor to explicitly modify strings, numbers, booleans or enumerations
- Detect client timezone and send SET time_zone to the server, so that NOW() and friends return UTC-fixed values
- Session manager: Add server specific icons for TokuDB, InfiniDB and Infobright
- Session handling: Use home brown file format for exporting and importing registry settings, as used for the portable version
- Implement usage of mysql_warning_count(). Ask for running SHOW WARNINGS in a new query tab.
- Fix command line for Wine users
- Introduce new preference option "Prefill empty date/time fields".
- Restore previous selection after refreshing process list (and neighbor tabs)

PlanetMySQL Voting: Vote UP / Vote DOWN

MySQL 5.6, InnoDB and fast storage

Mark Callaghan — Sat, 18 May 2013 22:03:00 +0000

I used a simple workload with sysbench to determine the rate at which InnoDB can read blocks from disk. The workload is read-only and each query fetches 1 row by PK. The workload was IO-bound with a 2G InnoDB buffer pool and 32G database. Storage was fast courtesy of buffered IO and enough RAM to cache the database in the OS filesystem cache.

Using MySQL 5.6.11 and InnoDB with a few hacks the peak throughput was about 240,000 QPS and 210,000 block reads/second. The test server has 32 cores (16 physical cores, 32 logical cores with HT enabled). This is a great result that can probably be even better. Contention on fil_system->mutex was the bottleneck and I think that can be improved (see feature request #69276). I wonder if 400,000 block reads/second is possible?

A few years back, in 2009 or 2010, I ran similar tests using a server with 8 physical cores. I think HT was disabled. Using MySQL 5.1 with the Facebook patch I was able to get about 40,000 QPS and 35,000 block reads/second. It is good to see software advance to keep up with hardware.
PlanetMySQL Voting: Vote UP / Vote DOWN

New Feature Qualification

Anitha Gopi — Sat, 18 May 2013 11:00:00 +0000

Early this year Oracle released MySQL 5.6 - Best MySQL Release Ever. This release delivered not only quality, but also quantity in terms of number of features. See a comprehensive list here . The blogs below also refer to the massive changes introduced in 5.6
http://www.mysqlperformanceblog.com/2013/01/27/mysql-5-6-improvements-in-the-nutshell/
http://www.flamingspork.com/blog/2013/03/05/mysql-code-size/

It is no mean task to deliver so many features with high quality that too for a feature rich product like MySQL. This was made possible by the increased focus given to testing during 5.6 development . Testing was integrated far better than in the past with MySQL processes and test team was grown significantly. After the first few milestone releases developers and other stake holders started seeing benefits of the new processes and that resulted in better cooperation between development and test teams which in turn resulted in a bunch of happy testers :). In this post I will take you through the process that was established for new feature qualification.

Features are delivered through milestones which Tomas Ulin has explained very well at http://insidemysql.com/the-milestone-release-model-revisited/. MySQL Server QA has the following goals for new features

Complete functional and non functional test coverage of changed and new functionality
No regressions
More than 80% Code Coverage

QA involvement starts as soon as the requirements and specifications of the feature are finalized by the development team. QA reviews available documents and provides feedback on the design, usability, testability etc. A discussion follows with the developer and changes are made as needed to ensure that the feature can be tested.

Once the specifications and requirements are acceptable QA starts working on the test plan. First QA documents all scenarios that needs to be tested. This includes stand alone tests, integration tests, non functional tests etc. Next step is to identify tools for automating the scenarios. Most commonly used functional test tools are

MTR(mysql-test-run)- Used to cover simple to medium complexity test cases
RQG(Random Query Generator) - This is a very powerful tool and is used for concurrency tests, generating large number of SQL queries etc

MTR based tests are added by default for any new feature. RQG use started from 5.5 and it gained wide acceptance in 5.6. At least 90% of the features in 5.6 have gone through RQG testing and this helped weed out many bugs which would have been nearly impossible to find using MTR.

While the developers are working on the product code , QA engineers start working on the automated tests, test infrastructure improvements etc.Final round of testing starts after the feature has passed code reviews. This phase can last anywhere between a few days to months depending on the complexity of the feature, stability of the code etc.

Features gets signed off only when the following conditions are met:

No open bugs in the new feature - This is very strictly enforced and even minor bugs are not allowed. We believe that bugs are easiest to fix when the code is new and hence this can help us deliver features that are of high quality.
No regressions - A feature gets developed on a tree which gets tested regularly through a continuous integration testing tool. Any regressions are detected and fixed before sign off.
Acceptable Code coverage numbers - Code coverage report is generated for the changed lines of code and the minimum expected coverage is 80%. Most features have coverage of more than 90%. Any uncovered lines of code are analyzed and wherever possible new tests are added to increase code coverage.
All new tests are added to the automated regression suite.
All planned testing has completed with satisfactory results.

This process worked really well in 5.6 and we will use our experiences to improve this further and deliver an even better 5.7 !!

PlanetMySQL Voting: Vote UP / Vote DOWN

MySQL binlogs - Don't forget to do your homework!

Phil Hildebrand — Sat, 18 May 2013 04:34:00 +0000

Now that I'm back doing just database stuff, I've come to realize I've gotten a little sloppy about doing my homework. Homework's never been my favorite thing in the world, but it often reduces stress when your under the gun during an outage or upgrade... We had a MySQL database server that's been slow on DML changes, and based on the slowest statements being 'COMMIT', we had a good mind
PlanetMySQL Voting: Vote UP / Vote DOWN

Percona XtraBackup 2.1.2 for MySQL available for download

MySQL Performance Blog — Sat, 18 May 2013 04:25:53 +0000

Percona is glad to announce the release of Percona XtraBackup 2.1.2 for MySQL on May 18, 2013. Downloads are available from our download site here and Percona Software Repositories.

This release fixes number of high-priority bugs since version 2.1 became GA. It’s advised to upgrade your latest 2.1 version to 2.1.2. This release is the latest stable release in the 2.1 series.

Bugs Fixed:

Using Perl’s DBD::MySQL package for server communication instead of spawning the MySQL command line client introduced a regression which caused innobackupex –galera-info option to fail. Bug fixed #1180672.
The format of xtrabackup_galera_info was missing the ‘:’ separator between the values of wsrep_local_state_uuid and wsrep_last_committed. Bug fixed #1181222.
innobackupex automatic version detection did not work correctly for latest Percona Server and MySQL 5.1 releases which could cause innobackupex to fail. Bugs fixed #1181092, #1181099 and #1180905.
When backing up a server that is not a replication slave with the innobackupex –slave-info option, innobackupex failed with a fatal error. Replaced the fatal error with a diagnostic message about innobackupex –slave-info being ignored in such a case. Bug fixed #1180662.
Low values for wait_timeout on the server could cause server to close the connection while backup is being taken. Fixed by setting the bigger value for wait_timeout option on the server to prevent server from closing connections if the global wait_timeout value is set too low. Bug fixed #1180922.

Other bug fixes: bug fixed #1177182.

Release notes with all the bugfixes for Percona XtraBackup 2.1.2 are available in our online documentation. Bugs can be reported on the launchpad bug tracker.

The post Percona XtraBackup 2.1.2 for MySQL available for download appeared first on MySQL Performance Blog.

PlanetMySQL Voting: Vote UP / Vote DOWN

OurSQL Episode 140: More Performance

Technocation — Fri, 17 May 2013 18:20:13 +0000

This week we explain performance_schema a bit deeper. In Ear Candy, we talk about max_binlog_cache_size and At the Movies presents Max Mether of SkySQL talking about "High Availability Solutions for MySQL".

Events
DB Hangops - every other Wednesay at noon Pacific time
Upcoming MySQL events

Training
SkySQL Trainings
Tungsten University trainings

PlanetMySQL Voting: Vote UP / Vote DOWN

Log Buffer #320, A Carnival of the Vanities for DBAs

The Pythian Group — Fri, 17 May 2013 12:23:12 +0000

The red carpet has been laid down at this Log Buffer Edition, and you can witness and cheer the cat-walking blog posts from Oracle, SQL Server and MySQL. Every one of them is chic, elegant, sensual in its own right. Enjoy.
Oracle:

Create colored heat maps in SQL*Plus with Kyle Hailey.

Here’s a quick and dirty script to create a procedure (in the SYS schema – so be careful) to check the Hakan Factor for an object.

Connor has a good post about default null for collection parameter.

This is yet another blogpost on Oracle’s direct path read feature which was introduced for non-parallel query processes in Oracle version 11.

Owen Allen has seen some questions about provisioning Oracle Solaris 11. They boil down to this.

SQL Server:

Shashank Srivastava tells us as how to Change the SQL Server Instance Name after Renaming the Windows Host.

Daniel Calbimonte shares as how to synchronize two SSAS Servers.

Data Architecture underpins just about everything we do in IT. Without a clear understanding of how data is structured, there is no reliable way to derive meaning from it.

Orlando Colamatteo is login-less in Seattle.

Lets get started testing database with tSQLt with Robert Sheldon.

MySQL:

After a lot of fuzz, Anders Karlsson is now releasing MyQuery version 3.5.1.

Nothing like reestablishing a tradition and Dave Stokes is doing just that for MySQL.

Mare Alff is spreading the word about the performance schema.

Slava Akhmechet talks about secondary indexes, batched inserts performance improvements, soft durability mode.

It is a central part of the MySQL philosophy to try and help you as much as you can. There are many occasions when it could tell you that what you are asking for is utterly stupid or give you a bad execution plan because “you asked for it”.

PlanetMySQL Voting: Vote UP / Vote DOWN

Replication Enhancements in MySQL 5.7: SHOW SLAVE STATUS NONBLOCKING

Nuno Carvalho — Fri, 17 May 2013 11:17:00 +0000

2013 is on it's initial months and we already have 5.6 GA and the first release of 5.7, 5.7.1 DMR with lots of exciting new features, future is promising!

A new feature added in replication was the NONBLOCKING option to SHOW SLAVE STATUS command.

In the past if we stop slave during a big transaction, until the transaction was applied we cannot execute SHOW SLAVE STATUS to see the slave progress. The latter operation would block until the former finish, this would disable all external monitoring or third-party applications that need a immediate response from server.

To solve this problem, NONBLOCKING option was added to SHOW SLAVE STATUS, when it is used we can obtain immediate response from server about slave progress. As a tradeoff, the returned info may not be the latest data but it won't block waiting for STOP SLAVE.

This enhancement and many more can be downloaded from MySQL development releases page.
PlanetMySQL Voting: Vote UP / Vote DOWN

Virident vCache vs. FlashCache: Part 2

MySQL Performance Blog — Fri, 17 May 2013 10:00:56 +0000

This is the second part in a two-part series comparing Virident’s vCache to FlashCache. The first part was focused on usability and feature comparison; in this post, we’ll look at some sysbench test results.

Disclosure: The research and testing conducted for this post were sponsored by Virident.

First, some background information. All tests were conducted on Percona’s Cisco UCS C250 test machine, and both the vCache and FlashCache tests used the same 2.2TB Virident FlashMAX II as the cache storage device. EXT4 is the filesystem, and CentOS 6.4 the operating system, although the pre-release modules I received from Virident required the use of the CentOS 6.2 kernel, 2.6.32-220, so that was the kernel in use for all of the benchmarks on both systems. The benchmark tool used was sysbench 0.5 and the version of MySQL used was Percona Server 5.5.30-rel30.1-465. Each test was allowed to run for 7200 seconds, and the first 3600 seconds were discarded as warmup time; the remaining 3600 seconds were averaged into 10-second intervals. All tests were conducted with approximately 78GiB of data (32 tables, 10M rows each) and a 4GiB buffer pool. The cache devices were flushed to disk immediately prior to and immediately following each test run.

With that out of the way, let’s look at some numbers.

vCache vs. vCache – MySQL parameter testing

The first test was designed to look solely at vCache performance under some different sets of MySQL configuration parameters. For example, given that the front-end device is a very fast PCIe SSD, would it make more sense to configure MySQL as if it were using SSD storage or to just use an optimized HDD storage configuration? After creating a vCache device with the default configuration, I started with a baseline HDD configuration for MySQL (configuration A, listed at the bottom of this post) and then tried three additional sets of experiments. First, the baseline configuration plus:
innodb_read_io_threads = 16 innodb_write_io_threads = 16
We call this configuration B. The next one contained four SSD-specific optimizations based partially on some earlier work that I’d done with this Virident card (configuration C):
innodb_io_capacity = 30000 innodb_adaptive_flushing_method = keep_average innodb_flush_neighbor_pages=none innodb_max_dirty_pages_pct = 60
And then finally, a fourth test (configuration D) which combined the parameter changes from tests B and C. The graph below shows the sysbench throughput (tps) for these four configurations:

As we can see, all of the configuration options produce numbers that, in the absence of outliers, are roughly identical, but it’s configuration C (shown in the graph as the blue line – SSD config) which shows the most consistent performance. The others all have assorted performance drops scattered throughout the graph. We see the exact same pattern when looking at transaction latency; the baseline numbers are roughly identical for all four configurations, but configuration C avoids the spikes and produces a very constant and predictable result.

vCache vs. FlashCache – the basics

Once I’d determined that configuration C appeared to produce the most optimal results, I moved on to reviewing FlashCache performance versus that of vCache, and I also included a “no cache” test run as well using the base HDD MySQL configuration for purposes of comparison. Given the apparent differences in time-based flushing in vCache and FlashCache, both cache devices were set up so that time-based flushing was disabled. Also, both devices were set up such that all IO would be cached (i.e., no special treatment of sequential writes) and with a 50% dirty page threshold. Again, for comparison purposes, I also include the numbers from the vCache test where the time-based flushing is enabled.

As we’d expect, the HDD-only solution barely registered on the graph. With a buffer pool that’s much smaller than the working set, the no-cache approach is fairly crippled and ineffectual. FlashCache does substantially better, coming in at an average of around 600 tps, but vCache is about 3x better. The interesting item here is that vCache with time-based flushing enabled actually produces better and more consistent performance than vCache without time-based flushing, but even at its worst, the vCache test without time-based flushing still outperforms FlashCache by over 2x, on average.

Looking just at sysbench reads, vCache with time-based flushing consistently hit about 27000 per second, whereas without time-based flushing it averaged about 12500. FlashCache came in around 7500 or so. Sysbench writes came in just under 8000 for vCache + time-based flushing, around 6000 for vCache without time-based flushing, and somewhere around 2500 for FlashCache.

We can take a look at some vmstat data to see what’s actually happening on the system during all these various tests. Clockwise from the top left in the next graph, we have “no cache”, “FlashCache”, “vCache with no time-based flushing”, and “vCache with time-based flushing.” As the images demonstrate, the no-cache system is being crushed by IO wait. FlashCache and vCache both show improvements, but it’s not until we get to vCache with the time-based flushing that we see some nice, predictable, constant performance.

So why is it the case that vCache with time-based flushing appears to outperform all the rest? My hypothesis here is that time-based flushing allows the backing store to be written to at a more constant and, potentially, submaximal, rate compared to dirty-page-threshold flushing, which kicks in at a given level and then attempts to flush as quickly as possible to bring the dirty pages back within acceptable bounds. This is, however, only a hypothesis.

vCache vs. FlashCache – dirty page threshold

Finally, we examine the impact of a couple of different dirty-page ratios on device performance, since this is the only parameter which can be reliably varied between the two in the same way. The following graph shows sysbench OLTP performance for FlashCache vs. vCache with a 10% dirty threshold versus the same metrics at a 50% dirty threshold. Time-based flushing has been disabled. In this case, both systems produce better performance when the dirty-page threshold is set to 50%, but once again, vCache at 10% outperforms FlashCache at 10%.

The one interesting item here is that vCache actually appears to get *better* over time; I’m not entirely sure why that’s the case or at what point the performance is going to level off since these tests were all run for 2 hours anyway, but I think the overall results still speak for themselves, and even with a vCache volume where the dirty ratio is only 10%, such as might be the case where a deployment has a massive data set size in relation to both the working set and the cache device size, the numbers are encouraging.

Conclusion

Overall, the I think the graphs speak for themselves. When the working set outstrips the available buffer pool memory but still fits into the cache device, vCache shines. Compared to a deployment with no SSD cache whatsoever, FlashCache still does quite well, massively outperforming the HDD-only setup, but it doesn’t even really come close to the numbers obtained with vCache. There may be ways to adjust the FlashCache configuration to produce better or more consistent results, or results that are more inline with the numbers put up by vCache, but when we consider that overall usability was one of the evaluation points and combine that with the fact that the best vCache performance results were obtained with the default vCache configuration, I think vCache can be declared the clear winner.

Base MySQL & Benchmark Configuration

All benchmarks were conducted with the following:
sysbench --num-threads=32 --test=tests/db/oltp.lua --oltp_tables_count=32 \ --oltp-table-size=10000000 --rand-init=on --report-interval=1 --rand-type=pareto \ --forced-shutdown=1 --max-time=7200 --max-requests=0 --percentile=95 \ --mysql-user=root --mysql-socket=/tmp/mysql.sock --mysql-table-engine=innodb \ --oltp-read-only=off run

The base MySQL configuration (configuration A) appears below:
#####fixed innodb options innodb_file_format = barracuda innodb_buffer_pool_size = 4G innodb_file_per_table = true innodb_data_file_path = ibdata1:100M innodb_flush_method = O_DIRECT innodb_log_buffer_size = 128M innodb_flush_log_at_trx_commit = 1 innodb_log_file_size = 1G innodb_log_files_in_group = 2 innodb_purge_threads = 1 innodb_fast_shutdown = 1 #not innodb options (fixed) back_log = 50 wait_timeout = 120 max_connections = 5000 max_prepared_stmt_count=500000 max_connect_errors = 10 table_open_cache = 10240 max_allowed_packet = 16M binlog_cache_size = 16M max_heap_table_size = 64M sort_buffer_size = 4M join_buffer_size = 4M thread_cache_size = 1000 query_cache_size = 0 query_cache_type = 0 ft_min_word_len = 4 thread_stack = 192K tmp_table_size = 64M serverid = 101 key_buffer_size = 8M read_buffer_size = 1M read_rnd_buffer_size = 4M bulk_insert_buffer_size = 8M myisam_sort_buffer_size = 8M myisam_max_sort_file_size = 10G myisam_repair_threads = 1 myisam_recover

The post Virident vCache vs. FlashCache: Part 2 appeared first on MySQL Performance Blog.

PlanetMySQL Voting: Vote UP / Vote DOWN

MyQuery 3.5.1 beta released!

Anders Karlsson — Thu, 16 May 2013 22:38:00 +0000

After a lot of fuzz, I am now releasing MyQuery version 3.5.1. This version introduces one big feature, a brand new Dictionary viewer. In addition to that, there are numerous bug fixes and the removal of one feature, which is the option to run with just 1 connection: In this version, 2 connections will always be used, and I have some good reasons to remove this as being optional, fact is, running with 1 connection was hard to diagnose, caused a lot of problems, and had no real benefit actually, just drawbacks.

So, for you Windows users, MyQuery 3.5.1 is now out there, but it is really a beta. The beta is caused by the new Dictionary viewer, the rest should be pretty stable.

Download it from sourceforge.

Happy SQLing
/Karlsson
PlanetMySQL Voting: Vote UP / Vote DOWN

Reestablishing a MySQL Tradition

Dave Stokes — Thu, 16 May 2013 16:45:06 +0000

Every so often you see something from the past and wonder “Why don’t we do that anymore?” Well, in this case it was a former co-worker wearing his MySQL Contributor shirt.

This is Antony Curtis in one of the original MySQL Community Contributor shirt

So the MySQL Community Team had a quick meeting and the result is that we are reestablishing the tradition. So if you have a signed Oracle Contributor Agreement and have contributed to MySQL, you should have in your inbox a request for your shirt size and a shipping address. If you do not see an email and you qualify for a short, let us know (we probably have an old email on record for you). And if you are working on some code for MySQL and have that OCA ready but not filed, please expedite your actions so you do not miss out on this batch.

PlanetMySQL Voting: Vote UP / Vote DOWN

Understanding Tokutek Fractal Tree Indexes

Effective MySQL — Thu, 16 May 2013 14:40:45 +0000

Download PDF Presentation

Thanks to Tim Callaghan for speaking Tuesday night at the Effective MySQL New York meetup on Fractal Tree Indexes : Theory and Practice (MySQL and MongoDB). There was a good turnout and a full room to learn how the TokuDB storage engine from Tokutek is changing how to handle big data in MySQL.

Also interesting is how the same technology has been applied for use in MongoDB including giving MongoDB transactions; a big change for NoSQL.

PlanetMySQL Voting: Vote UP / Vote DOWN