This seems like an appropriate place to mention a few words about the conference organization. The number of people involved is staggering (100+). The logistics — the number of tasks, vendors, contracts, and so on — blows the mind. The upfront cost is literally unmentionable. It’s an exponentially bigger deal in every way than any of the conferences we’ve done before. There is no way to explain it to anyone who isn’t involved. I don’t even comprehend it myself.

Despite this, we are working hard to ensure that the traditions we’ve known and loved for years are continued. It turns out that they all have a very high cost in real dollars. We are constantly faced with hard decisions that always involve “how are we going to pay for this?” I wish that, for example, it could be simple and lightweight to organize BOFs. But there are unions and contracts and room minimums and overtime pay and bundled quotes and restrictions every which way. Nothing is simple, nothing is easy, nothing is cheap — regardless of how simple, easy, and cheap it should be.

And yet, there will be BOFs and dot-org booths and Drizzle Day and all the rest. And the ticket price is a lot less than it was last year. If you appreciate this, you can help by getting attendees to come. Please promote the conference to everyone you know, in every way you know. Use Twitter, LinkedIn, Facebook, everything. Blog about it. Mention it in-person. Ask your boss to send you and your colleagues, or send your team if you’re the boss. Send email to your meetups and user groups and mailing lists, and ask them to promote it to their networks too. Enough said.

Further Reading:

• There will be an O’Reilly MySQL Conference in April 2010
• I’ll be speaking at the O’Reilly MySQL Conference 2010
• Speaking at EdUI Conference 2009
• The MySQL Conference will be very good this year
• Postgres folks, consider the 2011 MySQL conference

Based on MySQL 5.5.20, including all the bug fixes in it, Percona Server 5.5.20-24.1 is now the current stable release in the 5.5 series. All of Percona ‘s software is open-source and free, all the details of the release can be found in the 5.5.20-24.1 milestone at Launchpad.

Drizzle Day 2012

read more

Benjamin Wood talks at the Dallas MySQL Tech tour on the history of MySQL

Benjamin Wood started with a presentation on the history of MySQL and the changes in the product over the last few releases. Craig Sylvester showed how to use embedded MySQL. Then Benjamin capped off the event with a presentation on database monitoring and performance tuning. The event did go slightly over scheduled time due to an extended question and answer period following the presentations.

Thanks to all who attended.

Craig Sylvester talking at the Dallas MySQL Tech Tour about embedding MySQL

For MySQL Tech Tour events in your ares, click here.

For those of you who don’t know what this is, it’s another of the O’Reilly traditions we’re trying to continue. (We are trying very hard to make this event as close to a clone of O’Reilly’s as we can.) It’s a free table in the expo hall where people who participate in a non-commercial open source project can exhibit. I organized a Maatkit booth a few times in the past, and was always really grateful to O’Reilly for making the space available. Space in the expo hall is at a premium, but we think that these dot-org booths are even more valuable to the open-source projects and the conference attendees.

So, please tell your friends who care about open source, and ask them to tell their friends too. Let’s get some great open-source projects into the expo hall, alongside the commercial vendors!

Please RSVP below by leaving a comment so we can plan on a) enough pizza, b) any special pizza topping request, and c) have enough meeting space for the pizza.

Oracle Office
6031 Connection Drive
Irving, TX 75039

So I love and need only binary tar balls. Installation and removal is done within seconds and no remainings are left over after a removal. To operate the whole I use myenv.
Some software providers unfortunately do not provide binary tar balls at all or not in the form I want and need them. Thus I was thinking about how to get those by extracting them from packages. Up to now I have not had the time to write this down. But today was the right time...

RPM

rpm2cpio galera-22.1.1-1.rhel5.x86_64.rpm | cpio -vidm
tar czf galera-22.1.1-1.rhel5.x86_64.tar.gz usr
rm -rf usr

Extract with:

tar xf galera-22.1.1-1.rhel5.x86_64.tar.gz

DEB

ar vx galera-22.1.1-amd64.deb
mv data.tar.gz galera-22.1.1-amd64.deb.tar.gz
rm debian-binary control.tar.gz

Extract with:

tar -mxf galera-22.1.1-amd64.deb.tar.gz

The packages look quite the same in size:

-rw-r--r-- 1 oli oli 6725416 2012-02-08 13:49 galera-22.1.1-1.rhel5.x86_64.rpm
-rw-r--r-- 1 oli oli 6769606 2012-02-08 14:18 galera-22.1.1-1.rhel5.x86_64.tar.gz

-rw-r--r-- 1 oli oli 1386762 2011-12-12 17:12 galera-22.1.1-amd64.deb
-rw-r--r-- 1 oli oli 1385994 2012-02-08 14:18 galera-22.1.1-amd64.deb.tar.gz

so I assume that there is nothing lost.

The differences in size between DEB and RPM seems to come from the packaging itself:

usr_deb/lib/galera/libgalera_smm.so:   ELF 64-bit (SYSV), dynamically linked, stripped
usr_rpm/lib64/galera/libgalera_smm.so: ELF 64-bit (SYSV), dynamically linked, not stripped

So nothing to worry. The programs itself worked without any problems after the first tests. So I am optimistic that this is a good workaround until I can convince the software vendor to make good binary tar balls...

• QueryScript: server side scripting is now supported by common_schema, which acts as an interpreter for QueryScript code.
• Throttling for queries is now made available via the throttle() function.
• Enhancements to processlist-related views, including the new slave_hosts view.
• Inline documentation/help is available via the help() routine.
• more...

QueryScript

common_schema makes for a QueryScript implementation for MySQL. You can run server side scripts, interpreted by common_schema, which allow for easy syntax and greater power than was otherwise previously available on the MySQL server. For example:

foreach($table, $schema, $engine: table like '%')
  if ($engine = 'ndbcluster')
    ALTER ONLINE TABLE :$schema.:$table REORGANIZE PARTITION;

QueryScript includes flow control, conditional branching, variables & variable expansion, script throttling and more.

Read more on common_schema's QueryScript implementation.

Query throttling

Throttling for MySQL queries was suggested by means of elaborate query manipulation. It is now reduced into a single throttle function: one can now just invoke throttle(3) on one's query, so as to make the query execute for a longer time, while taking short sleep breaks during operation, easing up the query's demand for resources.

Read more on query throttling.

Process views

The processlist_grantees view provides with more details on the running processes. slave_hosts is a new view, listing hostnames of connected slaves.

Read more on process views.

help()

The common_schema documentation is now composed of well over 100 pages, including synopsis, detailed internals discussion, notes and examples. I can't exaggerate in saying that the documentation took the vast majority of time for this code to release.

The documentation is now made available inline, from within you mysql client, via the help() routine. Want to know more about redundant (duplicate) keys and how to find them? Just type:

call help('redundant');

and see what comes out!

The entire documentation, which is available online as well as a downloadable bundle, is embedded into common_schema itself. It's rather cool.

Tests

common_schema is tested. The number of tests in common_schema is rapidly growing, and new tests are introduced for new features as well as for older ones. There is not yet full coverage for all views, but I'm working hard at it. common_schema is a robust piece of code!

Get it!

Download common_schema on the common_schema project page.

Read the documentation online, or download it as well (or call for help())

common_schema is released under the BSD license.

QueryScript is available fro MySQL via common_schema, which adds MySQL-specific usage.

What does QueryScript look like? Here are a few code samples:

Turn a bulk DELETE operation into smaller tasks. Throttle in between.

while (DELETE FROM archive.events WHERE ts < CURDATE() LIMIT 1000)
{
  throttle 2;
}

Convert all InnoDB tables in the 'sakila' database to compressed format:

foreach ($table, $schema, $engine: table in sakila)
{
  if ($engine = 'InnoDB')
    ALTER TABLE :$schema.:$table ENGINE=InnoDB ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8;
}

Shard your data across multiple schemata:

foreach($shard: {USA, GBR, JAP, FRA})
{
  CREATE DATABASE db_:$shard;
  CREATE TABLE db_:$shard.city LIKE world.City;
  INSERT INTO db_:$shard.city SELECT * FROM world.City WHERE CountryCode = $shard;
}

This tight integration between script and SQL, with the power of iteration, conditional statements, variables, variable expansion, throttling etc., makes QueryScript a power tool, with capabilities superseding those of stored routines, and allowing for simplified, dynamic code.

QueryScript code is interpreted. It's just a text, so it can be read from a @user_defined_variable, a table column, text file, what have you. For example:

mysql> set @script := "while (TIME(SYSDATE()) < '17:00:00') SELECT * FROM world.City WHERE id = 1 + FLOOR((RAND()*4079));";
mysql> call run(@script);

For more details, consult the QueryScript site.

If you're a MySQL user/DBA, better read the common_schema QueryScript documentation, to better understand the specific common_schema implementation and enhanced features.

common_schema, including the QueryScript interpreter, can be downloaded from the common_schema project page.

For events registration we use the Signup module and use a theme template function to provide a set of standard fields.  The code looks something like this:

function ourtheme_signup_user_form($node) {
  $form = array();
  // If this function is providing any extra fields at all, the following
  // line is required for form form to work -- DO NOT EDIT OR REMOVE.
  $form['signup_form_data']['#tree'] = TRUE;

  $form['signup_form_data']['FirstName'] = array(
    '#type' => 'textfield',
    '#title' => t('First Name'),
    '#size' => 40, '#maxlength' => 64,
    '#required' => TRUE,
  );
  $form['signup_form_data']['LastName'] = array(
    '#type' => 'textfield',
    '#title' => t('Last Name'),
    '#size' => 40, '#maxlength' => 64,
    '#required' => TRUE,
  );

And so on, building up the elements and then returning the form.  This is great because it allows us to have a standard set of fields for all signup pages, making life a lot simpler when creating content that requires registration.  But the Solutions Day event required an extra field.  I could have done this a number of ways, including putting logic in the template file to check for that particular node and only display the field then, or perhaps some other hack specific to this node.  I, however, don't like specifics and tend to look for a generic solution, as the exception invariably becomes the rule.

For this exercise I wanted to be able to have a way of specifying for a particular node any extra fields that are available for this form.  So I now have in the template.php file the following code:

// If there is a special field required for this, check and display
  if (!empty($node->field_signup_extra) && !empty($node->field_signup_extra[0]['value'])) {
    $extras = explode("\n", $node->field_signup_extra[0]['value']);
    foreach ($extras as $field_def) {
        $field_def = trim($field_def);
        if (empty($field_def)) {
            continue;
        }
        $elems = explode('|', $field_def);
        $field_name = array_unshift($elems);

        $form['signup_form_data'][$field_name] = array();
        foreach ($elems as $field_element) {
          list($key, $val) = explode('=',$field_element);
          if ($key == 'options') {
              $val = explode(',', $val);
          }
          $form['signup_form_data'][$field_name]['#' . $key] = $val;
        }
    }
  }

Now all I need to do is create a field that is non-displayable but contains information to build extra fields.  For example the content that describes the Dietary Requirements field is:

dietary_requirements|title=Dietary Requirements|size=40|type=textfield

The production version does a little more analysis of the input to ensure there are no possible attack vectors, but I've left that out for clarity sake.

Now, if I have an event (or other content type) that needs extra signup fields, I ensure that the content type has the new Signup Extras field and fill it on the new content with a simple field definition that Signup can use.

Henrik has already posted it over on the Drizzle Blog, but I thought I’d give a shout out here too.

We’re holding a Drizzle Day right after the Percona Live MySQL Conference and Expo in April. So, since you’re all like me and don’t book your travel this far in advance, it’ll be easy to stay for the extra day and come and learn awesome things about Drizzle.

I’m also pretty glad that my employer, Percona is sponsoring the event.

• February 16 – Verifying Replication Integrity with Percona Toolkit via ODTUG.
• February 22 – EXPLAIN Demystified via Percona.
• March 14 – Optimizing MySQL Configuration via Percona.

Further Reading:

• Free webinar Wednesday: verifying replication integrity
• Free webinar on MySQL performance this Thursday
• Free webinar on preventing MySQL downtime
• MySQL: Free Software but not Open Source
• Get a free sample chapter of High Performance MySQL Second Edition

[xbm@localhost ~]$ xbm status

XtraBackup Manager v0.8 - Copyright 2011-2012 Marin Software

Currently Running Backups:

+--------+-----------+-------------+---------------------+-------------------+------+
| Job ID |   Host    | Backup Name |     Start Time      |      Status       | PID  |
+--------+-----------+-------------+---------------------+-------------------+------+
| 14     | localhost | test-backup | 2012-02-07 14:19:19 | Performing Backup | 2525 |
+--------+-----------+-------------+---------------------+-------------------+------+

[xbm@localhost ~]$ xbm kill 14

XtraBackup Manager v0.8 - Copyright 2011-2012 Marin Software

Action: Backup Job ID 14 was killed.

2012-02-07 14:19:30 -0800 [ERROR] : [ The backup job was killed by an administrator. Aborting... ]
2012-02-07 14:19:30 -0800 [INFO] : [ Cleaning up files... ]
2012-02-07 14:19:30 -0800 [INFO] : [ Released lock on port 10000. ]
2012-02-07 14:19:31 -0800 [ERROR] : [ Exiting after the backup job was killed... ]

2012-02-07 14:19:19 -0800 [INFO] : [ Staging an INCREMENTAL xtrabackup snapshot of /var/lib/mysql via ssh: mysql@localhost to /tmp/xbm-3592510/deltas... ]
2012-02-07 14:19:19 -0800 [DEBUG] : [ Attempting to run the incremental backup with command:
ssh -o StrictHostKeyChecking=no -p 22 mysql@localhost 'cd /tmp/xbm-3592510 ; innobackupex --ibbackup=xtrabackup --slave-info --incremental-lsn=2325647 /tmp/xbm-3592510/deltas --user=root --safe-slave-backup  --password=XXXXXXX --no-timestamp --incremental --throttle=0 1>&2 ' 
 ]

During this one day event, attendees will receive free, hands-on training on MySQL database solutions from the experts at SkySQL, MariaDB, Continuent, ScaleDB, Severalnines, Sphinx, and Webyog, among others.

As well, SkySQL and MariaDB will bring together industry visionaries, including Michael (Monty) Widenius and David Axmark, the original authors of the MySQL database, to share their insights on current and future trends pertaining to the world’s most popular open source database.

Register Now!

Benefits of Attending:

• Obtain free training and hands-on experience that will help you optimize, scale, and enhance your MySQL database deployments
• Learn how to integrate your applications with popular MySQL solutions from Sphinx, ScaleDB, among others
• Learn how to create a high availability (HA) environment for your MySQL databases using the latest HA solutions
• Gain valuable insights into how to adapt your MySQL solutions to the Cloud
• Hear from industry leaders about current and future trends surrounding the MySQL database and technology ecosystem
• Network with and learn from your peers in the MySQL Community

Who Should Attend:

• MySQL Database Administrators
• Application Developers building solutions on MySQL
• Development/Engineering Management
• IT Architects

Cost of Attending:

• The SkySQL & MariaDB: Solutions Day for the MySQL Database – featuring keynotes from industry visionaries, six intensive tutorials, and lunch – is absolutely free to attendees.

To learn more, take a look at our event schedule, featured speakers, the venue.  As well, follow our discussions on Twitter @skysql #mysqlday, and on Facebook.

Register Now!

mysql> use mysql
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed 

mysql> select * from user into outfile '/tmp/user.sql';
Query OK, 10 rows affected (0.00 sec) 

mysql> select * from user into outfile '/tmp/user.sql';
ERROR 1086 (HY000): File '/tmp/user.sql' already exists

mysql> \! rm -rf /tmp/user.sql
mysql> select * from user into outfile '/tmp/user.sql';
Query OK, 10 rows affected (0.00 sec) 

mysql> \! rm -rf /tmp/user.sql
mysql> select * from user into outfile '/tmp/user.sql';
Query OK, 10 rows affected (0.00 sec)

Enjoy the podcast!

For additional information about MySQL & PHP, check out our last special edition newsletter.

What’s new? You can read the changelog for the details, but here are the highlights:

Brand new pt-diskstats, thanks to Brian Fraser. This tool is completely rewritten, and it’s finally the iostat replacement I always wanted. Not only does it have the functionality I want (interactive, slice and dice, smart defaults) but it has the detailed statistics on I/O, so you can see whether your reads are slow versus your writes, and whether things are waiting on the disk or waiting on the queue scheduler (cfq, hint hint). Finally, it has transparency, so you can read the documentation and understand, really, what it’s doing at the low level and what that means for your server. We really need specific, precise information on exactly how the I/O is behaving so we can make good decisions when there are problems or when doing things like capacity planning.

Brand new pt-stalk, courtesy of Daniel Nichter. This tool is also completely rewritten. Instead of a Bash script that you have to configure with environment variables and run in a screen session, this is now a first-class fault detection daemon. Everyone needs post-mortem forensic data when there is a problem, and pt-stalk aims to be a core part of your infrastructure that fills this gap. It now supports things a “real” Percona Toolkit tool ought to have, such as command-line options and a configuration file. In addition, we merged pt-collect into it, so as of Percona Toolkit 2.0.3, there isn’t a separate pt-collect tool anymore.

There’s a lot more to this release, but those are the major points. Download it and let us know how it works, please! If you find bugs, file them on Launchpad, and if you need support, you know where to get it.

package com.schoolfeed.spymemcached;

import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.util.zip.InflaterInputStream;

import net.spy.memcached.CachedData;
import net.spy.memcached.compat.CloseUtil;
import net.spy.memcached.transcoders.BaseSerializingTranscoder;
import net.spy.memcached.transcoders.Transcoder;
import org.lorecraft.phparser.*;


public class PHPSerializedTranscoder extends BaseSerializingTranscoder implements Transcoder <Object> {
 
 static final int COMPRESSED=2;
 /**
  * Get a serializing transcoder with the default max data size.
  */
 public PHPSerializedTranscoder() {
  this(CachedData.MAX_SIZE);
 }

 /**
  * Get a serializing transcoder that specifies the max data size.
  */
 public PHPSerializedTranscoder(int max) {
  super(max);
 }
 
 /**
  * decode the byte data from Memcache decompress it if necessary and return the Object
  * @param CacheData - the byte data is turned into a object
  * @returns Object 
  */
 public Object decode(CachedData d){
  byte[] data=d.getData();
  
  Object rv=null;
  String ds="N;";
  
  if((d.getFlags() & COMPRESSED) != 0) {
   getLogger().debug("Looks like d is compressed");
   data=decompress(d.getData());
  }
  
  ds=decodeString(data);
   
  getLogger().debug("DECODED: [" + ds + "] about to SerializedPhpParser");
  
  SerializedPhpParser sp = new SerializedPhpParser(ds);
  
  try {
   rv = sp.parse();
   getLogger().debug("Parse was cool!!");
  } catch(Exception e){
   getLogger().debug("Not a PHP Object? : " +  ds);
   rv = ds;
  }
  
  return rv;
 }
 
 /**
  * PHP Memcache stores compress data in ZLIB format override the base class decompress method to handle ZLIB
  * 
  * @param byte array - raw data from Memcache
  * @returns byte array
  */
 protected byte[] decompress(byte[] in) {
  ByteArrayOutputStream bos=null;
  final int BUFFER = 2048;
  if(in != null) {
   ByteArrayInputStream bis=new ByteArrayInputStream(in);
   bos=new ByteArrayOutputStream();
   InflaterInputStream iis = null;
   try {
    iis = new InflaterInputStream(bis);

    byte[] buf=new byte[BUFFER];
    int r=-1;
    while((r=iis.read(buf, 0, BUFFER)) > 0) {
     bos.write(buf, 0, r);
    }
   } catch (IOException e) {
    getLogger().warn("Failed to decompress data", e);
    bos = null;
   } finally {
    CloseUtil.close(iis);
    CloseUtil.close(bis);
    CloseUtil.close(bos);
   }
  }
  
  return bos == null ? null : bos.toByteArray();
 }
 
 /**
  * encode -- not implemented yet
  *
  */
 public CachedData encode(Object o){
  int flags = 0;
  byte[] b=null;
  return new CachedData(flags, b, getMaxSize());
 }
 
 /**
  * no need to async Decode let's do it realtime
  */
 public boolean asyncDecode(CachedData d) {
  return false;
 }

}

For those of you who don’t know what this is, it’s another of the O’Reilly traditions we’re trying to continue. (We are trying very hard to make this event as close to a clone of O’Reilly’s as we can.) It’s a free table in the expo hall where people who participate in a non-commercial open source project can exhibit. I organized a Maatkit booth a few times in the past, and was always really grateful to O’Reilly for making the space available. Space in the expo hall is at a premium, but we think that these dot-org booths are even more valuable to the open-source projects and the conference attendees.

So, please tell your friends who care about open source, and ask them to tell their friends too. Let’s get some great open-source projects into the expo hall, alongside the commercial vendors!

Further Reading:

• Percona Live MySQL Conference comes to Washington DC
• One-day Percona Live conference in San Francisco
• Maatkit at the dot-org pavilion at MySQL Conference and Expo 2009
• Schedule for MySQL-and-beyond conference is live
• Percona Live from a community insider’s perspective

Diversification

In the past few years there was a lot of fear and doubt about MySQL due to Oracle taking over the ownership. But if you ask me, I was more worried for MySQL because of MySQL itself. I've often said that if MySQL had been a healthy open source project - like the other 3 components in the LAMP stack - then most of the NoSQL technologies we've seen come about would never have been started as their own projects, because it would have been more natural to build those needs on top of MySQL. You could have had a key-value store (HandlerSocket, Memcache API...), sharding (Spider) and clustering (Galera). You could have had a graph storage engine (OQGraph engine isn't quite it, I understand it is internally an InnoDB table still?). There could even have been MapReduce functionality, although I do think the Hadoop ecosystem targets problems that actually are better solved without MySQL.

read more

Fellow Community Manager Keith Larson and I working hard to keep up with all the upcoming shows. If you think we might have not notices your upcoming event, please drop us a line. We will have thumb drive, baggage tags, stickers, and maybe (fingers crossed) t-shirts or plush dolphins soon.

The summit was well attended, with 350 registrants and keynotes from prominent MySQL users such as Mark Callaghan. The coverage was quite broad, with presentations running the gamut from grad students (complete with bodyguards and intimidating academic advisors) to established companies such as StreamBase. The sponsor list was an A-list this year as well, with EMC and Microsoft being the two biggest backers.

There were far too many and diverse topics to write about all of them. That said, here were a few of the notable ones.

Keynote #1: Johannes Gehrke (Cornell): Declarative Data-Driven Coordination

Johannes Gehrke of Cornell kicked off with the first keynote on Declarative-Driven Coordination. His methodology shed light on an alternative to out-of-band communication. The presentation focused on how to successfully handle entangled queries.

More Sleep for Tom and Meg if They Can Just Coordinate

In brief, what he showed is a way for someone to see if their friend is on a flight and have the database go about satisfying mutual constraints. With a proof that is outlined in his Sigmod paper, his main theorem is that any schedule that is entangled-isolated is also oracle-serializable. It’s a clever approach, as long as one’s set of friends being entangled remains small.

Keynote #2: Mark Callaghan (Facebook): Performance is Overrated

The room got a little quiet when Mark took the stage. Some people were expecting a possible rehash of this summer’s brouhaha between Mike Stonebraker and Facebook on the fate of MySQL. But, instead Mark jumped into some very practical discussions about managing MySQL at scale.

First, he noted that manageability needs more attention since…

• The cost of extra hardware can be predicted
• The cost of downtime cannot
• Downtime comes in many forms (server down and server too busy)

For Mark, manageability has a number of meanings. This includes the rate of interrupts/server for the operations team. Mark finds that while the server count grows quickly, his operations team grows slowly. Hence, it is imperative that the quality-of-service improve over time (i.e., Does work get done? Does work get done on time?).

Mark and his team use MySQL for a number of reasons. First, it was there when Mark arrived. Second, Mark and his team made it scale 10x. Finally, Mark likes MySQL for OLTP.

As Facebook has grown though, so have the number of servers. This is due to “Big Data” x high QPS. Hence, they have had to add servers to add IOPs. To address this, Mark noted that flash memory (SSD) is very interesting as are (we blush) write-optimized databases.

The last part of his presentation focused on advice for scaling: More Data, More QPS. His tips were quite straightforward:

• Fix stalls to make use of capacity
• Don’t make MySQL faster, make it less slow
• Improve efficiency to use less
• Repeat

 Additional details can be found in Sheeri’s excellent live blog of the presentation.

New Tools and Systems Session: Willis Lang (University of Wisconsin): Energy-Conscious Data Management Systems

Just as Mark stressed that performance isn’t everything when he spoke about management, Willis Lang pointed out another key concern.  His slides noted that “three decades of database research has optimized for the highest possible performance possible regardless of energy consumption.” (We agree and have written about this topic as well).

Willis and his team have been looking at various techniques for addressing this such as using variable speed disks. He has been systematically studying the power/performance trade-offs of hardware components. The preliminary memory-based results showed that interesting trade-off opportunities exist if one rethinks database design principles. His presentation focused on the improvements that can be seen with memory parking. Additional details on his research can be found here.

As mentioned previously, there were many good talks — much more could be written about the event. Other interesting speakers included David Karger who introduced Dido, which seeks to make database manipulation as easy as document editing, and Alvin Cheung whose Pyxis project eases application development with automatic code partitioning based on application and server characteristics.

Kudos to Samuel Madden (MIT) and Ugur Cetintemel (Brown University) for organizing the event. Additional details can also be found via the Twitter hashtag #nedb12 and the event homepage.

I thought I’d try to build it, and see what happens:

1. Download XtraDB 5.5 Source
2. cd C:\xtradb-5.5
3. mkdir bld
4. cd bld
5. cmake ..
6. VS08: File -> Open -> Solution -> C:\xtradb-5.5\bld\MySQL.sln

Build Ended With:

========== Build: 70 succeeded, 17 failed, 2 up-to-date, 10 skipped ==========

The first failure had to do with innobase:

18>Generating Code...
18>Build log was saved at "file://c:\..\innobase.dir\Debug\BuildLog.htm"
18>innobase - 9 error(s), 3 warning(s)

I checked the innobase build log and found this:

sql_prepare.cc
..\sql_prepare.cc(2199) : error C3861: 'gettimeofday': identifier not found
..\sql_prepare.cc(2232) : error C3861: 'gettimeofday': identifier not found
..\sql_prepare.cc(2639) : error C3861: 'gettimeofday': identifier not found
..\sql_prepare.cc(2672) : error C3861: 'gettimeofday': identifier not found
..\sql_prepare.cc(2808) : error C3861: 'gettimeofday': identifier not found
..\sql_prepare.cc(2848) : error C3861: 'gettimeofday': identifier not found
..\sql_prepare.cc(2940) : error C3861: 'gettimeofday': identifier not found
..\sql_prepare.cc(2973) : error C3861: 'gettimeofday': identifier not found
...
sql_parse.cc
..\sql_parse.cc(5750) : error C3861: 'gettimeofday': identifier not found
..\sql_parse.cc(5827) : error C3861: 'gettimeofday': identifier not found

The ‘gettimeofday’ function is a Unix-only (i.e., there is no direct analog of the gettimeofday() in Windows) .. more details here.

So, this part of the code has simply not been ported to Windows yet.

I got to looking, and it is a reported bug already:

https://bugs.launchpad.net/percona-server/+bug/737895
https://bugs.launchpad.net/percona-patches/+bug/421925
https://bugs.launchpad.net/percona-patches/+bug/390156

But, do not worry. If you want to take advantage of the InnoDB enhancements found in XtraDB on Windows, you can simply use MariaDB.

There are pre-built MariaDB binaries for Windows, and MariaDB contains XtraDB+ (which is XtraDB plus some additional enhancements on top of it).

Hope this helps.

• What data I can store in it: Only numbers? Printable alaphanumeric characters? Unicode? Binary data? An object?
• What I can do with a value of that particular type and how does it behave? Can I concatenate two values? Add them (that is NOT the same as concatenate! The same operator (+) may be used, but it's not the same thing!)? Save it to a file?
• How is it stored and represented in technical terms. And this can be argued, a C int doesn't define how it is stored on disk, but a MySQL INT does! And a BLOB is a type that is largely there just because it defines how to handle things on disk!

• RAW Data - This should be a basic type. As long as can be, or as short. Any data can be stored in it, and nearly as much as you would like.
• Integers - Although an integer can be considered to be a special case of a more generic number, the integer domain is so generic, and the integer is such a common and basic type that is deserves a place here.
• Decimal – Fixed point decimal numbers.
• Float - Now we are getting close to the application domain here, but these guys deserve a place here anyway, as they are so common and hooked up to the programming languages and hardware.
• DATETIME - Only one of them is really needed. One of DATETIME, DATE, YEAR, MONTH, TIMESTAMP, TIME etc etc. that is. Just one, very generic base type, where the specific domain is, you've heard it before, application specific!
• Text - Reluctantly I add this to the list. Although this is just a RAW with some limitations (usually only UTF8 or ISO8859-15 or something characters), there are some operations and attributes that are so tied up with text strings that a TEXT type is reasonable to include.

• First we create a domain:
  CREATE DOMAIN yesno AS CHAR(1)
  DEFAULT 'N'
  NOT NULL CHECK (VALUE = 'Y' OR VALUE = 'N');
• Then we create a table that uses that domain:
  CREATE TABLE user_active(username CHAR(10), is_active YESNO);
• And then we check if it works using a few SQL statements:
  INSERT INTO user_active VALUES('foo', NULL);
  ERROR: domain yesno does not allow null values
  INSERT INTO user_active VALUES('foo', 'Y');
  INSERT 0 1
  INSERT INTO user_active VALUES('foo', 'N');
  INSERT 0 1
  INSERT INTO user_active VALUES('foo', 'A');
  ERROR: value for domain yesno violates check constraint "yesno_check"

Percona Toolkit is Maatkit & Aspersa combined. Opensource and the tools are very useful for a DBA.

You need Perl, DBI, DBD::mysql, Term::ReadKey. Most tools are written in Perl, and whatever is in Bash is being re-written in Perl. There is also a tarball or RPM or DEB packages.

Know your environment. The hardware & OS are crucial for you to know. How much memory/CPU do you use? Do you use swap? Is this a physical/virtual machine? Do you have free space? What kind of RAID controller? Volumes? Disk? What about the network interfaces? What IO schedulers are used? Which filesystem is the data stored on? To answer all that, just use pt-summary.

Know your MySQL environment. Version? Build? How many databases? Where is the data directory? What about replication? What are key InnoDB settings? Storage engine in use? Index type? Foreign keys? Full text indexes? To answer all this and more use pt-mysql-summary.

pt-slave-find shows you the topology and replication hierarchy of your MySQL replication instances. An inventory of replicas!

Where is my disk I/O going? Use pt-diskstats which is an improved iostat. There is pt-ioprofile but it can be dangerous in production.

Now its time to get more intimate with your database. Let’s try to find the answer to these questions: how are the indexes used? Are there duplicate keys? Which queries are eating most of the resources? You can use pt-duplicate-key-checker to check for duplicate/redundant indexes or foreign keys. pt-index-usage can tell you which indexes are unused. If you think you have bad SQL, check out pt-query-advisor.

You can use pt-query-digest to analyze the slow query log and show a profile of the workload. You mostly use this with slow query logs & tcpdump’s. Be careful when you have dropped packets — results may tend to be fake then!

After all this, its time to maintain your environment.

pt-deadlock-logger checks InnoDB status to log MySQL deadlock information. It needs to run continually to capture things.

pt-fk-error-logger extracts and logs MySQL foreign key errors.

pt-online-schema-change to alter tables. It makes a “shadow copy” and swaps them. Extremely useful for large, long-running ALTER. Facebook uses the same technique.

Validate your upgrades as upgrades are the leading cause of downtime. Are queries using different indexes? Is query execution plan different? New errors? See pt-upgrade for this. Best to run this on a third machine (i.e. the old machine and a new machine to see how it goes).

Verify replication integrity – pt-table-checksum. Perform an online replication consistency check or checksum MySQL tables efficiently on one or many servers. Use it routinely (mandatory for 95% of MySQL users). Put it in a weekly crontab. Repair differences with pt-table-sync.

Repair out-of-sync replicas – pt-table-sync

Measure delay acfurately – pt-heartbeat

Deliberately delay replication – pt-slave-delay

Watch & restart MySQL replication after errors – pt-slave-restart

When there are problems, get the symptoms when it hurts. Look at pt-stalk (wait for a condition to occur them begin collecting data – eg. everytime the threads go over 2,000 you have a problem, so it collects stuff – it calls pt-collect), pt-collect (collect information from a server for some period of time), and pt-sift.

pt-mext looks at many samples of MySQL SHOW GLOBAL STATUS side-by-side. Default STATUS shows counter since the MySQL instances started. It is very helpful to see a delta of recent activity.

The future: pt-query-digest will do query reviews; pt-stalk will do “magical fault detection algorithm”. Its all opensource and its all on Launchpad at lp:percona-toolkit.

Related posts:

1. Practical MySQL Indexing guidelines by Stéphane Combaudon
2. Replication features of 2011 by Sergey Petrunia
3. MySQL synchronous replication in practice with Galera by Oli Sennhauser

MySQL 5.5 GA at the end of 2010. MariaDB 5.3 RC towards the end of 2011 (beta in June 2011).

MySQL 5.5 is merged to Percona Server 5.5 which included semi-sync replication, slave fsync options, atuomatic relay log recovery, RBR slave type conversions (question if this is useful or not), individual log flushing (very useful, but not many using), replication heartbeat, SHOW RELAYLOG EVENTS. About 2/3rds of the audience use MySQL 5.5 in production, with only 2 people using semi-sync replication.

MariaDB 5.3 brings replication features brings group commit in the binary log, which is merged into Percona Server 5.5. Checksums for binlog events which is merged from MySQL 5.6. Sergey goes in-depth about the group commit for the binary log. To find out a little more about MariaDB replication changes, see Replication in the Knowledgebase.

There are several implementations of group commit. Facebook started it, followed by MariaDB & Oracle. Percona 5.5 is GA so the feature is there, its not in MySQL 5.6 (yet?), and MariaDB 5.3 is where its at. Seems like the MariaDB implementation is the best so far – refer to the Facebook benchmark performed by Mark Callaghan.

Annotated RBR poses a compatibility problem. MariaDB 5.3 has annotate_rows, while MySQL 5.6 has rows_query event. They are different events. So you cannot have a MariaDB 5.3 master and a MySQL 5.6 slave at this moment. So MySQL 5.6 will have a flag to mark “ignorable” binlog events which will be merged into MariaDB and this will make binary logs compatible again.

There is now also optimized RBR for tables with no primary key.

MySQL 5.6 also has crash-safe slave (replication information stored in tables). Crash-safe master (binary log recovery if the server starts & sees the binary log is corrupted). Parallel event execution is something that is new in MySQL 5.6 which is the most important feature for Sergey.

Pre-heating: There is mk-slave-prefetch (famous quote: “Please don’t use mk-slave-prefetch on #MySQL unless you are Facebook.”). There is replication booster by Yoshinori Matsunobu. There is a Python version of mk-slave-prefetch that Facebook uses.

Related posts:

1. MariaDB 5.3 query optimizer by Sergey Petrunia
2. Where is MariaDB today?
3. Building simple & complex replication clusters with Tungsten Replicator by Giuseppe Maxia

Only works on Unix-like servers. Works with MySQL, Percona & MariaDB servers. MySQL server has the data directory, the port and the socket – you can’t share these.

To use it: make_sandbox foo.tar.gz. Then just do ./use.

$SANDBOX_HOME is ~/sandboxes. You can also create ~/opt/mysql/ and if you have MySQL 5.0.91 binary in that directory, you can just do “sb 5.1.91″.

Sandbox has features to start replication systems as well. You can have varying master/slave setups with varying versions as well (good idea to test from MySQL -> MariaDB master->slave for migration).

You can now also play with tungsten-sandbox, which is a great way to start playing with Tungsten Replicator (see documentation and tungsten-toolbox). There is apparently also a MySQL Cluster sandbox tool that someone is working on.

Related posts:

1. Building simple & complex replication clusters with Tungsten Replicator by Giuseppe Maxia
2. MySQL HA reloaded by Ivan Zoratti
3. MySQL synchronous replication in practice with Galera by Oli Sennhauser

Work that has been done to make Zarafa better. Why do you optimise your buffer pool? To decrease your I/O load. How can you do it? Buy more RAM, page compression, less (smaller) data, rearrange data.

MariaDB or Percona Server allows you to inspect your buffer pool (unsure if this is now available in MySQL 5.6). Giuseppe in the audience says this is available in MySQL 5.6, but Steve used this on MariaDB 5.2.

Strategies to fix it: Make records smaller. Remove indexes if you can use others almost as efficiently. Make records that are accessed around the same time have a higher chance of being on the same page. Use page compression. Buy more RAM. Try Batched Key Access (BKA) in MariaDB 5.3+.

Best to view the presentation since there are specific examples that speak about how Zarafa solves their problems like a user trying to sort their email, etc.

Related posts:

1. Batched Key Access Join Previews available
2. MariaDB 5.3 query optimizer by Sergey Petrunia
3. MariaDB/MySQL users in Paris & Brussels

Index: separate data structure to speed up SELECTs. Think of index in a book. In MySQL, key=index. Consider that indexes are trees.

InnoDB’s clustered index – data is stored with the Primary Key (PK) so PK lookups are fast. Secondary keys hold the PK values. Designing InnoDB PK’s with care is critical for performance.

An index can filter and/or sort values. An index can contain all the fields needed for the query you don’t need to go to the table (a covering index).

MySQL only uses 1 index per table per query (not 100% true – OR clauses), so think of a composite index when you can. Can’t index TEXT fields (use a prefix). Same for BLOBs and long VARCHARs.

Indexes: speed up queries, increases the size of your dataset, slows down writes. How big is the write slowdown? Simple test by Stephane, for in-memory workloads he says adding 2 keys makes performance 2x worse; for on-disk workloads he says its 40x worse. Never neglect the slowdown of your writes when you have an index. There is a graph in the slidedeck.

What is a bad index? Unused indexes. Redundant indexes. Duplicate indexes.

Indexing is not an exact science, but guessing is probably not the best way to design indexes. Always check your assumptions – EXPLAIN does not tell you everything, time your queries with different index combinations, SHOW PROFILES is often valuable. Slow query log is a good place to start.

Many slides with examples, so I hope Stephane posts the deck soon. If possible, try to sort & filter (an index is not always the best for sorting).

InnoDB’s clustered index is always covering. SELECT by PK is the fastest access with InnoDB.

An index can give you 3 benefits: filtering, sorting, covering.

See Userstats v2 - you need Percona Server or MariaDB 5.2+. See also pt-duplicate-key-checker to find redundant indexes easily. See also pt-index-usage to help answer questions not covered by userstats.

Related posts:

1. Sphinx user stories by Stéphane Varoqui
2. Managing MySQL with Percona Toolkit by Frédéric Descamps
3. MySQL Full Text Search by Alex Rubin

Synchronous multi-master replication with the Galera plugin. Your application connects to the load balancer and it redirects read/write traffic to the various MySQL Galera nodes. Tested a setup with 17 SQL nodes and you can have even more. Scaling reads and also a little bit for scaling writes is what Galera is good for.

If one node fails, the other two nodes still communicates with each other and the load balancer is aware of the failed node.

Why Galera? There is master-slave replication but its not multi-master, and its asynchronous and you can get inconsistencies. There is master-master replication but its asynchronous and can have inconsistencies and conflicts if you write on both nodes. MHA/MMM/Tungsten are not providing new technology but are based on the MySQL replication technology. MySQL Cluster is another solution but its not InnoDB storage & your need new know-how for Cluster. Also Cluster has problems with fast JOINs. Active/Passive failover clustering, but too often you have resources idling. Schooner being closed & expensive is hard to know much about what they’re doing.

Galera is synchronous & based on InnoDB (others should in theory be possible). Active-active real multi-master topology. True parallel replication on row level. Cluster speaks with each other. There is no slave lag. Won’t lose transactions. Read/write scalability, write throughput can be improved but can’t scale in the way like MySQL Cluster.

Disadvantages? Its not native MySQL binaries/sources but a patch. Codership provides binaries. Higher probability of deadlocks. When you do a full sync (like when a node comes back after downtime), one node is blocked. This is why the minimum you need a 3-node cluster. Also if you do a full sync with a database larger than 50GB, the recommended method is to use mysqldump (which can be very slow). You can use rsync. Percona is working on xtrabackup to do a full sync between nodes.

Setup: 3 nodes are recommended. Or just 2 nodes and one for garbd (Galera Arbitrator Daemon). 2 nodes works but pay attention to a split brain scenario. Go to the Codership website, download their binaries and wsrep (the Galera plug-in). Create your own user on all nodes (don’t use the default root user). You then need to configure my.cnf (there have been discussions for a galera.conf, but Oli just uses my.cnf). Galera works only with InnoDB, so in my.cnf make the default storage engine InnoDB (don’t for example, by accident have MyISAM tables).

The demo has a strange Galera start script, but its not been easy to work. Just start MySQL usually like you would do.

SST is Snapshot State Transfer (SST). Its the initial full sync between the 1st and the other node. SST blocks the donor node (hence why you need 3 nodes). With Galera v2.0, there is also incremental state transfer. It should be GA in February 2012. You can get deltas as opposed to the full sync. You can configure which will be the donor node.

Currently there are 27 variables about Galera in v1.1. You can do it just by doing SHOW GLOBAL VARIABLES LIKE ‘wsrep%’;. The plugin itself, wsrep_provider_options has plenty of options & plenty of room for tuning. SHOW GLOBAL STATUS LIKE ‘wsrep%’; currently has 38 status information fields in Galera v1.1.

For load balancing, you can do it in your application (on your own). You can also use Connector/J which provides load balancing. There is also a PHP MySQLnd that works.

Related posts:

1. MySQL HA reloaded by Ivan Zoratti
2. Ticketmaster thrives on MySQL Replication
3. New MySQL 5.6 Features by Oli Sennhauser

This was a talk about the future in general. What people would like. Etc. Not about something that exists yet, hence the sparse notes.

Trace statements are good for the “why”. You move on to the debugger, but what is a useful SQL debugger? Profilers addresses the “when”.

SQL developer tools: manually run queries (traditional route for development, effective way to test SQL statements, some indication of performance), server logs (alerts developers of serious issues like the slow query log – identify poorly written queries, requires server access, not effective in a cloud environment (?)), external monitoring (network sniffing of connection packets, Ethereal dissectors – allow individual connections to be traced, no server access, privileged network access, complex to interpret, possibility of packet loss), intrusive tools (insert “proxy” between client & server to intercept all traffic, imposes delay & requires setup modification).

Client-side tools – client modification or hook via connectors (query logging, profiling). The Java connector has profiling. Should there be a connector slow query log? So there’s no requirement for server access and its per client rather than per server.

Plans: roll out connector query logs for Java, C, and scripting languages built on Connector/C. SQL Parser for report generation/fuzzy matching. Create post-processing tools. Query log comparisons – detect fluctuation in execution time, plan flip alerting, replay log.

Related posts:

1. Managing MySQL with Percona Toolkit by Frédéric Descamps
2. Open source tools to run a small-medium sized business
3. Practical MySQL Indexing guidelines by Stéphane Combaudon

Cluster: shared nothing architecture (no single point of failure), synchronous replication between nodes, ACID transactions, row level locking. In-memory storage (some data can be stored on disk, but indexes must be in-memory). Checkpointing to disk for durability. It supports two types of indexes – ordered T-trees, unique hash indexes. Online operations like adding node groups, software upgrades, table alterations. Quick standard architecture diagram displayed about MySQL Cluster.

Network partitioning protocol is designed to avoid a split brain scenario. Is there at least one node from each node group? If not then this part cannot continue – graceful shutdown. Are all nodes present from any node group? If so, then this is the only viable cluster – continue to operate. Ask the arbitrator – the arbitrator which parts will continue if no arbitrator is available the cluster shuts down.

Durability – in order for a node to recover fast some data is stored locally. The REDO log is synchronized by global checkpoints (GCP). The DataMemory is synchronized by local checkpoints (LCP).

I agree with Ralf — almost impossible to talk about NDB in 25 minutes. Its very deep, you’d need at least three hours to grasp it well.

Related posts:

1. MySQL synchronous replication in practice with Galera by Oli Sennhauser
2. Practical MySQL Indexing guidelines by Stéphane Combaudon
3. Managing MySQL with Percona Toolkit by Frédéric Descamps

MySQL replication is single threaded. Multi-master replication is complex with MySQL. Circular replication works but is very fragile. Once you’ve achieved the feat, how do you avoid conflicts? The lack of global transaction ID today also means you may have slaves that may not be synchronized fully. Finally, some people like to replicate to PostgreSQL, Oracle and MongoDB.

This is where Tungsten Replicator comes into place. Opensource. 100% GPLv2. You can do easy failover (no need to synchronize the slaves manually when a master dies), have multiple masters, multiple sources to a single slave, conflict prevention, parallel replication, and replicate to Oracle/PostgreSQL database (heterogeneous replication – Oracle->MySQL is not opensource, but MySQL->Oracle is).

Parallel replication: ability to replicate with multiple thread at once. Sharded by database (all big sites already use this naturally). Good choice for slave lag problems. Bad choice for single database projects. In their tests to measure slave catch up time between standard MySQL & Tungsten slave with 1hr of sysbench and some 130GB of data, MySQL replication takes 4.5h to catchup, and Tungsten takes less than 1 hour to catchup. This comparison was not made with MySQL 5.6 (the same parallel replication feature at this moment in 5.6 is quite buggy). No need to install Tungsten on the master (Tungsten needs to be on slave). Replication can revert to native salve with 2 commands, but the failover is an issue — so install Tungsten on both!

Conflict prevention. When you have multiple masters you are tempted to do many things that you shouldn’t. Tungsten provides the multi-master topology. Tungsten can help you avoid conflicts. Decide you want to shard your information by database – Tungsten can enforce such rules. Define the rules, applied to either the master or the slave and you can then state to make replication fail or drop silently or drop with a warning.

You can create clusters very quickly with Tungsten. You can install from a centralized point to many servers. You need Java, Ruby, SSH access and a MySQL user with all privileges (used for replication).

Related posts:

1. MySQL Creatively in a Sandbox by Giuseppe Maxia
2. MySQL HA reloaded by Ivan Zoratti
3. Replication features of 2011 by Sergey Petrunia

Presentation started out with a very nice presentation of candies to all the audience members.

What is Sphinx? Another (C++) daemon on your boxes. Can be queried via API (PHP, Python, etc.) or MySQL-compatible protocol and SQL queries (SphinxQL). Some query examples are in the slides, here’s one about SphinxSE in the KB.

MyISAM FTS is good but becomes slow with half a million documents. InnoDB has FTS now but he’s not tried it (and neither has anyone in the audience to see it compare with MyISAM FTS).

Geographical distance is the distance measuring the surface of the earth (two pairs of float values – latitude, longitude). In Sphinx, there is support for GEODIST(Lat,Long,Lat2,Long2) in Sphinx.

Segments are good for price ranges on a site, date ranges, etc. Use INTERVAL(field, x0, x1, …, xN).

Keep huge text collections out of the database. sql_field = path_to_file_text. Tell Sphinx to index text not from MySQL but out in the filesystem. Keep the metadata inside the database but keep the actual data outside of the database. max_file_field_buffer needs to be set properly.

You can do proximity search with Sphinx — find the words “hello world” within a ten word block, for example.

Resources: the documentation, a book by O’Reilly: Introduction to Search with Sphinx: From installation to relevance tuning (sold out at the FOSDEM O’Reilly booth!), and their community page including wiki, forum, etc.

Related posts:

1. MySQL Full Text Search by Alex Rubin
2. Sphinx user stories by Stéphane Varoqui
3. Morning sessions at MySQL MiniConf

What exactly is not working in MySQL? MySQL is poor at decision support/analytics. With large datasets you need special disk access strategies. Complex queries like insufficient subquery support and big joins are common int he MySQL world.

DBT-3 is used, scale=30, with a 75GB database and run a query “average price of item between a range of dates”. Query time took some 45 minutes to execute. Why? Run iostat -x to see what is going on. See that the CPU is mostly idle, so its an IO-bound load. Next you run SHOW ENGINE INNODB STATUS and you’ll see how many reads per second is happening. Possible solution is to get more RAM or get an SSD (good to speedup OLTP workloads, but analytics over data is probably not viable since SSDs are small and not cheap).

The MySQL/MariaDB solution to the above problem is improved disk access strategies: multi-range read (MRR) and batched key access (BKA). In MariaDB, MRR/BKA need to be enabled (they are not turned on by default). The query time only took 3 minutes 48 seconds, which is some 11.8x faster than the previous 45 minutes.If you look at EXPLAIN output, its almost as same as before, expect the Extra filed. iostat -x will now show some CPU load, svctm down as well (so its not random disk seeks anymore — some 8ms seek time on a regular 7,200rpm disk), SHOW ENGINE INNODB STATUS will show some 10,000 reads per second rather than the previous 200.

If you are on Fedora, check out the Systemtap feature to look at I/O patterns. stap deviceseeks.stp -c “sleep 60″.

Subqueries handling in MariaDB 5.3: check out the Subquery Optimizations Map. Only about 10% of the audience use optimizer hints in MySQL.

Related posts:

1. Replication features of 2011 by Sergey Petrunia
2. MariaDB 5.1.42 released!
3. MariaDB 5.1.44 released

Questions to ask: which level of high availability do I need? Do I require no loss of data? Do I need failover or is switchover enough? Can I provide a reasonable service when a component is down? Remember, five nine’s high availability also means a lot of infrastructure costs.

Other things to clarify: availability vs scalability. HA costs. HA for your entire architecture, not just for your database servers. Review your SLAs.

The best high availability solution today is combined solutions.

MySQL replication – asynchronous & semi-synchronous (lots of people use MySQL 5.5, about 4 people in room were on Percona Server – question asked due to semi-sync replication only being available in 5.5 & greater), there are pros & cons of row based replication vs statement based replication.

MySQL Replication via Multi-Master replication Manager (MMM). Features such as monitoring, automatic failover, data backup & resync. Unfortunately, it has some problems with the stability & automatic failover. The project is not improved anymore, so there are other solutions that you should consider today.

MySQL Replication with MHA is a preferred solution. Something to consider: –read-only=1 and log-bin on slaves. Master IP failover. FIltering rules. Multi-tier replication.

Tungsten Replicator – open source, heterogenous replication. Truly multi-master and fan-in with Global ID. Per-schema multi-thread. You can also use it to replicate to Postgresql, Oracle and other databases. There is also Tungsten Enterprise.

Synchronous replication with DRBD is typical for active/standby environments. People don’t really like this because they feel that there is a server doing nothing. You can always do it active/passive. It works with InnoDB only.

Synchronous replication with Galera works for InnoDB. Its multi-master with no SPOF. Its new/young technology so you may find some issues with it. Application failover must be managed, but the conflict resolution is quite tricky (when you commit a transaction you might be fine, but you may have transactions that are removed due to conflicts).

There is a commercial SchoonerSQL that provides synchronous master-slave replication for InnoDB. Its defined explicitly as a master-slave solution.

Active/Passive clusters using Shared Storage. Points to consider are the fact that redundancy & replication must be guaranteed by the shared storage. InnoDB only. What about filesystems?

Virtualized environments – data storage, high availability & load balancing are provided and managed by the virtualized software. The faults are handled by the software, not the database.

There is also geographical replication for disaster recovery, having a master-master asynchronous replication is used to update the backup data centre. There is also storage snapshots for disaster recovery (not-specific to MySQL, its storage systems based, use only InnoDB).

There is also MySQL Cluster but there is another presentation about this later at FOSDEM. Very nice closing slide, “The absolutely necessary comparison chart” which some may disagree, but Ivan thinks is the best way forward.

Related posts:

1. MySQL synchronous replication in practice with Galera by Oli Sennhauser
2. MySQL at Google
3. Building simple & complex replication clusters with Tungsten Replicator by Giuseppe Maxia

Pros: Filtering takes place on attributes in separate tables. Rely on the optimizer choice. HASH JOIN can help (MariaDB 5.3). Table elimination can help (MariaDB 5.2). ICP Index Condition Pushdown can help (MariaDB 5.3/MySQL 5.6). Max 80M documents at Pixmania, all queries come in less than 1s using 128GB of RAM (MariaDB 5.2). At PAP.fr, there is 16GB RAM with MariaDB 5.2.

Cons: CPU intensive (replication with many slaves). Need covering indexes to cover various !filter !order. Join & sorting cost on lazy filtering.

The more indexes you have in the system, the more you need to increase the main memory of the server. Keep the Btree’s in memory.

What about denormalized schemas? Not really CPU intensive, just IO. Can go to disk, full partition scan with filtering taking place on record order using covering index. Can shard but not that easy. Use the spider storage engine or shard-query. Can use memory engine for isolation. There are cons like duplicate data, duplicate indexes, missing material views, merge index cost, impact on write performance, and can consume a lot of memory with many indexes.

MySQL can push hardware, so read less/do less/read serialized/map reduce to get better latency. Chose data type wisely, replace string with numeric, vertical & horizontal sharing, snowflake compression (combination of attributes, build a table of the combination and replace it with an ID). If you are lazy, just use Sphinx!

Sphinx is just another daemon that can serve queries. Its easy to setup, easy to scale, storage engine makes it accessible to current MySQL users, API in core MariaDB (SphinxSE), SphinxQL, SphinxSE is transparent to the application layer of MySQL protocol.

Demo done using the Employees DB.

Pierre & Vacances – Centerparcs. Free text search, they use MariaDB using Levenshtein UDF implementation. Went live 01/2011. First implementation of Sphinx (12 indexes). Its grown, they use PHP API. The new goal is to never send an empty result. 1 index per website/market, with a total of 15 million docs. Index built on standalone server. Using internal gearman job schedule to generate index before cache generated. Current monitoring is via Nagios & perl, but the next step is to use Monyog & MariaDB INFORMATION_SCHEMA plugin.

Delcampe is an auction website with 45M ‘active’ items. Its dedicated for collectors. 3 string fields, and 15 attributes. 40-120K new items daily. Started with mysql fulltext in 2007, moved to Sphinx in 2008. There was a need to have more filters. Now they have 5 sphinx servers + 1 MySQL server. HAproxy to load balance.

Related posts:

1. Practical MySQL Indexing guidelines by Stéphane Combaudon
2. How to offload MySQL server with Sphinx by Vladimir Fedorkov
3. MySQL Full Text Search by Alex Rubin

New Release Model: starts with at least in beta quality, milestone releases are RC quality (every 3-6 months), between milestones new features are allowed, GA releases every 12-18 months, no more than 2 releases in active support. There is also MySQL Labs, in where features can make it into a release model, but not necessarily.

Oli’s guess: MySQL 5.6 GA in April 2012 (Collaborate) or June 2012. As a consequence, MySQL 5.0 and MySQL 5.1 will be EOL probably by April this year.

New improvements in partitioning: explicit partition selection, exchanging partitions (good for ETL jobs).

New improvements in InnoDB: InnoDB INFORMATION_SCHEMA has got some new entries. Buffer (INNODB_BUFFER), Fulltext (INNODB_FT), Metrics (INNODB_METRICS), Data Dictionary (INNODB_SYS). The InnoDB Monitor is now obsolete. innodb_purge_threads can be set > 1. Kernel mutex split which should lead to improved concurrency. Persistent optimizer statistics (since InnoDB used to do random dives, so when you do a mysqldump, things can be different) – you can do SET GLOBAL innodb_analyze_is_persistent = 1; (read more: InnoDB Persistent Statistics at last, InnoDB Persistent Statistics Save the Day).

New improvements in optimizer: ORDER by on non-indexed columns, Multi Range Read (MRR), Index Condition Pushdown (ICP), query execution plan for DML statements, Batched Key Access (BKA). There is also optimizer tracing.

There is now some new instrumentations in Performance Schema (introduced in MySQL 5.5). There is also improved replication features, but that will be covered in a later talk.

Fractional seconds (microseconds) is introduced in MySQL 5.6. GET DIAGNOSTICS for stored procedures. Pluggable authentication (socket). memcached/InnoDB still in labs, hasn’t made it to a milestone release. Some 400+ bugs have been fixed. Lots of cleanup’s (old unused variables, commands removed).

Pay attention to MySQL 5.6 incompatible changes when you plan to upgrade.

Probably some good resources: What’s New in MySQL 5.6, MySQL 5.6 (Early Access Features) – InnoDB & Replication.

Related posts:

1. MySQL synchronous replication in practice with Galera by Oli Sennhauser
2. MySQL HA reloaded by Ivan Zoratti
3. MariaDB/MySQL users in Paris & Brussels

For example, suppose you have two transactions which run in parallel:

Transaction T1:

BEGIN;
    SET @t = NOW();
    UPDATE xtradb_table SET a= @t WHERE id = 5;
    UPDATE pbxt_table SET b= @t WHERE id = 5;
    COMMIT;

Transaction T2:

SET TRANSACTION ISOLATION LEVEL REPEATABLE READ;
    START TRANSACTION WITH CONSISTENT SNAPSHOT;
    SELECT t1.a, t2.b
      FROM xtradb_table t1 INNER JOIN pbxt_table t2 ON t1.id=t2.id
    WHERE t1.id = 5;

In the above case, it is possible, even with a "consistent" snapshot, to see the changes in a transaction only in InnoDB/XtraDB tables, and not in PBXT tables.

Naturally, it would be much better if the changes were visible no matter the storage engine used by the table. Thankfully this is possible, thanks to the introduction in MariaDB 5.3 of group commit and an enhanced storage engine API for COMMIT which allows storage engines to coordinate commit ordering and visibility with each other and with the binary log.

With these improvements, the START TRANSACTION WITH CONSISTENT SNAPSHOT statement in MariaDB 5.3 has been enhanced to ensure consistency in-between storage engines which support the new API. Currently the storage engines which support this are XtraDB (MariaDB's enhanced version of InnoDB) and PBXT. In addition, the binary log, while not a storage engine as such, also supports the new API and can provide a binlog position consistent with storage engine transaction snapshots.

This means that with transaction isolation level at least REPEATABLE READ, the START TRANSACTION WITH CONSISTENT SNAPSHOT statement can be used to ensure that queries will see a transaction-consistent view of the database between storage engines.

No longer is it possible for a query to see the changes from some transaction T in XtraDB tables without also seeing the changes T makes to PBXT tables.

More information on this can be found in the AskMonty Knowledgebase .

The 1st Latin American Conference about MySQL, NoSQL and Cloud technologies will be held in Buenos Aires in June. It is called the MariaDB NoSQL & Cloud Latin American Conference and we'll bring you more information as it becomes available.

read more

This variable is great for those who want to append comments to various queries, but still want the query cache to be able to serve such queries. Unfortunately, with MySQL, this is not currently possible.

In the past, I wrote a post on using MySQL Proxy which described a technique of monitoring queries through the proxy by appending IP addresses to the queries so one could track where they originated from. However, one pitfall to that was the MySQL query cache *does not* ignore the comment and treats them all as different queries (see the user comments for further discussion). (I did subsequently enhance that functionality implementing the SHOW PROXY PROCESSLIST command (often used in the Proxy Admin module), in large part because of this limitation.)

To enable it (in MariaDB 5.3+ and XtraDB), just add query_cache_strip_comments under the [mysqld] section in your my.cnf file and restart mysqld.

Alternatively, you can also set it dynamically:

mysql> set @@global.query_cache_strip_comments=1;
Query OK, 0 rows affected (0.04 sec)

mysql> show global variables like 'query_cache_strip_comments';
+----------------------------+-------+
| Variable_name              | Value |
+----------------------------+-------+
| query_cache_strip_comments | ON    |
+----------------------------+-------+

mysql> select version();
+---------------+
| version()     |
+---------------+
| 5.3.3-MariaDB |
+---------------+

Fwiw, from examining the source code, both implementations seemed to differ, but the end result is the same, and it’s a welcome addition, if you ask me.

Try reading the groups that are not in your regular reading list. Is there a question in there you can not answer even after a peek at the documentation?

After running MySQL for a few years, it is easy to skip over the problems of novices found in the Newbie section. But like the old story of the Emperor’s New Clothes, a new set of eyes can point out things we do not notice. Recently someone in the Newbie group wanted to set up a round robin for inserts over eight disks in an attempt to get 8x performance gain. Hmmm, if that did scale would it scale linearly(1) or close to it? Too bad I do not have the hardware or spare cycles to give it a try.

Or question on how to remove a ‘width=xxx’ pattern from a text field. Hmm, the old GIGO rules would dictate that some sort of ETL tool or script remove the unwanted chaff before it goes into the database. Do you use regular expressions in a PHP, Perl, or Java hack? Could a column oriented editor let you WYSIWYG the data? Maybe a trigger on insert that culls the unwanted width data? Hey, does your data have anything hidden in a text filed that should not be there and how best to get rid of it?

So read through a few forums and use them to exercise you mind. And if you see something in a question that you can answer, please do so.

1. Nothing scales linearly that you need to scale linearly. At least not on your budget and or schedule.

Peak performance is thrown out, 95%-98% is important.

Variance shouldn't be large.

Data manageability is rate of interrupts per server for the operations team. Rate of server growth much bigger than rate of new hires for the systems teams. A lot of the db team is from University of Wisconsin-Madison!

Why MySQL? Because it was there when he came. Mark and ops/engineering peers made it scale 10x. He likes MySQL for OLTP, InnoDB is "An amazing piece of software."

They can get 500,000 qps using a cached workload, which is on par with memcached.

What Facebook really does is OLTP for the social graph. The workload is secondary indexes, index-only queries, small joins but most queries use one table, multi-row transactions, majority of workload does not need SQL/optimizer, they do a physical and logical backup.

Most of this does not require SQL [blogger's note - they built Cassandra]. Why is the grass greener on the other side? automated replacement of failed nodes, less downtime on schema changes and/or fewer schema changes, multi-master, better compressions, etc.

Circa 2010, 13 million queries per second, 4 ms reads, 5 ms writes, 38GB peak network per second, etc.

Why so many servers? Big data high queries per seconds. They add servers to add IOPS, so they're interested in compression and flash, so they can get more IOPS. If they do remain on disk, write-optimized dbs are interesting too. About 10 people on the db team, which is very small for a company that size.

How to scale MySQL? Fix stalls to make use of capacity, improve efficiency to use fewer queries/fewer data. Fixing stalls doesn't make MySQL faster, makes it less slow.

[blogger's note - I stopped taking notes here because this is a rehash of the "How Facebook Does MySQL" talk that has been done over and over...]

[restarted when he started talking about data manageability again]

How Facebook got it's data manageable.

pylander- sheds load during a query pileup - kills dup queries, limits # of queries from some specific accounts -- take off on Highlander: there can be only one.

dogpile - collects data during a query pileup - gets perf counters and list of running queries, generates HTML page with interesting results.

Online schema change tool, for frequent schema changes, especially adding indexes. This briefly locks the table, to setup triggers to track changes, copy data to a new table with the new desired schema, replay changes to the new table, then briefly lock the table again as you rename the new table as the target table.

Manageability is a work in progress -- working on:
- make InnoDB compression work for OLTP
- Faker - tool for prefetching for replication slaves - replay workload is: page read, do some modification, page write. bottleneck might be disk reads, work is done by a single thread, transactions on master are concurrent. Faker has multiple threads replay transactions in "fake-changes" mode, no undo, no rollback, read-only, fetches into the buffer pool the pages needed for that transaction. Captures about 70% of disk reads for replication, they're working on fixes to get it up to 80-90%.
- auto replacement - replace failed and unhealthy MySQL servers.
- Auto resharding - sharding is easy, re-sharding is hard.

open issues in manageability:
diagnose why one host is slow, others are not.
....and some more.

Thus, we'd like to know what you'd like to see in the upcoming releases of the server.

Please take a couple seconds (literally) and let your vote be heard:

http://www.skysql.com/content/new-server-functionality-have-your-say

We thank you and look forward to hearing your thoughts and ideas!