As announced, I held a short talk on switching from MySQL community edition (especially 5.1) to MariaDB (currently 5.2.6) at this years LinuxTag in Berlin. Here are the (German) slides for reference: (In case you cannot see the embedded presentation, … Weiterlesen →

Demonstrating distributed set processing performance Shard-Query + ICE scales very well up to at least 20 nodes

This post is a detailed performance analysis of what I’ve coined “distributed set processing”.

Please also read this post’s “sister post” which describes the distributed set processing technique.

Also, remember that Percona can help you get up and running using these tools in no time flat. We are the only ones with experience with them.

20 is the maximum number of nodes that I am allowed to create in EC2. I can test further on our 32 core system, but I wanted to do a real world “cloud” test to show that this works over the network, in a real world environment. There is a slight performance oddity at 16 nodes. I suspect the EC2 environment is the reason for this, …

Can Shard-Query scale to 20 nodes?

Peter asked this question in comments to to my previous Shard-Query benchmark. Actually he asked if it could scale to 50, but testing 20 was all I could due to to EC2 and time limits. I think the results at 20 nodes are very useful to understand the performance:

I will shortly release another blog post focusing on ICE performance at 20 nodes, but before that, I want to give a quick preview, then explain exactly how Shard-Query works.

Yes, Shard-Query scales very well at 20 nodes
Distributed set processing (theory) What is SQL?

As you probably know, SQL stands for “structured query language”. It isn’t so much the language that is structured, but …

Why would one want to do this, you may ask?  Well, for starters, it makes a great ‘canary-in-the-coal-mine‘ in regards to backwards compatibility!

For Drizzle, we’ve created some tables (via the randgen’s data generator if you are curious), saved a copy of the datadir, and then created a test case that uses said datadir for the test server.  The test executes some simple SQL queries to make sure we can read the tables properly.  This way, if we ever do something to either the server or .dfe format (data format exchange – had a most enlightening conversation with the team about …

Over the last few years I have worked for few different companies that utilized versions of relayed replication with the MySQL databases. Their data handler connections where done with Java connection pooling or custom built PHP/PERL code. This allowed each company to spread the load of heavy queries and maintain a decent up time based on the load of their servers. One of the companies comes to mind often when I think of the MySQL cluster.
The company was growing and we made plans to move out of leased servers and build out our racks in a co-location center. This would have been a great time to think ahead and move towards or at the very least heavily evaluate a MySQL Cluster solution. I often got push back on this topic. Granted hindsight is always 20/20. Everything was new and fresh and we could have tested and benchmarked before live traffic was on the systems. Instead we continued with a basic MySQL replication set up (1 master --> 2 …

It has only been a few days since the 0.2.0 release of mydumper but there have been some big changes since then.  I will try and go over them all here.
Mydumper 0.2.1
Mydumper 0.2.1 has been released today, many thanks for all those who have been testing the trunk source, the feedback has gone a long way to making fixes and improvements to mydumper.  You can download the source for it here.
Myloader
A few days ago Mark Callaghan asked about restoring mydumper backups.  There is of course the great ' …

At Open Database Camp in Sardinia last week, there was a hackfest to try to get replication working from MySQL to MongoDB. In only 2 hours and 45 minutes, a small group of hackers did it! This week I interview Flavio Percoco, the MongoDB guru on the team, who talked about getting replication from MySQL to MongoDB working via Tungsten Replicator (which we talked about on last week's podcast).

read more

It has only been a few days since the 0.2.0 release of mydumper but there have been some big changes since then.  I will try and go over them all here.

Mydumper 0.2.1

Mydumper 0.2.1 has been released today, many thanks for all those who have been testing the trunk source, the feedback has gone a long way to making fixes and improvements to mydumper.  You can download the source for it here.

Myloader

A few days ago Mark Callaghan asked about restoring mydumper backups.  There is of course the great ‘ …

Please note that when upgrading from a version previous to 4.5  to this one, it is required that MONyog detects the replication topology of registered servers before the ‘registered servers’ page can display. This may take a few minutes with a large number of registered servers. Please be patient for as long. Note that this delay will only happen once.

Changes (as compared to 4.5) include:

Features:
* This release focuses on improving performance of the MONyog built-in HTTP daemon. You will find up to 10 times performance improvements for most MONyog pages. This has been achieved by internal code optimizations, use of compression (if browser supports) and minification/optimization of HTML, Javascript, CSS and graphics.

Bug fixes:
* Prepared statements were not handled properly by the Query Analyzer.
* Events (as …

There are two ways InnoDB can organize tablespaces. First is when all data, indexes and system buffers are stored in a single tablespace. This is typicaly one or several ibdata files. A well known innodb_file_per_table option brings the second one. Tables and system areas are split into different files. Usually system tablespace is located in ibdata1 file and every InnoDB table has two files e.g.  actor.frm and actor.ibd.

The annoying thing about .ibd files you can’t easily copy the an .ibd file to another MySQL server. If you try to very often you’ll get an error in the log:

InnoDB: Error: tablespace id is 10 in the data dictionary
InnoDB: but in file ./sakila/actor.ibd it is 15!

However sometimes you have to connect the .ibd file to an alien ibdata1.

There are several situation when you have to:

1. ibdata1 is erroneously removed

2. ibdata1 is …