At Twitter, MySQL is an important part of our storage story. We can have a typical database node with thousands of open connections. It works mostly fine because generally only a few of these connections actively working inside the server. Given the scale at which we have to operate spikes happen – mostly at unexpected times and with unexpected ferocity. The challenge is how to control concurrency within MySQL server during spikes.

Historically we have been using innodb_thread_concurrency to control number of active threads within the server. In Twitter MySQL-5.5 we introduced the idea of Query Throttling where incoming requests are throttled if the number of active threads goes beyond a configurable threshold. These solutions focus on maintaining the health of the system. However, they don’t help us to scale to the workload spikes. With …

So, I’ve been looking around for a while (and a few times now) for any good resources that cover a bunch of MySQL architecture and technical details aimed towards the technically proficient but not MySQL literate audience. I haven’t really found anything. I mean, there’s the (huge and very detailed) MySQL manual, there’s the MySQL Internals manual (which is sometimes only 10 years out of date) and there’s various blog entries around the place. So I thought I’d write something explaining roughly how it all fits together and what it does to your system (processes, threads, IO etc).(Basically, I’ve found myself explaining this enough times in the past few years that I should really write it down and just point people to my blog). I’ve linked to things for more reading. You should probably …

It’s been a little while since I blogged on MySQL on POWER (last time was thinking that new releases would be much better for running on POWER). Well, I recently grabbed the MySQL 5.6.20 source tarball and had a go with it on a POWER8 system in the lab. There is good news: I now only need one patch to have it function pretty flawlessly (no crashes). Unfortunately, there’s still a bit of an odd thing with some of the InnoDB mutex code (bug filed at some point soon).

But, with this one patch applied, I was getting okay sysbench results and things are looking good.

Now just to hope the MySQL team applies my other patches that improve things on POWER. To be honest, I’m a bit disappointed many of them have sat there for this long… it doesn’t help build a development community when patches can sit for months without either …

Today we released TokuDB® v7.5, the latest version of Tokutek’s storage engine for MySQL and MariaDB.

I’ll be publishing two blogs next week to go into more details about our new “Read Free Replication”, but here are high level descriptions of the most important new features.

Read Free Replication

TokuDB replication slaves can now be configured to process the binary logs with virtually no read IO. This is accomplished via two new server parameters: one to allow the skipping of uniqueness checks (for inserts and updates), the other to eliminate read-before-write behavior (for updates and deletes). The two other conditions are that the slave must be in read-only mode and replication must be row based.

Hot Backup Now Supports Multiple Directories (Enterprise Edition)

The original implementation of our Hot Backup functionality was only capable of …

Over the past few months, I’ve seen an increase in the following use case while working on performance and schema review engagements:

I need to store exponentially increasing amounts of data and analyze all of it in real-time.

This is also known simply as: “We have big data.” Typically, this data is used for user interaction analysis, ad tracking, or other common click stream applications. However, it can also be seen in threat assessment (ddos mitigation, etc), financial forecasting, and other applications as well. While MySQL (and other OLTP systems) can handle this to a degree, it is by no means a forte. Some of the pain points include:

• Cost of rapidly increasing, expensive disk storage (OLTP disks need to be fast == $$)
• Performance decrease as the data size increases …

September 18, 2014 By Severalnines

Galera Cluster is a popular choice for achieving high availability using synchronous replication. Though if you are planning to run huge sites with many DB objects (tables), a few tweaks are necessary. 

Yes, you might have been successful in loading your 1000s of databases and 1000s of tables, but what happens if you have a node failure and Galera recovery fails?

In this blog post we will show you how to determine one common error related to the open_files_limit that MySQL imposes, and also to spot another potential pitfall.

Open_files_limit

If you are using wsrep_sst_method=xtrabackup or wsrep_sst_method=xtrabackup-v2 then you will find a log file in the data directory of the donor node. This log file is called innobackup.backup.log.

140912 19:10:15  innobackupex: Done. …

I recently encountered a situation in which after running Percona Toolkit’s pt-table-checksum on a customer system, 95% of the table on the MySQL master was different on the MySQL slave. Although this table was not a critical part of the infrastructure, from time to time, writes to the table from the master would break replication. Additionally, this table has about 6 million rows, and running pt-table-sync would take sometime. Rebuilding the slave from backup of the master would not be an easy option as well since the slave acts as an archive where it has a lot more data than the master.

So how did we solve it? With pt-online-schema-change and a NOOP ALTER.

pt-online-schema-change --alter 'ENGINE=INNODB' D=dbname,t=tblname

How is it …

September 17, 2014 By Severalnines

The Severalnines team is pleased to announce the release of ClusterControl 1.2.8. This release contains key new features along with performance improvements and bug fixes. We have outlined some of the key new features below. 

Highlights of ClusterControl 1.2.8 include:

• YUM/APT repositories for ClusterControl
• Deployment and scaling of single-node MySQL, MariaDB and MongoDB
• Alerts and incident tracking with PagerDuty 
• Unified Event Viewer
• New flexible alarms/email notification system
• Audit logging - Administrator activity tracking
• Global MySQL User Management
• New default dashboards for MySQL/MariaDB
• Puppet Module …

It's that time of the year again: yes, Oracle Open World is coming up and with that I'll be travelling to San Francisco. New for this year is that we are part of the main Open World event and therefore have our own MySQL Central. Here you will have the opportunity of meeting many of the engineers behind MySQL, discuss technical problems you have, and also learn some about how we look at the future of the MySQL ecosystem.

This year, me and Narayanan Venkateswaran will be presenting two sessions:

Elastic Scalability in MySQL Fabric with OpenStack (Thursday, Oct 2, 1:15 PM-2:00 PM in Moscone South, 252)

In this session you will see …

As the amount of data companies are interested in collecting grows, life becomes all the more difficult for IT staff at all levels within an organization. SAS Enterprise storage devices that were once considered giants are now being phased out in favor of SSD Arrays with features such as de-duplication, tape storage has pretty much been abandoned and the same goes without saying for database engines.

For many customers just storing data is not enough because of the CAPEX and OPEX that is involved, smarter ways of storing the same data are required and since databases generally account for the greatest portion of storage requirements across an application stack. Lately they are used not only for storing data but also for storing logs in many cases. IT managers, developers and system administrators very often turn to the DBA and pose the time old question “is there a way we can cut down on the space the database is taking …