Do you ever find yourself wanting to open several terminal windows and send the same commands to all of them? I’ve had this need many times, and I’ve never found a completely satisfactory solution. I’ve also known a lot of people who’ve written various sets of scripts to help them accomplish such tasks.

In no particular order, here are a few ways I’ve done this in the past:

1. Facebook’s pmysql client
2. The dsh tool
3. Several screen windows named remoteXXX, followed by a bash for-loop: while read cmd; do screen -X at remote# stuff "$cmd"; done
4. Using many PuTTY windows and the puttycs tool
5. Opening many tabs in KDE’s Kterm tool and selecting the options to send input to all tabs

Here …

If you ever had a replication slave that is severely behind, you probably noticed that it’s not catching up with a busy master at a steady pace. Instead, the “Seconds behind master” is going up and down so you can’t really tell whether the replica is catching up or not by looking at just few samples, unless these are spread apart. And even then you can’t tell at a glance when it is going to catch up.

Normally, the “severely behind” thing should not happen, but it does often happen in our consulting practice:

• sometimes replication would break and then it needs to catch up after it is fixed,
• other times new replication slave is built from a backup which is normally hours behind,
• or, it could be that replication slave became too slow to catch up due to missing index

Whatever the case is, single question I am being asked by the customer every time this happens is this: …

This is one of many perl script tools from Percona that makes life easier administrating, monitoring and testing mysql, this one solves the need to alter a table (adding a key, column, partitioning, etc…) on a production system that can’t be taken down. Many of the current Percona tools came from maatkit, developed by Baron [...]

What is Ansible?

Ansible is a configuration management and deployment system, like Puppet, Capistrano, Fabric, and Chef. Its aim is to be radically simple and let you use your existing scripts to help with cluster configuration and software deployment whenever possible. Here are the ways that Ansible differentiates itself.

Simplicity

Ansible does not include a client/server architecture with pull-based clients (although in more recent versions, it does include pull-based configuration and deployment). Rather, it uses pre-existing network infrastructure: SSH. Every company has SSH installed on their cluster servers, and Ansible simply rides on top of this infrastructure to get the code and configuration out to the nodes.

Language Agnostic

You can write modules for Ansible in …

A set of internet tools, is available at otala.net/tools.shtml

• Traceroute
• Whois & JWhois
• NSLookup
• Ping
• DNSWalk
• NetStat
• ENUM (e164.org phone number lookup)

More to come, as (if/when) I get time -- suggestions welcome

Everyone has their preferred method of communication, which they tend to overuse at the cost of other methods. At MySQL AB, email was preferred in many situations where picking up the phone would have saved loads of work, frustration and distractions. SkySQL Ab as a company is equally distributed across continents and time zones. Not surprisingly, we have inherited some of MySQL's bad habits, and created a set of new habits, good and bad. The purpose of this write-up is to take a step back and reflect upon which tool fits which purpose, focusing on a virtual company with colleagues spanning many time zones.

read more

Do you rely on pt-diskstats from Percona Toolkit instead of the standard iostat a lot? There appears to be a nasty bug in pt-diskstats 2.1, which makes it produce bad results.

I noticed some of the numbers I was getting weren’t right, so I tried running iostat and two different releases of pt-diskstat side by side. Here’s what I got:

I can understand the slight differences between the lines in iostat and pt-diskstats 2.0 as they probably weren’t reading /proc/diskstats contents in the same moments, so the values they were seeing could be a bit different. However both lines practically show the same thing.

On the other hand, the line based on the pt-diskstats 2.1 output looks completely different. It seems like maybe the script returns some kind of averages rather than per second deltas. I filed a …

1. Percona Toolkit
Percona Toolkit (aka Maatkit and Aspersa) is must have collection of advanced command-line tools which helps in performing tasks that are too difficult or complex to perform manually.

2. Mydumper
Mydumper is a high-performance multi-threaded backup/restore tool for MySQL. It’s up to 10x faster compared to mysqldump, can take consistent snapshots and provides File compression on-the-fly. Though it’s still under active development but is well tested/used in production on some large installations.

3. MySQL Master HA
This tool helps to maintain your Master-Slave replication setup. A primary objective is automating master fail-over and slave …

This week I was given the task of repopulating our entire primary database cluster.  This was due to an alter that had to be performed on our largest table.  It was easiest to run it on one host and populate the dataset from that host everywhere.

I recalled a while back reading a blog post from Tumblr about how to chain a copy to multiple hosts using a combination of nc, tar, and pigz.  I used this, with a few other things to greatly speed up our repopulation process.  As I was repopulating production servers, I did a combination of raw data copy and xtrabackup streams across our servers, depending on the position in our replication setup.

For a normal straight copy, here’s what I did:

On the last host, configure netcat to listen and then pipe the output through pigz and tar to …

Taxonomy of Database Tools

In the MySQL ecosphere there is an ecosystem of tools.  Like real-world ecosystems, the “creatures” in the MySQL tools ecosystem can be classified and organized by a taxonomy.  There are already multiple taxonomies of software bugs (e.g. A Taxonomy of Bugs), but as far as I know this is the first Taxonomy of Database Tools.  A taxonomy of database tools serves useful purposes, as discussed in the previously linked page.  For me, the most useful purpose is the high-level ecosystem view which I use to compare MySQL tools to  …