One common job for a DBA is working with a Development Team member on a batch loading process that is taking more time than expected.

Often once you start probing into what has already been done to performance tune the load process, it seems you often learn that very little has, in fact, been done to improve efficiency and speed.

Since there are so many things that can be done to improve this process, it really is outside the scope of this article to cover them all.  Hopefully this will serve as a guide to get you started in the process.

Let’s outline best practices for loading data into MySQL very quickly.  While this is not a comprehensive list of loading methods and configuration, it is a good starting point.

MySQL Configuration

Assuming you are loading into InnoDB tables (and you should probably be doing so), you will want to ensure that MySQL is properly …

Sometimes data sets are so large, a mysqldump to load a slave is just not practical.  With some of the systems we have administrated, we have had data so large it would have taken days to load the slave when it became out of sync with the master.  When this happens, we usually rely upon Percona’s XTRABackup utility which allows us to make a hot/online backup of the master to use for loading the slave.

In the old days we had to rely upon a third-party tool called ibbackup, or InnoDB Hot Backup utility to do this task. In many ways XTRABackup is a replacement for this tool and has in fact surpassed the ibbackup utility in features and function.

The most efficient way we have found to transfer that data to the slave is the use of the netcat utility.

We also use the screen command since we expect this could take quite some time and don’t want to take the chance that a network connection issue, or a dropped VPN, …

Sometimes you encounter a server with multiple tables of a particular storage engine which you need to convert to another storage engine.  For us, this often happens when we find systems running MyISAM and we want to get these over to InnoDB.

There are a number of reasons to consider converting a table to another storage engine, such as performance, gaining additional features such as Foreign Keys, and so on.  You should, however, stop to consider that not all storage engines are created the same and do not offer the same features.

If there are hundreds of tables, the process can be very time consuming so we put together a simple bash script to automate this process.

#!/bin/sh

MY_USER="root"
MY_PASSWORD="mypassword"
MY_HOST="127.0.0.1"
MY_PORT=3306
NEW_ENGINE="InnoDB"

TABLES=`mysql -u$MY_USER -p$MY_PASSWORD -h$MY_HOST -P$MY_PORT -e"SELECT CONCAT(TABLE_SCHEMA,'.',TABLE_NAME) AS 'TABLE' FROM …

Have you ever tried to start a server like MySQL and been amazed to see an error that the port is already in use? You rack your brain and try to figure out what it would be to no avail. Sometimes you do a “ps” in Linux and don’t even see anything that you think would be using the port. Well, forutantely, there are some tricks to help you find out without doing a reboot. If it is a production server, a reboot may not be an option anyway!

Below are some methods to help. We will start by looking at the “fuser” utility provided with many Linux distros:

fuser -n tcp 80
80/tcp:               1029  1030  1824  1838  1839  1840  1841 13972 14136 14137 14712

This example shows a simple check of everything using port 80. What you see above is a list of PIDs that are using that port. Now we could probably just do a simple “ps” to figure out what it is. You might also want to get more info by doing something like the …

Some time back, when a client wanted us to setup MySQL Enterprise Monitor, we were surprised to find out that disk monitoring was not available! We worked hard to come up with a solution. Eventually, we decided to setup a custom agent to monitor the disk. Below is the result of that.

While this script may not work as-is for everyone, it should at least provide a basis for such a script. This script has been modified to send an email instead of plug directly into the MySQL Enterprise Monitor. But, it hopefully will get our creative juices flowing…

#!/bin/bash
#
# This script does a very simple test for checking disk space.
#
# Itchy Ninja Software: http://www.itchyninja.com
#

CHECKDISK=`df -h | awk '{print $5}' | grep % | grep -v Use | sort -n | tail -1 | cut -d "%" -f1 -`
ALERT_VALUE="80"
MAIL_USER="root@localhost.com"
MAIL_SUBJECT="Daily Disk Check"

if [ "$CHECKDISK" -ge "$ALERT_VALUE" ]; then
echo "At least one of my disks is nearly …

Many of the DBA’s at Itchy Ninja Software, began their careers as PHP Programmers, or before… We remember times sitting in a small office together writing PHP code before there were even books on the subject! We were scouring the web looking for examples and such and relying heavy on the documentation at the php.net site!

We wrote custom shopping carts before the days of oscommerce, zencart, and such. We wrote code to interface with payment gateways and process credit cards live. We also wrote code to pull products from MySQL databases, show staff members, and such. We worked for a newspaper group and were writing news “engines” to pull the articles, archive data, etc.

Why am I reminiscing about all of that? Well, because frankly, we have been there. We know a lot of other developers are still there doing the kind of work that we began with. We know that we can help others not to make the same mistakes we have seen some of …

Part of an article originally published in the article MySQL Forks: Which one is right for me?, published in the June edition of php[architect]. The preceding portions of the article are available: A Brief History of MySQL Oracle MySQL Percona Server MariaDB Other Derivatives The MySQL ecosystem has never been stronger.  Oracle has responded to [...]

I was intrigued by a post from David Stokes entitled MySQL and Password Security.  It discusses some new options for command line password security in MySQL 5.6.  First, I want to establish that it doesn’t really provide “security”, but then I’ll explain why that doesn’t matter. MySQL 5.6 appears to encrypt the password in a [...]

Part of an article originally published in the article MySQL Forks: Which one is right for me?, published in the June edition of php[architect]. The preceding portions of the article are available: A Brief History of MySQL Oracle MySQL Percona Server MariaDB Facebook patch The Facebook Patch is a patchset used by Facebook to run [...]

Part of an article originally published in the article MySQL Forks: Which one is right for me?, published in the June edition of php[architect]. The preceding portions of the article are available: A Brief History of MySQL Oracle MySQL Percona Server Having left Sun in September 2008, Monty Widenius started Monty Program AB to continue [...]