Eric Bergen from Proven Scaling (which I had the pleasure to meet in person during the MySQL Conference & Expo in Santa Clara last month) was kind enough to send me a patch for the mylmbackup tool, which justifies a new release:

Attached is a patch file for mylvmbackup that adds the ability to use
lvm version 2 and perform innodb recovery on the snapshot prior to
creating a tar ball. The option is named --innodb-recover.

I've also fixed a bug with default value handling for command line
options. In version 0.4 if a config file was specified default values
in the script were all changed to blank. This means that the config
file had to supply values for every variable instead of just the
values that need to be changed from default.

This …

We’ve been doing a lot of performance analysis of MySQL and SATA disks over the last few days and I wanted to get some thoughts out in the open to see what you guys thought.

Now that Debian Etch is out we’re seriously looking at making the jump to a full 64bit OS with 8G of memory on each of our boxes.

This also involves benchmarking SATA and potentially migrating away from MyISAM which, while very trusty, is seriously showing its age for us.

First off is raw IO tuning.

XFS benchmarks show about a 20% performance boost from bonnie++ over ext3. This is pretty obvious. Tuning readahead with ‘blockdev’ didn’t seem to help performance much. SATA seems pretty tuned when compared to hdparm and IDE drives.

After fully tuning it seems we can get about 90MBps out or our disks. Not to shabby. The Linux md driver on RAID 0 didn’t seem to boost IO performance much (and I’m pretty disappointed). Even …

Baron Schwartz was nice enough to make recordings of about a dozen talks at the recent MySQL conference. The original uploaded files were in ogg format and wouldn’t work on an ipod and weren’t very portable.

With his help, I uploaded the original MP3 files and made a podcast which contains all the links to the sessions and should work fine with iTunes and an ipod.

Here’s the link to the podcast. Just add it to iTunes or your favorite podcasting software and off you go.

In two weeks from now, the MySQL Conference & Expo 2007 will take place in Santa Clara, California. Our team has been very busy with working with the conference management team from O'Reilly to prepare for this event, especially Jay who is in charge of organizing and arranging the more than 110 sessions that will be held in several parallel tracks. The conference program looks very exciting, I am impressed by the wide range of topics and quality of speakers that we have lined up.

In addition to the presentations and tutorials, there will also be an Expo, where partner companies will exhibit their products …

We’ve been working on automatic master promotion for MySQL using lbpool and I’m reminded of a few ideas we had after reading about MMM.

The MySQL Master-Master replication (often in active-passive mode) is popular pattern used by many companies using MySQL for scale out. Most of the companies would have some internal scripts to handle things as automatic fallback and slave cloning but no Open Source solution was made available.

Few months ago we were asked to implement such solution for one of the customers and they kindly agreed to let us release things under GPL2 License, and we gave them reduced rate for being Open Source friendly.

We actually considered a dual master approach but abandoned because it was non …

So far, the MySQL Server RPM packages as provided by MySQL AB used to automatically start the mysqld process after the package has been installed. It has been like that since the very beginning and we think of it as a convenience for our users when they want to get up and running quickly.

However, Kristian raised an interesting point in BUG#27072 where he points out that automatically starting mysqld during the RPM installation might not always be the desired behaviour, especially in automatic installation environments or during a fresh installation (where the system might not be fully configured yet). Therefore he proposes to change this behaviour to not start mysqld as part of the installation.

While I personally agree with his proposal, this is of course a tricky decision: our users are familiar with …

I just stumbled over Sebastian's blog entry:

The second Free and Open Source Conference "FrOSCon" takes place on August, 25th and 26th 2007 in Sankt Augustin, near Bonn, Germany. The conference is once again hosted by the faculty of computer science of the University of Applied Sciences Bonn-Rhein-Sieg in collaboration with the student body and the Linux/Unix User Group Sankt Augustin.

I attended FrOSCon last year and found it a very pleasant conference to be …

So, the new DST changes in the US caused a small stir among system administrators recently. We got all of our servers updated and verified they were working before the even. Or so we thought.

I noticed today that our 3PM Eastern newsletters arrived in my inbox at 3PM alright. However, I am in Central time. My immediate assumption was that we missed the server that sends that email out. Logging in I found the time correct on the server. It had received the appropriate updates thanks to portage. So, what happened? I looked at /etc/crontab and all was fine. I then looked at the system log where cron jobs are logged. Oddly, that log line said the job started at 15:00. I knew that was not correct. I started looking around at other cron jobs on other servers, especially ones that wrote files to disk. Sure enough, every server I checked was doing things an hour behind except one. It just so happens we had restarted cron on this server however last …

OK lazy web…. You have to help me out!

I try to monitor every performance metric I can within my cluster. Memcached efficiency, MySQL key buffer efficiency, etc.

One thing I can’t benchmark is the efficiency of the Linux filesystem buffer (buffer cache).

From all the research I have done there’s no way to to see the number of hits or total reads done by the file system cache.

Has anyone solved this problem?

I did some comparison of ZFS vs XFS last week to review the current state of the art in filesystems.

Long story short. XFS is still the reigning champion (at least on Linux). XFS beats out most filesystem benchmarks across the board. Reiser does well on directories with lots of small files but not enough to justify not using XFS.

Reiser FS is out of the picture honestly. First, it just doesn’t perform very well. Second, Hans Reiser is probably going to prison for murdering his wife and is selling the company to pay off his legal costs.

ZFS would have a shot on …