We’ve been working on automatic master promotion for MySQL using lbpool and I’m reminded of a few ideas we had after reading about MMM.

The MySQL Master-Master replication (often in active-passive mode) is popular pattern used by many companies using MySQL for scale out. Most of the companies would have some internal scripts to handle things as automatic fallback and slave cloning but no Open Source solution was made available.

Few months ago we were asked to implement such solution for one of the customers and they kindly agreed to let us release things under GPL2 License, and we gave them reduced rate for being Open Source friendly.

We actually considered a dual master approach but abandoned because it was non …

So far, the MySQL Server RPM packages as provided by MySQL AB used to automatically start the mysqld process after the package has been installed. It has been like that since the very beginning and we think of it as a convenience for our users when they want to get up and running quickly.

However, Kristian raised an interesting point in BUG#27072 where he points out that automatically starting mysqld during the RPM installation might not always be the desired behaviour, especially in automatic installation environments or during a fresh installation (where the system might not be fully configured yet). Therefore he proposes to change this behaviour to not start mysqld as part of the installation.

While I personally agree with his proposal, this is of course a tricky decision: our users are familiar with …

I just stumbled over Sebastian's blog entry:

The second Free and Open Source Conference "FrOSCon" takes place on August, 25th and 26th 2007 in Sankt Augustin, near Bonn, Germany. The conference is once again hosted by the faculty of computer science of the University of Applied Sciences Bonn-Rhein-Sieg in collaboration with the student body and the Linux/Unix User Group Sankt Augustin.

I attended FrOSCon last year and found it a very pleasant conference to be …

So, the new DST changes in the US caused a small stir among system administrators recently. We got all of our servers updated and verified they were working before the even. Or so we thought.

I noticed today that our 3PM Eastern newsletters arrived in my inbox at 3PM alright. However, I am in Central time. My immediate assumption was that we missed the server that sends that email out. Logging in I found the time correct on the server. It had received the appropriate updates thanks to portage. So, what happened? I looked at /etc/crontab and all was fine. I then looked at the system log where cron jobs are logged. Oddly, that log line said the job started at 15:00. I knew that was not correct. I started looking around at other cron jobs on other servers, especially ones that wrote files to disk. Sure enough, every server I checked was doing things an hour behind except one. It just so happens we had restarted cron on this server however last …

OK lazy web…. You have to help me out!

I try to monitor every performance metric I can within my cluster. Memcached efficiency, MySQL key buffer efficiency, etc.

One thing I can’t benchmark is the efficiency of the Linux filesystem buffer (buffer cache).

From all the research I have done there’s no way to to see the number of hits or total reads done by the file system cache.

Has anyone solved this problem?

I did some comparison of ZFS vs XFS last week to review the current state of the art in filesystems.

Long story short. XFS is still the reigning champion (at least on Linux). XFS beats out most filesystem benchmarks across the board. Reiser does well on directories with lots of small files but not enough to justify not using XFS.

Reiser FS is out of the picture honestly. First, it just doesn’t perform very well. Second, Hans Reiser is probably going to prison for murdering his wife and is selling the company to pay off his legal costs.

ZFS would have a shot on …

Some of the common conceptions of data storage seem to have been blown out of the water.

Two things I found interesting:

In their study they found that there was no correlation between disk failure rates and utilization, environmental conditions such as temperature, or age. This means that high disk utilization or age of the disk have no significant impact on the probability that it will fail.

…

They observed that older disks had a much lower failure rates then newer disks, where the newer disks in general were less expensive.

Which makes me think that buying used HDDs off Craigslist might not be a bad idea.

One could buy cheap 15k RPM low latency disks from a few years ago and forget about the storage capacity in exchange for FAST seek time.

Of course it depends on how …

Hurry up, submit a paper! The LinuxTag Call for Papers ends tomorrow, February 16th.

Short info about LinuxTag from the homepage:

LinuxTag 2007 opens doors from May 30 to June 2, 2007 on Berlin Expo Center under the Funkturm. We invite users and experts to learn at Europe’s leading conference and expo more about the potential of Linux, Open Source, and Free Software.

FOSDEM, the Free and Open Source Software Developers' European Meeting will again take place in Brussels, Belgium on February, 24th & 25th. I went there last year and really had a great time. This year, I submitted a proposal for a lightning talk, which has been accepted! I've just been informed that the lightning talk schedule is now published. My talk "What's new at MySQL AB" will be on Saturday, at 17:00. I look forward to being there!

The following is what i wrote on “43people.com” about the boss. I thought it was worth keeping in my own archives, since it’s actually a story about my life as it pertains to Mr. Bezos.

Back a few years ago, I was taking some classes down in Edmonds. The one I’m thinking of in particular was on the care and feeding of unix. We were using red hat linux 6.0 or some crufty version that wasn’t so crufty at the time.

Anyway, the prof didn’t require that we buy any books, but he made some suggestions. And he also suggested that we buy them on this new fangled “Internet” thing through a few of his friends down south in Seattle at this place called Amazon.com.

And thus was my introduction to O’Reilly and Associates. I soon thereafter bought a book called “ …