Over the past few weeks (years really) there has been some discussion on sharding. Instead of discussing when sharding is required, as there are good discussions on this already, I want to discuss how I would like to have sharding implemented in the database.

I want the database to handle sharding automatically, and where it can't be automatic, I want the database to help as much as it can.  Just like I want my business logic in a language ideally suited to it, and not stored procs (generally, there are always exceptions); I want all my physical persistence to be handled by the place that already does most of it, the database.  Having the database handle some of the physical persistence and the object relational layer handle the sharding logic isn’t ideal to me, and not just because the current object relational layers don’t have all the sharding functionality I want.  So here is what I want my database to do.   …

Mark has found Double sysbench throughput with TCMalloc with is quite amazing.

In Drizzle, if TCMalloc is present, we’ve linked it in (and on Solaris, mtmalloc) for years now. well…. Drizzle years, which are rather like dog years.

Monty Taylor recently took my import progress patch and ported it into Drizzle.  I managed to spot a small bug in his conversion so took a bzr branch and fixed it.  I then spoke to Jay Pipes who suggested I could improve it by making it take a parameter which would be the number of lines per output (rather than fixed at 1000 for the original patch).

After I made the improvement I have started taking on more work for the Drizzle project in my spare time and have somehow got myself into the top 20 contributors.  But that is another story.

I have now taken my work on Drizzle …

I'm pleased to announce the release of DBD::drizzle 0.003. In this release, I've made many changes, particularly those to get it working with the stock drizzle client library, which is a bit of a moving target, but that's how it goes with new, quickly-evolving projects.

Caveat: This is an Alpha release.

I've added back the tests from DBD::mysql and modified them to work with Drizzle. Some pass just fine, others report 'out of memory'. I hope to figure this out soon, though this is that last version where I'll use the client library that comes with Drizzle. I'll be changing to using Eric Day's improved client library, libdrizzle, and maybe in the process of changing to that, will squash these bugs.

The files:

  file: $CPAN/authors/id/C/CA/CAPTTOFU/DBD-drizzle-0.003.tar.gz
  size: 89682 bytes
   md5: …

Now that the core of Gearman is rewritten in C, I’ve started working on a new module to provide persistent queues and queue replication. The current Gearman implementation is fine when the jobs can be unreliable, but there are many cases when we want to guarantee that jobs get run.

Because Gearman is such a flexible system, there are a couple of places where we can plug in a persistent queue and/or replication. Now, in most cases, these features are only going to be useful for background jobs since failed foreground jobs can be detected and restarted by a client. Many applications using Gearman want to just throw a job in the queue and go back to doing something else, not caring when it gets done. Enter persistent queues. It may also be the case that a job should be run multiple times, mot likely on a separate set of servers for backup and geographic redundancy. I’m currently …

#mysql isn’t too fair to include, as it’s really about users, not dev. #mysql-ndb is there because i heart ndb.

Oh, and linux.conf.au is there because it’s *awesome* and you should go.

Totally unscientific due to i’m only taking a sample once and whatever… but it kinda interests me…

Today I spent a couple of hours at Amazon, because I have a friend who works there who invited me, giving some completely unstructured presentations.

The first I call "All these MySQL forks. WTF?!", giving some history and background and direction and impressions on MySQL mainline development, the Percona tree, the OurDelta effort, and of course Drizzle.

The second I call "Why distributed source control rocks!", talking about the workflow of git/bzr/hg compared and contrasted with p4/svn.

It was fun. And I had some ideas for bzr (that can also work for git and hg) that I will mention in a later post.

Red Hat gets a new VP. rPath appoints former Red Hat COO as chairman. Ubuntu Launchpad to go open source. Open source Windows. SCO formally files for Chapter 11. And more.

Official announcements
Industry Veteran Greg Symon Joins Red Hat as Vice President and General Manager of North American Sales Red Hat

rPath Welcomes Ex-Red Hat COO Tim Buckley rPath

Liferay Portal Released in Enterprise Edition LifeRay

Adaptive Planning …

Sun Microsystems - Employment

As usual, you can likely ignore the “Menlo Park, CA” part (I haven’t checked though… but am 99% sure).

A feature of the MySQL server that is used a lot, and yet is a source of much user confusion, code complexity, and multiprocessor lock contention, is logging. Query logging, slow query logging, and the new 5.1 feature, "log to table".

I've removed most all of that stuff from Drizzle (and removed two or three sets of now-no-longer-necessary mutex locks in the process), and replaced it with hooks into a logging plugin subsystem, and have implemented two plugins for it. One logs to a file, and the other logs to syslog.

The output looks almost completely unlike the current MySQL logging. There are no hash-prefixed pseudocomments, for one thing. And there is no distinction between the query log and the slow query log. Queries get logged, and the amount of time each query takes gets logged with it. This subsumes the "micro-slow patch" that is spreading around in the MySQL legacy world.

The current format is …