Last week at the Drizzle developer day a few people were surprised at the number of plugin interfaces we’ve created already. I created this image to give a sampling of interfaces and plugins:

As you can see, things are quite different from MySQL.

Thanks to everyone who came out for the Gearman and libdrizzle talks at the MySQL Confrence & Expo and the Percona Performance Conference! Links to the slides are provided below:

• The Return of Gearman (Percona)
• libdrizzle: A New Client Library for Drizzle and MySQL (MySQL)
• Map/Reduce and Queues for MySQL Using Gearman (MySQL)

If you want to learn more about Gearman, be sure to check out gearman.org or the upcoming 3-hour tutorial or session at OSCON in July!

I’m wrapping up some Gearman work and packing for the users conference, it should be fun and exciting! I thought I’d send out a reminder that Brian and I will be talking about Gearman at both the MySQL UC and Percona Performance conference, including the possibility of a new release of Gearman. Here’s a hint:

lap> ./gearmand/gearmand -vv -t 4
Starting up
Initializing libevent for main thread
Method for libevent: epoll
Trying to listen on 0.0.0.0:4730
Listening on 0.0.0.0:4730
Creating wakeup pipe
Creating 4 threads
Initializing libevent for IO thread
Creating IO thread wakeup pipe
Thread 1 created
[   1] Entering thread event loop
Initializing libevent

Over the past couple days I’ve been tearing through the network I/O code inside of Drizzle. I moved anything that touched NET into the Protocol class (reading commands, preferences, error checking) and wrapped them with new Protocol methods. I then moved the NET structure out of Session (or THD for the MySQL folks) and into Protocol. At this point all network I/O was going through Protocol, which allowed me to do a proper abstraction layer with the old libdrizzle as a derived implementation class. The next step was to add in a new plugin hook for Protocol, and move the old libdrizzle implementation into a plugin. Since the server is the only thing using the old libdrizzle, all of that code also went into the plugin (note: this means the final death of old libdrizzle!). So now each new connection grabs a Protocol object from the default registered Protocol plugin and Session will

Good news! If you want to learn more about Gearman, you’ll have plenty of opportunities in the coming months:

• Percona Performance Conference 2009: The Return of Gearman
• MySQL Conference & Expo 2009: Map/Reduce and Queues for MySQL using Gearman
• MySQL Conference & Expo 2009: Gearman at the DotOrg pavillion, come visit our booth!
• OSCON 2009: Gearman: Bringing the Power of Map/Reduce to Everyday Applications (session)
• OSCON 2009: Gearman: Build Your Own Distributed Platform in 3 Hours (tutorial)

I’m pleased to announce three Gearman releases today! They are:

• Gearman C Server and Library - This includes changes to support new part of the protocol, a new Gearman command line tool, and a few other bug fixes and cleanup. Note: This now installs ‘gearmand’ into your preferred ’sbin’ directory, so if you are upgrading be sure to remove the old one from ‘bin’
• Gearman PHP Extension - This is a brand new PHP extension that wraps the C library to provide PHP with a faster client and worker interface. See the README file and the exampes directory for instructions and how to use it.
• Gearman MySQL UDF - New version of the UDF to include updates

Reading “Test Center: Slacker databases break all the old rules” has finally inspired me to write this post, which I’ve been meaning to do for a while. The title of my post may be bold, but it is just one version of how things may go moving forward (and was meant to grab your attention). :)

First off, a few issues I have with the article mentioned above:

• “…tacking the two letters “db” onto a pile of code that breaks with the traditional relational model.” - Since when does a database need to be relational? Sure, relational databases are the most popular, but they are only a subset of all the possible database types (let’s not forget hierarchical or tuple stores).
• “None of them is right for everyone, and

This Sunday I’ll be flying down to the bay area for my final week of work at Concentric. The following week I’ll be starting as a full-time Drizzle/Gearman developer with Sun Microsystems. This was the position Brian Aker mentioned back in January.

Over nine years ago I was flying to the bay area for my job interview at Concentric. It’s been a great run and I’ve learned a tremendous amount there, writing everything from customized, multi-threaded HTTP and DNS servers to a clustered, hierarchical object database with a full query language. The environment at Concentric feels like a family, probably because a number of co-workers have been there even longer than me. It was a tough decision to make, and I wish them all the

Thanks to those of you who came to listen in on the webinar. I’ve posted the slides here if you want them. If you have any follow-up questions, check out #gearman on irc.freenode.net.

Living in Portland, OR, I was sad to see O’Reilly move OSCON to San Jose this year (don’t they have enough with MySQL, Velocity, …?). So what did the Portland locals do? Start their own:

Open Source Bridge (we have a lot of bridges here in Portland).

It looks like they just opened up their call for papers recently, quoting from their page: “Open Source Bridge is accepting proposals now through March 31st for our 2009 conference, which will take place June 17–19 at the Oregon Convention Center in Portland, OR.”

Tomorrow I’m going to be giving a free webinar on Gearman through MySQL (http://mysql.com/news-and-events/web-seminars/display-280.html). I’ll be describing the basics of Gearman and then look at a few common use cases. I’ll even be touching on some more advanced topics such as Map/Reduce and persistent queues.

Click here to sign up now! (http://mysql.com/news-and-events/web-seminars/display-280.html)

This new release of Gearman includes improved client library code, a few bug fixes, new benchmarking tools, and man pages for the API. I also changed the default port in this release to use our new IANA assigned port, 4730 (was previously 7003). This means if you try to mix existing APIs or job servers with this release, you’ll need to specify port options on one side to make sure they match up. You can find the new tarballs on the Gearman Download page or on Launchpad.

I’ve also released a new version of the Gearman MySQL UDFs due to some of the C library changes. You can find the updated tarballs on the Gearman Download page or on

Over a month ago I applied for a real port for Gearman (before the default port was conflicting with AFS). After some back and forth with IANA:

We have assigned the following registered port number to 'gearman'
with you as the point of contact:

gearman         4730/tcp   Gearman Job Queue System
gearman         4730/udp   Gearman Job Queue System

See: http://www.iana.org/assignments/port-numbers

I’ll be updating this in the C version for the next release (which will trickle down into the MySQL UDFs). I’m also in the process of notifying other API and server maintainers to get their ports updated. So, if you use Gearman, watch out for possible port mismatches while the various packages transition!

This release includes a rework of the Gearman worker library internals, along with some improvements and bug fixes to the server, gearmand. You can find it off the wiki download page (http://www.gearmanproject.org/) or directly from Launchpad.

Remember to check out the various APIs, including the new MySQL UDFs released last week!

Now that the core of Gearman is rewritten in C, I’ve started working on a new module to provide persistent queues and queue replication. The current Gearman implementation is fine when the jobs can be unreliable, but there are many cases when we want to guarantee that jobs get run.

Because Gearman is such a flexible system, there are a couple of places where we can plug in a persistent queue and/or replication. Now, in most cases, these features are only going to be useful for background jobs since failed foreground jobs can be detected and restarted by a client. Many applications using Gearman want to just throw a job in the queue and go back to doing something else, not caring when it gets done. Enter persistent queues. It may also be the case that a job should be run multiple times, mot likely on a separate set of servers for backup and geographic redundancy.

Yesterday we announced the new release of the Gearman server, library, and UDFs, and now I’d like to provide a more useful example. This example assumes you’ve taken the steps to install the C server and MySQL UDFs as explained in that post, and now we’ll be using those pieces and the Perl API to show how to do URL processing. This could be used by a number of applications - anything where you have a URL, need to fetch it, possibly do some processing, and then store it somewhere (RSS feed cache, pulling links out of a HTML page, image conversion, …). In this example, MySQL is being used as the repository and “trigger point”, helping track the requests and results. Since we’ll be using Perl for our worker code, the first thing to do is install the Perl API from CPAN:

> perl -MCPAN -eshell
cpan shell -- CPAN exploration and

We’re happy to announce the first version of the Gearman rewrite in C, along with some interesting new MySQL UDFs based on the C library. Check out the Gearman wiki for an overview of what this is, download details, and API documentation.

For the anxious, here is a quick how-to:

Download: http://launchpad.net/gearmand/trunk/0.1/+download/gearmand-0.1.tar.gz
tar xzf gearmand-0.1.tar.gz
cd gearmand-0.1/
./configure
make
make install

You should now have the job server (gearmand) installed in /usr/local/bin, along with headers in libraries in /usr/local/lib & include. You can now see it work by running some simple clients and workers in the examples/ directory:

gearmand &      (this is assuming /usr/local/bin is in your path)
cd examples
./reverse_worker &
./reverse_client "Hello, Gearman!"

If everything went well,

The gearmand job server written in Perl (current production server from Danga):

mysql> SELECT length(gman_do("reverse", repeat('x',10000000))) AS test;
+----------+
| test     |
+----------+
| 10000000 |
+----------+
1 row in set (49.08 sec)

The new gearmand job server written in C:

mysql> SELECT length(gman_do("reverse", repeat('x',10000000))) AS test;
+----------+
| test     |
+----------+
| 10000000 |
+----------+
1 row in set (0.30 sec)

Mmm, efficiency. Oh, and are those some new MySQL UDFs? Much more coming soon…

Like a few others I’ve seen this week, I had two proposals for the MySQL Conference & Expo 2009 accepted. I’m very excited for both topics (for different reasons), and will be blogging about each of them in more detail soon. They are:

• libdrizzle: A new client library for Drizzle and MySQL - This will be a detailed introduction of the new library, covering important topics like client-side buffering, non-blocking I/O, and concurrent queries. If time permits I’ll also go over features in the new Drizzle protocol.
• Map/Reduce and Queues for MySQL using Gearman - I’ll be co-presenting this one with Brian Aker, and we will show how the work we’ve been doing with Gearman can be tied into MySQL (and Drizzle).

I spent this past week down in San Jose, CA at my employer’s office for team meetings and to officially kick-off my next big project. The design and architecture was very well received, and I drummed up some excitement with Gearman and working with the OSS community in general (which we’ve not done too much of in the past). We’ll be developing it entirely on Launchpad under GPLv2, and I’ll be writing a number of blog posts covering each component in detail. Why would anyone else find this interesting? It covers many topics of how to write a high-performance application in the cloud. Specific topics will include Gearman, persistent Gearman queues, eventual consistency data models (and related schemas), lightweight Map/Reduce for real-time applications, and how to combine all this with MySQL

Last weekend I attended the OpenSQL Camp in Charlottesville, VA. There was a great turnout, and Baron did an excellent job organizing it! I saw a few folks I met at OSCON over the summer, along with meeting many new people. What a great group - intelligent, fun, and know how to get things done. I had some great conversations, especially with Brian, Stewart, Arjen, Patrick, Mark, and Jay. The food was great too, I was a bit worried about finding vegan food there. Oh, and there was the wine bar, and my new found love for dessert wine.

After an amazing summer of getting married, graduating college, adopting a new dog, and taking a much needed extended vacation, I’m now back on track with where I left off. I’ve been making good progress on the asynchronous MySQL library I talked about in this post in the form of a new drizzle client library. This library is also compatible with MySQL since they share the same protocol, and if drizzle changes in the future I plan on supporting current and new MySQL protocols as well. All of the connection and I/O overhead is mostly done, and I’m just working through the protocol bits now. I’m hoping to have something ready to show and talk about for the OpenSQL camp.

In other news, I’ve recently started working with the

Keynotes

The keynote was kick started by Marten Mickos.  If you've never met Marten, he is, on a personal note, one of the greatest CEOs I've ever met.  The keynotes were especially interesting for me because it was the first time I've had the opportunity to listen to Jonathan Schwartz, the CEO of Sun Microsystems.  Jonathan seems like a great guy who gives the impression he "gets it".

The last keynote was by Werner Vogels of Amazon.  His talk covered Amazon's growth and the new services they offer including EC2.  He announced that EC2 now supports persistent storage, which is a huge improvement, but doesn't quite solve all of the problems.

Testing PHP/MySQL Applications with PHPUnit/DbUnit

I've never been big into testing, but I'm trying to