I’m happy to announce the first release of the new libdrizzle. This is a complete rewrite of the protocol and client interface for the Drizzle project and provides new functionality that the old library based on MySQL code did not offer. Oh, and this new library supports both Drizzle and MySQL protocols, so you can use it with MySQL servers as well!

So, what’s different?

• BSD License.
• Complete non-blocking I/O support.
• Concurrent query interface, letting you run multiple queries at the same time on any number of servers.
• Improved buffering support. Instead of just result and row buffering, you can also buffer per field, or not at all (reading fields in chunks). This can be very useful for streaming large blobs through your client.
• Complete server …

This new release of Gearman includes improved client library code, a few bug fixes, new benchmarking tools, and man pages for the API. I also changed the default port in this release to use our new IANA assigned port, 4730 (was previously 7003). This means if you try to mix existing APIs or job servers with this release, you’ll need to specify port options on one side to make sure they match up. You can find the new tarballs on the Gearman Download page or on Launchpad.

I’ve also released a new version of the Gearman MySQL UDFs due to some of the C library changes. You can find the updated tarballs on the Gearman Download page or on …

Over a month ago I applied for a real port for Gearman (before the default port was conflicting with AFS). After some back and forth with IANA:

We have assigned the following registered port number to 'gearman'
with you as the point of contact:

gearman         4730/tcp   Gearman Job Queue System
gearman         4730/udp   Gearman Job Queue System

See: http://www.iana.org/assignments/port-numbers

I’ll be updating this in the C version for the next release (which will trickle down into the MySQL UDFs). I’m also in the process of notifying other API and server maintainers to get their ports updated. So, if you use Gearman, watch out for possible port mismatches while the various packages transition!

This release includes a rework of the Gearman worker library internals, along with some improvements and bug fixes to the server, gearmand. You can find it off the wiki download page (http://www.gearmanproject.org/) or directly from Launchpad.

Remember to check out the various APIs, including the new MySQL UDFs released last week!

Now that the core of Gearman is rewritten in C, I’ve started working on a new module to provide persistent queues and queue replication. The current Gearman implementation is fine when the jobs can be unreliable, but there are many cases when we want to guarantee that jobs get run.

Because Gearman is such a flexible system, there are a couple of places where we can plug in a persistent queue and/or replication. Now, in most cases, these features are only going to be useful for background jobs since failed foreground jobs can be detected and restarted by a client. Many applications using Gearman want to just throw a job in the queue and go back to doing something else, not caring when it gets done. Enter persistent queues. It may also be the case that a job should be run multiple times, mot likely on a separate set of servers for backup and geographic redundancy. I’m currently …

Yesterday we announced the new release of the Gearman server, library, and UDFs, and now I’d like to provide a more useful example. This example assumes you’ve taken the steps to install the C server and MySQL UDFs as explained in that post, and now we’ll be using those pieces and the Perl API to show how to do URL processing. This could be used by a number of applications - anything where you have a URL, need to fetch it, possibly do some processing, and then store it somewhere (RSS feed cache, pulling links out of a HTML page, image conversion, …). In this example, MySQL is being used as the repository and “trigger point”, helping track the requests and results. Since we’ll be using Perl for our worker code, the first thing to do is install the Perl API from CPAN:

> perl -MCPAN -eshell
cpan shell -- CPAN exploration and modules installation (v1.9205)
cpan[1]> …

We’re happy to announce the first version of the Gearman rewrite in C, along with some interesting new MySQL UDFs based on the C library. Check out the Gearman wiki for an overview of what this is, download details, and API documentation.

For the anxious, here is a quick how-to:

Download: http://launchpad.net/gearmand/trunk/0.1/+download/gearmand-0.1.tar.gz
tar xzf gearmand-0.1.tar.gz
cd gearmand-0.1/
./configure
make
make install

You should now have the job server (gearmand) installed in /usr/local/bin, along with headers in libraries in /usr/local/lib & include. You can now see it work by running some simple clients and workers in the examples/ directory:

gearmand &      (this is assuming /usr/local/bin is in your path)
cd examples
./reverse_worker &
./reverse_client "Hello, Gearman!"

If everything went well, your terminal should …

The gearmand job server written in Perl (current production server from Danga):

mysql> SELECT length(gman_do("reverse", repeat('x',10000000))) AS test;
+----------+
| test     |
+----------+
| 10000000 |
+----------+
1 row in set (49.08 sec)

The new gearmand job server written in C:

mysql> SELECT length(gman_do("reverse", repeat('x',10000000))) AS test;
+----------+
| test     |
+----------+
| 10000000 |
+----------+
1 row in set (0.30 sec)

Mmm, efficiency. Oh, and are those some new MySQL UDFs? Much more coming soon…

Like a few others I’ve seen this week, I had two proposals for the MySQL Conference & Expo 2009 accepted. I’m very excited for both topics (for different reasons), and will be blogging about each of them in more detail soon. They are:

• libdrizzle: A new client library for Drizzle and MySQL - This will be a detailed introduction of the new library, covering important topics like client-side buffering, non-blocking I/O, and concurrent queries. If time permits I’ll also go over features in the new Drizzle protocol.
• Map/Reduce and Queues for MySQL using Gearman - I’ll be co-presenting this one with Brian Aker, and we will show how the work we’ve been doing with Gearman can be tied into MySQL (and Drizzle).

I spent this past week down in San Jose, CA at my employer’s office for team meetings and to officially kick-off my next big project. The design and architecture was very well received, and I drummed up some excitement with Gearman and working with the OSS community in general (which we’ve not done too much of in the past). We’ll be developing it entirely on Launchpad under GPLv2, and I’ll be writing a number of blog posts covering each component in detail. Why would anyone else find this interesting? It covers many topics of how to write a high-performance application in the cloud. Specific topics will include Gearman, persistent Gearman queues, eventual consistency data models (and related schemas), lightweight Map/Reduce for real-time applications, and how to combine all this with …