In preparation for an upcoming demo, we decided to use Sphinx for the full text search. We want to do this with Drizzle but hit an obvious problem - Sphinx does not know how to speak to Drizzle to grab documents for indexing. At least that was the case until last night when I created a patch to do so using the new libdrizzle library.

You can download the patch here: sphinx-0.9.8.1-drizzle.patch

I’m releasing this patch BSD, so do with it what you want. I’m hoping Andrew (the Sphinx author) will include this in the main package in an upcoming release. :)

So for now, to get things working, we first need to patch the sphinx-0.9.8.1 source and build/install:

I’ve decided to start a series of blog entries on not-so-obvious findings that I’ve found while working on my new project. By archiving the findings, I’m hoping that I can help those that are looking into developing a storage engine for the MySQL family in the future.

Accumulating these mini-knowledge would also be useful for me since I can refer back to it when I forget something. Also, once I write enough entries I’m planning on summarizing them and making it available on the Drizzle Wiki. If MySQL is interested in updating the engine documentation, I would be more than happy to help there too.

So to begin with, I’ll describe something trivial that I stumbled across while trying to catch an error on duplicate primary key insertion to the data table.

I read Jeremy’s post last week on jobs that matter and was also reminded of Tim O’Reilly’s post about the same topic from earlier this year (which is a great read by the way). This got me thinking, and I realized it has been about one year since I made the commitment to myself to get more involved with open source. I had been a passive member for over ten years, but I decided it was time for a change. After looking at various GNU projects, the Linux kernel, MySQL, and a few others, I decided to focus on the MySQL community. After attending OSCON I was even more inspired with the announcement of Drizzle, along with learning about other projects such as …

In an email disscussion for pre-allocating binlogs for MySQL (something we’ll likely have to do for Drizzle and replication), Yoshinori brought up the excellent point of that in some situations you don’t want to be doing zero-fill as getting up and running quickly is the most important thing.

So what does Linux do? Does it zero-fill, or behave sensibly and pre-allocate quickly?

Let’s look at hte kernel:

Inside the fallocate implementation (fs/open.c):

if (inode->i_op->fallocate)
ret = inode->i_op->fallocate(inode, mode, offset, len);
else
ret = -EOPNOTSUPP;

and for ext4:
/*
* currently supporting (pre)allocate mode for extent-based
* files _only_
*/
if (!(EXT4_I(inode)->i_flags & EXT4_EXTENTS_FL))
return -EOPNOTSUPP;

XFS has always done fast pre-allocate, so it’s not a problem there and the only other …

Found in the libstdc++ docs:

Extremely weird solutions. If you have access to the compiler and linker at runtime, you can do something insane, like figuring out just how many bits you need, then writing a temporary source code file. That file contains an instantiation of bitset for the required number of bits, inside some wrapper functions with unchanging signatures. Have your program then call the compiler on that file using Position Independent Code, then open the newly-created object file and load those wrapper functions. You’ll have an instantiation of bitset for the exact N that you need at the time. Don’t forget to delete the temporary files. (Yes, this can be, and has been, done.)

Oh yeah – feel the love.

Brought to you by the stl-is-often-worse-for-you-than-meth dept.

As I was driving with a colleague in California a couple of weeks ago during the conference, the topic of conversation turned to the notion that Percona and the rest of the MySQL community really need the presence of a central entity that provides a recognized home for the MySQL server. The conversation went something like “I was talking to so-and-so, and he said, you know, you guys really need Sun/MySQL, because without the mother ship, things will fall apart and your own business will fail.”

I happen to believe this is FUD, and that the reverse might actually be true. (This is one reason why we’re competing head-on with MySQL.) Having a “mother ship” is in the long run, a very complex scenario to fully understand. There are all sorts of causes and effects that play out in subtle ways. I honestly doubt anyone can really …

While at the MySQL Conference last week, I stopped at the booth for HeidiSQL, and spoke with Bill Culp, who is the developer of jHeidi, about Drizzle.

A few days ago, he emailed me, saying that he had spent the time, and now jHeidi supports Drizzle.

So now we have a multiplatform GUI for monitoring, browsing, and querying a Drizzle database server.

Last week at the MySQL Conference & Expo I made a release right before my first Gearman talk. I didn’t get a chance to blog about it, so here it is now. This is a fairly big release since it includes a major refactoring of the job server to now be threaded. Doing some simple tests on it with a 16-core Intel machine:

The Y axis is total jobs/second, and the X axis is number of clients/workers (8×8 means 8 clients and 8 workers). The clients and workers are the ‘blobslap’ utilities included in the source, and they are shoving as many jobs/second through the server as they can. Job size was random between 0 and 1024 bytes. The job server peaked out at about 43,000 jobs/second, most likely due to the machine being so busy (running job server, workers, and clients). The job server uses all non-blocking I/O with a …

In the Apache world, you might be familiar with tweaking your config file(s) and then running

$ apachectl configtest

to see if the config parses. We've been discussing this on the drizzle mailing list and talking in general about configuration handling and management. Well, it turns out that you can fake it in MySQL and Drizzle too.

If you have a new configuration in /tmp/new.cnf, try this:

$ mysqld --defaults-file=/tmp/new.cnf --verbose --help

And it'll run mysqld (or drizzled), parse the config, report any problems, print help, and exit without initializing storage engines or trying to grab a port.

Neat trick!

Thanks to Baron Schwartz, Arjen Lentz, and Sheeri …

On the Drizzle-discuss mailing list, Baron just pointed out that MySQL has a means to validate its config file (/etc/my.cnf), similar to "apachectl configtest".

mysqld --help --verbose

Somehow, I never knew that...

Oh, and yes, this means I am following Drizzle. In fact, I'm trying to devote at least a few hours each day to reading the source and finding things to fix. In a weird way, it feels good to be writing C++ again!