I'm pleased to announce the release of Spider storage engine version 2.27(beta) and Vertical Partitioning storage engine version 0.16(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
Vertical Partitioning is a Storage Engine for vertical partitioning for a table.
http://launchpad.net/vpformysql

Please use the following for downloading binary file.
http://spiderformysql.com/download_spider.html

The main changes in this version are following.
Q4M is bundled for Linux 64bit.
  Bundled Q4M supports replication, but please use it carefully.

Spider
- Support R-Tree index.
- Support direct updating for SQL














  [Read more...]

Today at MySQL Conference & Expro 2009, I did a presentation introducing / explaining Q4M. Thank you to people who came to listen.

The presentation slides I used can be found on slideshare (slideshare).

At MySQL Conference & Expo 2009, I will be giving a presentation on Q4M tomorrow (Apr. 22) from 11:55am. If you are interested in using a simple, fast message queue as part of your system, please come to the session.

Details: Using Q4M: A Message Queue Storage Engine for MySQL

On the next day (Apr. 23), I will be doing another presentation at Percona Performance Conference, held at the same location. In the presentation, I will describe the techniques (mainly MySQL UDFs) used to squeeze maximum performance out from MySQL used in Pathtraq, one of the largest web access stats service in Japan.

Details:

  [Read more...]

Q4M 0.8.5 is now downloadable from q4m.31tools.com. Prebuilt binaries for MySQL 5.1.33 running on linux (i386 or x86_64) and Mac OS X 10.5 (x86) are available as well.

There are no bugfixes in this release. The only change from version 0.8.4 is bundle of boost header files necessary for building Q4M. The build process no more requires separate installation of the Boost C++ libraries.

Today I have uploaded Q4M 0.8.4 to q4m.31tools.com. Prebuilt binaries for MySQL 5.1.31 running on linux (i386 or x86_64) and Mac OS X (x86) are available as well.

The release fixes crash on linux (i386) systems when a table becomes larger than 2GB. There are no changes for other platforms.

I have uploaded prebuilt binaries of Q4M for MySQL 5.1.28-rc to http://q4m.31tools.com/dist/. For installation instructions, please refer to http://q4m.31tools.com/install.php.

Thanks to Akinori MUSHA, Q4M has become part of the FreeBSD Ports Collection.

If you are using FreeBSD, Q4M can be installed by following the steps below.

# cd /usr/ports/databases/mysql-q4m
# make install
# echo 'mysql_enable="YES"' >> /etc/rc.conf
# /usr/local/etc/rc.d/mysql-server start
# mysql -u root -f mysql 

Running either cvsup or portsnap might be necessary to update the installed ports collection to the newest state.  Since the port depends on mysql51-server, you should make deinstall if an older version of mysql is already installed via the ports collection.  If you want to test the installation, type:

# chmod 755 work/q4m-0.8.3/support-files/q4m-forward
# make test

  [Read more...]

Q4M (Queue for MySQL) Version 0.8.3 has been released with following Changes.

• fix race condition error that might lead to deadlock on shutdown
• support for FreeBSD

With support for FreeBSD added, prebuilt binaries are provided for the platforms below.

• linux (i386)
• linux (x86_64)
• freebsd-6 (i386) (should work on FreeBSD 7 as well with compat6x package installed)
• Mac OS X 10.4 (x86)

Last week, Mixi - Japan's largest social network service provider, launched an experimental microblogging service called Echo, and according to their delevopers' blog entry, they are using Q4M to level their write loads. Thank you to the developers of Mixi Echo for using Q4M, I hope Echo will go well and soon become a first-class service.

Meanwhile, several bugs were found in Q4M (that arise under rare cases or complex usage senarios), so I have just uploaded version 0.8, available from the Q4M homepage.

This release fixes the following bugs.

• block div-by-zero exception on conditional subcription
• do not crash when a nonexistent

  [Read more...]

Version 0.7 of Q4M (a pluggable message queue storage engine for MySQL) has been released with following changes.

Faster SELECT COUNT(*)

Q4M now caches the number of rows within a table. It is now possible to heavily issue SELECT COUNT(*) queries to monitor queue usage.

Dropped binlog capability flags

Q4M tables were incorrectly marked as binlog-capable in previous releases.

Added examples/crawler

Q4M now includes an example web spider implementation. According to a test using preliminary version (detail in Japanese), the implementation was about two times faster than a crawler based on

  [Read more...]

Today I have uploaded Q4M (a Queue for MySQL) 0.6, which is basically a performance-improvement from previous releases. Instead of using pread's and a small user-level cache, Q4M (in default configuration) now uses mmap for reads with a reader/writer lock to improve concurrency.

I also noticed that it would be possible to consume a queued row in one SQL statement.

SELECT * FROM queue_table WHERE queue_wait('queue_table');

This statement actually does the same thing as,

if (SELECT queue_wait('queue_table') == 1) {
  SELECT * FROM queue_table;
}

But since the former style requires only one SQL statement (compared to two statements of the second one), it has much less overhead.

And combining these optimizations together, consumption speed of Q4M has nearly doubled from

  [Read more...]

Today I had a chance to explain Q4M in detail, and here are the slides I used.

It covers from what (generally) a message queue is, the internals of Q4M, how it should be used as a pluggable storage engine of MySQL, to a couple of usage senarios. I hope you will enjoy reading it.

| View | Upload your own

I always use to blog my temporary ideas on one of my Japanese blog (id:kazuhooku's memos). When I wrote my thoughts on how to further optimize Q4M, Nishida-san asked me "how fast is the raw performance without client overhead?" Although it seems a bit diffcult to answer directly, it is easy to measure the performance of MySQL core and the storage engine interface, and by deducting the overhead, the raw performance of I/O operations in Q4M can be estimated. All the benchmarks were taken on linux 2.6.18 running on two Opteron 2218s.

So at first, I measured the raw performance of MySQL core on my testbed using mysqlslap, which was 115k queries per second.

$ perl -e 'print

  [Read more...]

Today I have released Q4M version 0.3. Changes from the previous are as follows.

Support for message relaying

Messages submitted to Q4M tables can be forwarded to another Q4M table on a different MySQL server. The transfer is trustable, there would be no message loss or duplications even on a server or a network failure. It is also possible to build a message broker above the API used for relaying.

Prioritized subscription to multiple tables

Clients can now subscribe to more than one table at once. The subscription priority can be specified, thus it is able to create a priority queue using multiple tables.

For more information, please refer to the Q4M homepage.

For several years I have been wondering if there was a RDBMS that serves as a message queue as well1. And in the end of last year, noticing that MySQL 5.1 with support for pluggable storage engine was entering its RC stage, and that (from little googling) nobody was creating such storage engine, I dicided to write my own.

Q4M (Queue for MySQL) is a message queue that works as a pluggable storage engine of MySQL 5.1, designed to be robust, fast, flexible. The development started in late December of 2007, and although it is very primitive, operates quite swiftly.
q4m.31tools.com

Yes, it is primitive, and might have stability problems (I do not recommend using Q4M in a production environment), but if you are interested, please have a try.

1: At least Oracle seems to, but I never had a chance to use it.