Even in times of a growing market of specialized NoSQL databases, the relevance of traditional RDBMS doesn't decline. Especially when it comes to the calculation of aggregates based on complex data sets that can not be processed as a batch like Map&Reduce. MySQL is already bringing in a handful of aggregate functions that can be useful for a statistical analysis. The best known of this type are certainly:

Read the rest »

Since I use MySQL for the statistical analysis on a project, I wanted to optimize the database queries and learned a lot about stuff like number theory, set theory and partial sums. I took my MySQL UDF, I've published two years ago, for this purpose and added new functions for a deeper statistical analysis. The project is around for a while, so it's time to share things with the public to start a discussion of how things could be further optimized. The source and a small documentation can be found on Github:

Read the rest »

Use DNS directly from your database

mysql> SELECT lookup('localhost');
+---------------------+
| lookup('localhost') |
+---------------------+
| 127.0.0.1           |
+---------------------+
1 row in set (0.00 sec)

mysql> SELECT reverse_lookup('127.0.0.1');
+-----------------------------+
| reverse_lookup('127.0.0.1') |
+-----------------------------+
| localhost                   |
+-----------------------------+
1 row in set (0.00 sec)

This is how you install these functions.

Build udf_example.so which is in your mysql source. (make udf_example.so)

Copy the udf_example.so file from your build directory to your plugin_dir.

Create the lookup and reverse_lookup

  [Read more...]

I’m pleased to announce the release of Memcached Functions for MySQL, version 1.1. It’s been a while since the 1.0 release, and I’ll use writing a book and having a baby as an excuse, but nevertheless, this release has several changes and new features:

• Fixed all the calls to libmemcached that were broken when libmemcached changed. This release works with Libmemcached 0.44
• Added my own string functions as the libmemcached string functions I relied on before were made local/private from the shared library
• Added memc_get_cas() and memc_get_cas_by_key() functions. You can now obtain the CAS value of an item! For instance:

mysql> select memc_servers_set('localhost:11211');
+-------------------------------------+
| memc_servers_set('localhost:11211') |
+-------------------------------------+
|

  [Read more...]

Note: It was my first attempt using the MySQL UDF Api back then. The library is still maintained and got some new features. Check out the new MySQL Infusion UDF.

Read the rest »

MySQL 5.1 Plugin Development,
by Sergei Golubchik and Andrew Hutchings.
Packt Publishing, 2010.
Executive summary: Highly recommended. If you want to develop MySQL extensions, buy this book. It's a must, written by two expert professionals who probably know more than anyone else on this matter. The book is full of practical examples explained with the theoretical information necessary to make it stick.

This book fills a gap in the world of MySQL documentation. Although the MySQL docs are extensive and thorough, to the point that sometimes you wished that the




  [Read more...]

MySQL 5.1 Plugin Development

MySQL 5.1 Plugin Development, by Sergei Golubchik and Andrew Hutchings, Packt 2010. About 250 pages. (Here’s a link to the publisher’s site.)

This book is well worth reading for anyone interested in MySQL internals. I learned a lot from it. It is well-written and understandable. I cannot say that I’m planning to write storage engines or more advanced plugins, but I have a great many ideas how to improve MySQL, and I now understand more clearly which of those are suitable to

  [Read more...]

I gave two presentations about Gearman at the Linux.conf.au. As part of the preparation for these talks, I created several sample applications. One of them, about remote replication administration, I will cover in a separate post. The most amazing one, which I cover here, is a quick and painless solution for multiple level crosstabs in MySQL.

Some background is needed. Crosstabs (also called data cubes or pivot tables, have been one of my favorite hacks for long time. In 2001 I wrote an article about a simple




  [Read more...]

If you haven't yet heard about Gearman, it's time to have a look at it. Its distributed client server architecture are a perfect match for today's cloud oriented applications.
When talking about Gearman, much stress is often given on its scalability features, such as map/reduce and distributed loads. But Gearman has also a distinctive advantage in the feature department, because of its design that I like to describe as cooperation of the fittest.
While a traditional application is written entirely in a given language, or it must find ways of integrating multiple languages parts, Gearman encourages easy cooperation between parts written in different



  [Read more...]

What a great day! I see that the drizzle team has merged in my (and Padraig O'Sullivan's) memcached Functions for Drizzle (UDFs). I'm really glad to have this in drizzle now as it adds a means of interacting with memcached from within Drizzle. I have most functions from the MySQL branch implemented now as well. I'm extremely grateful to Padraig O'Sullivan for getting this project off the ground. I was a bit stuck with the new API and C++ when I first attempted these and he designed the class setup and had the major functions working which I then picked up and added more functions as well as tests. I was also glad to have the drizzle team add in my sleep() UDF which allowed me to test expirations in these memcached functions.

These functions have some similar, but very little code from the memcached Functions for MySQL. The new UDF API is completely different than MySQL's UDF API. You have to

  [Read more...]

I decided to thee other day to investigate using Tokyo Tyrant because I was interested in the fact it has a memcached protocol and I wanted to get a feel for how it works with my memcached functions for MySQL (UDF). Matt Ingenthron came up with a good term recently: Mem-capable, which Tokyo Tyrant is. I find any key/value storage to be of great interest, particularly those that you simply change the port your memcached client is connecting and simply use it the same way you would memcached.

So, just what is Tokyo Tyrant?

Tokyo Tyrant is a database server, written by Mikio Hirabayashi, for Tokyo Cabinet. It provides for concurrent remote connections to Tokyo Cabinet . It provides its own binary protocol as well as a memcached and HTTP compatibility protocols. It also has C, Perl, PHP, Java, Erlang, Python and Ruby bindings. Other features Tokyo Tyrant offers:

* Hot backup and





  [Read more...]

Working at Northscale has been a lot of great fun lately! I have finally figured out how to get puppet to set up a stock Amazon instance with everything we need and have been impressed with how you can automate system setup with puppet. I remember when I worked at Grazr how much of a hassle it was for us to have to rebuild systems. Something like Puppet would have been a godsend.

Today I though I would post about how cool moxi is. If you don't know, moxi is a memcached proxy (http://labs.northscale.com/moxi/) which allows you to move any complexity of having to set up the list of memcached servers you are using. Also, moxi has some great features such as:

* De-duplication of requests for popular keys
* A front cache, L1 cache to avoid network hops
* Fire and forget SET (Like an async SET) this means "set a value, but don't wait to





  [Read more...]

I'm pleased to announce the release of the Memcached Functions for MySQL, version 1.0.

This release contains several changes, per the ChangeLog:

1.0 Thursday, July 30, 2009 12:00:00 EST 2009
* Fixed issue of setting NULLs with user-defined variables (Thanks to
Jean-Jacques Moortgat at aol dot com !)
* Fixed issue of obtaining a NULL value FROM memcached
* All set functions now return 0 (failure) or 1 (success)
* Other cleanups
* More tests

Importantly, there was an issue that I blogged about several days ago where in the UDF API, if you pass a user-defined variable that is set to NULL to memc_set(), the length of the argument is 8192 even though the value of the argument itself is NULL, which caused much unhappiness in MySQL (crash). That is fixed by setting the length to 0 if the argument itself is NULL. Also fixed is obtaining the NULL value











  [Read more...]

I'm glad to announce the release of Memcached Functions for MySQL version 0.9. This version contains a couple bug fixes:

* Switched to using calloc() instead of malloc(), check for return value
* Changed memc_servers_set() to not create a new connection when one already exists, free first

Also added:

* memc_server_version()

mysql> select memc_server_version('127.0.0.1:11211')\G
*************************** 1. row ***************************
memc_server_version('127.0.0.1:11211'): Server: 127.0.0.1 (11211) version 1.2.6

I'm glad to get development moving along again. The book really took a ton of time, so now I can get this project moving along. Version 0.10 I'll be adding even more functionality. I'll get some ideas at the user's conference next week as well. One thing I need to get working properly is the ability to set a value in a












  [Read more...]

I'm pleased to announce the release of the Memcached Functions for MySQL, version 0.8.

This version includes a bunch of fixes for behavior setting and retrieval. You now have:

Ability to fetch a behavior:

memc_server_behavior_get();

All behaviors can be set now successfully with memc_server_behavior_set()

Boolean:

select memc_behavior_set('MEMCACHED_BEHAVIOR_BUFFER_REQUESTS', '1');

Name of a value (string):

select memc_behavior_set('MEMCACHED_BEHAVIOR_HASH','MEMCACHED_HASH_MD5' );

Numeric:

select memc_behavior_set('MEMCACHED_BEHAVIOR_SOCKET_SEND_SIZE', 60000);

Also, you can now list the distribution types to use when calling memc_server_behavior_set('MEMCACHED_BEHAVIOR_DISTRIBUTION', ''), which produces a list of all the distrubution types:

select























  [Read more...]

Thanks to Bernardo Damele, the lib_mysqludf_sys package hosted by the Repository for MySQL UDFs can now boast a new sys_eval() function. You can read the details here on his blog. Bernardo also created a similar function for PostgreSQL.

Thank you,

  [Read more...]

I've taken some time out of writing my book to get some coding down, and I'm pleased to announce the release of the Memcached Functions for MySQL, version 0.6. This release includes:

* Complete rewrite of error handling
* Build configuration fixes/improvements from Trond Nordby (Thanks!)
* Fixed memc_server_count
* More tests

You can find the release information-- repository and source at:

http://tangent.org/586/Memcached_Functions_for_MySQL.html

As well as

http://patg.net/downloads/memcached_functions_mysql-0.6.tar.gz

Have fun! More to come...

In almost all application development situations, one needs to log debug information now and then. In almost all production systems, one needs to log serious error events somewhere too.

So, what can you do? Create a log table? Sprinkle your code with SELECT 'Debug: ...' statements?

At the UDF Repository for MySQL, we now have a solution to log messages to the MySQL error log: a user-defined function called log_error().

Currently it is all very crude: the log_error function takes one argument and writes it to the mysql_error





  [Read more...]

I'm pleased to announce the release of Memcached UDFs for MySQL. This release contains the ability to now pass expiration values. For instance, you can:

select memc_set('keyabc', 'somevalue')

or

select memc_set('keyabc', 'I die in 10 seconds', 10);

The functions that now have this ability are:

memc_set()
memc_set_by_key()
memc_append()
memc_append_by_key()
memc_prepend()
memc_prepend_by_key()
memc_add()
memc_add_by_key()
memc_replace()
memc_replace_by_key()

You can download the latest archive from http://patg.net/downloads/memcached_functions_mysql-0.5.tar.gz

Also, mercurial repository at http://hg.tangent.org/memcached_functions_mysql/
























  [Read more...]

Liveblogging: Extending MySQL by Brian “Krow” Aker

Brian wins the award for “most frequent great quotes during a talk”.

Before MySQL 5.1 a UDF was the only way to extend MySQL.

All you need in a UDF is: init() execute() deinit()

my_bool id3_parse_init(UDF_INIT *initid UDF_ARGS *args, char *message)

UDF_ARGS tell you about incoming args
char *message is the output that might return
args->arg_count is the # of args

WARNING: use STRICT mode in MySQL, otherwise there are tons of silent failures.

“When you work on databases you start to put everything in databases. Tip, don’t put a DVD into a database, because really long BLOBs aren’t actually supported….”

In MySQL 5.1, you can now install plugins (example is

  [Read more...]

The MySQL User's conference will be held in less than a month from now!!!

This year there is quite a good number of sessions on adding your own functions and procedures, such as:

• Advanced Stored Procedures


• A Tour of External Language Stored Procedures for MySQL


• Code Generators for MySQL Plugins and User Defined Functions (UDFs)


• Extending

  [Read more...]

Recently, I came upon Baron Swartz's (xaprb's) great now_usec UDF:
http://www.xaprb.com/blog/2007/10/30/how-i-built-the-now_usec-udf-for-mysql/

I use an Intel Macbook Pro for my primary development environment. GCC on OS X has some interesting quirks. Usually to compile a UDF on GNU/Linux I use the following command line:

gcc -fPIC -Wall -I/usr/include/mysql -shared -o udf_now_usec.so udf_now_usec.cc

When I tried to execute the above on my OS X box, I got some errors:

$ gcc -fPIC -Wall -I../include -shared -o udf_now_usec.so udf_now_usec.cc
i686-apple-darwin8-gcc-4.0.1: unrecognized option '-shared'
/usr/bin/ld: Undefined symbols:
_main
___gxx_personality_v0</b>
collect2: ld returned 1 exit status

The first error

  [Read more...]

This release contains bug fixes and new features. Click through to the full article for the details. I'll also write more about the changes in a separate article.

My posts lately have been mostly progress reports and release notices. That's because we're in the home stretch on the book, and I don't have much spare time. However, a lot has also been changing with Maatkit, and I wanted to take some time to write about it properly.