In what I laughingly call “spare time” I started hacking on ha_file.cc, otherwise known as the FILE storage engine. My idea is relatively simple, I want to be able to store and access my photos from MySQL. I also want the storage to be relatively efficient and have the raw image files on disk, not tied up too much in any different format (my file system is pretty good at storing multi-megabyte files thank you very much) - it also doesn’t require any fancy things to re-use space when I delete things. I should also be able to (efficiently) directly serve the images out of a web server (satisfying the efficiency itch). You could also use something like DMF to migrate old rows off to tape.
So, I started some hacking and some designing and have a working design and a nearly basically working read/write implementation. I’ll share the code when it does, in fact, actually work (by “work” i mean reads and writes basic rows).
I’ve …

At the MySQL User's Conference in April, there were a number of new startups coming to market. One market that I think could be huge is for open source EAI (Enterprise Application Integration) software.  I worked in the EAI space for a number of years, helping take Active Software public in 1999 as VP of Marketing and then staying on with webMethods when they acquired Active.  The EAI market had huge growth during that time, fueled by the need for new internet systems to integrate with legacy applications.  The need still exists today, but the growth has leveled off as traditional EAI solutions have been too complex and expensive for most CIOs. 

I think there's a great opportunity for someone to come in and simplify EAI with a solution that works for 90% of the needs out of the box, with the ability for the most demanding customers to customize it to their own needs.  This is exactly the type of middleware that could …

My blog link at PlanetMySQL.org is still not updated so here is list of posts me and Vadim are written during last couple of weeks.

Group Commit and XA Some more benchmark results for MySQL 5.0 performance regression because of broken group commit.

Speedup your LAMP stack with LigHTTPD The article is about Optimization of Web Server part in LAMP stack in general and how you can use lighttpd to speed things up in particular.

How Web Services world affect LAMP Stack Article discusses some things which become different with Web Services application. Kind of followup to …

There are many questions how InnoDB allocates memory. I'll try to give some explanation about the memory allocation at startup.
Some important constants:
NBLOCKS=count of block in innodb_buffer_pool = innodb_buffer_pool_size / 16384
OS_THREADS= if (innodb_buffer_pool_size >= 1000Mb) = 50000
else if (innodb_buffer_pool_size >= 8Mb) = 10000
else = 1000 (it's true for *nixes, for Windows there is a bit another calculation for OS_THREADS)
So InnoDB uses:

• innodb_buffer_pool
• innodb_additional_mem_pool_size
• innodb_log_buffer_size
• adaptive index hash, size= innodb_buffer_pool / 64
• system dictionary hash, size = 6 * innodb_buffer_pool_size / 512
• memory for sync_array, which is used for syncronization primitives, size = OS_THREADS * 152
• memory for os_events, which are also …

Everybody knows that valgrind is great.

Well, I was observing a problem in some MySQL code, it looked like we were writing over some memory that we weren’t meant to be (as the structure hadn’t been initialised yet). But, seeing as this was memory that had been allocated off a MEM_ROOT (one of our memory allocators), valgrind wasn’t gonig to spit out anything.

This is because this bit of memory had already been allocated and subsequently “freed”, but then reallocated. The “free”ing overwrites the memory with garbage (which is what the MEM_ROOT code does) so that you should see crashes (and a pattern) when you do something bad.

The traditional way to troubleshoot this in to modify your memory allocator so that it just calls malloc() and free() all the time (and valgrind will trap them). We have some code in there to do that too. However, this requires …

My good friend Zak Greant spotted a news item at Techworld, where the European Commission appears to have changed its position on software patents.

Have a read, it's interesting and I hope that it really does indicate a new direction from the commission and not just one loose statement. If it is for real, that's absolutely great, but let's not be complacent and think "that's it". The pro-swpat lobbyists certainly won't give up, they have strong interests and stacks of cash driving them.
Of course it's not helpful to preach to the converted, but encouragement to continue on this path seems like a good idea. I'll happily give politicians kudos and encouragement when they do good stuff!

Also, if you haven't yet …

So what do I do on Memorial Day Weekend?

Well in theory I did not work.

Friday I went to go to see a SIFF, Seattle Internal Film Festival, movie in the evening. The fact that I can not remember what it was, and I think I gave it three stars means that I will not be recommending it to anyone. I have eight movies to see in three weeks (of which in the three weeks I will go to Boston, Korea, and probably California). Got home from the movie and started to port Everydevel to Apache 2.2 and MySQL 5.0. No one has bothered to do either and I am getting tired of continuing to set it up on old application stacks. The Apache part took a minor amount of reading just because of the changes in mod_perl2 from mod_perl. It was minor and mostly annoying stuff. I also setup Apache to just use pre-fork since I am personally not sure that I consider a threaded version of Apache with mod_perl to be safe. Watched Fire Fly while coding. …

Did you know that you can access tables on handler/engine level? I didn?t until I read the new training material for our new upcoming ?Performance Tuning? course. So, if you didn?t know the HANDLER command, book the course! There?s much more that you might now know yet.

Read http://dev.mysql.com/doc/refman/5.0/en/handler.html for more background.

root@localhost [world]> HANDLER City OPEN; Query OK, 0 rows affected (0.10 sec)

root@localhost [world]> HANDLER City READ FIRST; +----+-------+---------+----------+------------+ | Id | Name | Country | District | Population | +----+-------+---------+----------+------------+ | 1 | Kabul | AFG | Kabol | 1780000 | +----+-------+---------+----------+------------+ 1 …

We had discussion today which involved benchmarks of Join speed for MyISAM and Innodb storage engines for CPU bound workload, this is when data size is small enough to fit in memory and so buffer pool.

I tested very simple table, having with about 20.000 rows in it on 32bit Linux. The columns "id" "i" and "c" were populated with same integers so we can allow the same job to be done using different kinds of columns - primary key, integer indexed column and indexed char column. The query is also trivial - the point was to make sure it is not index covered query so it reads the rows and it does not return many rows. I varied the join clause to be id, i and C columns appropriately.

PLAIN TEXT SQL:

1. CREATE TABLE `t1` (
2.   `id` int(10) UNSIGNED NOT NULL DEFAULT '0',
3.   `i` int(10) UNSIGNED NOT NULL DEFAULT '0',
4.   `c` char(15) DEFAULT NULL,
…

Check out this cool INSERT ON DUPLICATE KEY UPDATE feature in MySQL 4.1. This will make some logging code I have much easier and cleaner than it currently is.