We have benchmark parties!  Yes , I spend my weekend glued to the LCD running benchmarks day and night.  While it may seem like the work of a madman or a lonely bunch of geeks without a date who are just trying to pass the time before their WOW quest begins…  its really rather fun.

Check out this:  The RamSan 500 …. I have had access to this bad boy this week to perform all kinds of interesting test.  How fast is it?  50-50 read/write benchmarks I am pulling down 24K iops and I am not taxing the drive ( the benchmark system’s cpu is only a quad core with 2gb of memory )…  I will have a report for you later this week.

What else?  Well my ec2 trials and tribulations continue…  sometimes I feel like I am trying to squeze performance out of an old 386 ( without the co-processor ).

Of course I am …

I was testing up our latest Waffle release when I saw this horrid latency, average in the 12+ms range.  So I decided to dig a bit deeper today:

i’ll start with a Single threaded test on the ec2 setup I was complaining about, this is running waffle 0.4 and memcached 1.2.5.

12ms?  Really, I mean come on…. what are we using 10baseT?

at least it appears somewhat consistent.  Consistently bad, but consistent… and if you can not survive the odd spike to 35ms, what good are you.

As we have mentioned several times you probably know we are planning to migrate our development over to the innodb-plugin and start following the path of the storage engine.  That’s all well and good but what are the warts and bugs and things we are trying to fix?  Ahhh yes the seedy underbelly of Waffle Grid. The reason why we call it experimental.

Big on my list is going to be ensuring data integrity.  Currently you can manually ( or via some weird network corruption ) overwrite a good innodb page with an invalid one, the result? Well lets check it out:

I have just uploaded a technology preview release of Waffle Grid code named “Butter”.  Yes Butter is good on Waffles.

This release is a preview release, experimental, and should probably not be used in production unless you are brave…  I am including the full source for this release.  This includes patched memcached, libmemcached, and MySQL 5.1.30.  I bundled it this way to try and get folks to try it out and let us know what they think.  This is probably going to be the only time we package the full MySQL source as we are now working with the innodb-plugin.  You can still grab this as a patch off of launchpad.

What’s in this release?  Well this is our first “release”  but some of the notable items since the last …

Let’s grab your attention with a shameless marketing ploy:  How would you like up to a 14X performance boost in your Mysql database performance? Now your hooked who wouldn’t want a 14x boost?  Well we got that using Dolphin interconnects and Waffle Grid! Read on!

As you know over the last few weeks the good folks over at Dolphin Interconnect Solutions have lent me use of a couple of servers with their interconnects installed ( They also sent me a couple of cards to test with, but I ended up not being able to secure the hardware to put these through their paces at my home office ) .  I ran into several challenges in the testing, and we also found out a lot about Waffle Grid and where we have some open bugs ( its good to test Waffle at the fringe of performance and see what breaks).

The oddest thing I ran into here was the way using the Dolphin cards …

I am currently working on putting together and pushing a 0.4 version of Waffle Grid. Which I think will be the last until we switch over to Plugin model. This version includes the use of libmemcached NoReply, all kinds of new stats ( latency stats for disk & memcached ), turning read ahead on/off, and a slew of fixes. I was able to push a lot of transactions using this, so It should be good for people to play around with and see what kind of performance they can get.

Next up for us on the development side will be:

• the planned port to the Innodb Plugin – this version should include a dynamic server list which will allow adding memcached server without a restart.

• work on some scalability issues with  our current code ( namely we open 1 connection per innodb thread, per memcached server )

• evaluating more performance …

For the ones who are interested, there will be a webinar on WaffleGrid next Tuesday, February the 17th at 10:00 am PST (1:00 pm EST, 18:00 GMT). Matt will present benchmarks he did with Dbt2 and Sysbench using GbE, Dolphin, SSD while, for my part, I will present the WaffleGrid concepts, status and roadmap. I will also be a good way to learn how to get started with WaffleGrid if you want to start experimenting!

Registration is free!

So as I mentioned earlier I am seeing some rather odd behavior when I testing Waffle Grid and really fast interconnects.  In a nutshell, with faster access to pages it appears innodb does not need to throw items out of the buffer pool as often.  Test after test shows with Dolphin Fast interconnects I get outstanding performance, but they end up putting less into memcached ( LRU’s blocks ).  The same tests performed over and over on 1gbe, get a fraction of the performance but put 3x the data to memcached.  Looking at the numbers the fast interconnect tests all share a lower number of read aheads then the 1gbe test.   I was really leaning towards the read-ahead as the culprit.  So I commented out the read-ahead.  The result.  The same.  The 1gbe is now faster then before, but the dolphin interconnect tests still blow it out of the water, …

If you followed the recent blog posts by Matt, you have noticed the nice performance. This confirms that the Wafflegrid concept is a good one and we need to continue its development and improve it.

You have also probably noticed there has been no recent WaffleGrid release, the reason is that we were thinking about how to distribute WaffleGrid. The current “patch” like approach is, let’s say, pretty poor. Another approach would be to work on the whole tree of MySQL or Drizzle but then, it would be pretty hard for someone to use InnoDB and WaffleGrid at the same time, not ideal either.

Hey…. MySQL has a plugin interface for storage engines, why not just add a WaffleGrid storage engine. It would be much easier to add WaffleGrid to MySQL and Drizzle and we would also be far less impacted by new releases of the databases. So, the next release will be a branch of InnoDB plugin and from there, integrating to Drizzle will be also …