Home |  MySQL Buzz |  FAQ |  Feeds |  Submit your blog feed |  Feedback |  Archive |  Aggregate feed RSS 2.0 English Deutsch Español Français Italiano 日本語 Русский Português 中文
Showing entries 1 to 30 of 48 Next 18 Older Entries

Displaying posts with tag: Waffle Grid (reset)

Filtering by table is now possible with WaffleGrid
+1 Vote Up -0Vote Down

Since I have been a home recently, I put some time correcting bugs in WaffleGrid and adding new features. Thanks to gdb, I have been able to understand a silly bug that was affecting WaffleGrid with sysbench but, weird enough, not with dbt2. Everything is in the way connections are established. I will blog more about that soon.

Regarding the new features, it is now possible to choose which tables you want to push to memcached. For that purpose, two new parameters have been introduce:

innodb_memcached_table_filter_enable = 0 | 1   (default to 0)

to enable the filtering and

innodb_memcached_table_list = db1/table1,db2/table2   

to list the tables. This feature is filtering based on the space id so, innodb_file_per_table has to be set. Right now, the association table space_id is done only at startup so, the table has to exist. Also, since an alter table

  [Read more...]
You know you have really screwed up when …
+3 Vote Up -0Vote Down

You crash MySQL/Innodb and you trace the error to a function that says:

“This function checks the consistency of an index page when we do not
know the index. This is also resilient so that this should never crash
even if the page is total garbage. ”

Oops… I guess its not that resilient after all:)

Waffle Grid Like Features in EnterpriseDB
+1 Vote Up -0Vote Down

Yves pointed this out to me:


“Moving beyond just providing Oracle compatibility, EnterpriseDB is adding new scalability features to their database. One of them is a feature called ‘”Infinite Cache,” which is based on the open source memcached application.

Mlodgenski commented that distributed memory caches are common, though management from a system management and a database perspective is often difficult.

“What we did is we took the distributed memory cache that was sitting in front of the database and put it behind the database,” Mlodgenski said. “This allows a simple SQL (define) interface for developers that allows access to the distributed memory cache and now the system management overhead is handed by the database itself.”

Mlodgenski added that Infinite Cache leads to a

  [Read more...]
Testing Waffle
+0 Vote Up -0Vote Down

So Ronald Bradford has been getting some EC2 AMI’s setup for Waffle the last couple of days. Really he is the first person to work with us on testing out the 0.5 cream release. It’s funny but when you are heads down in the code you tend to overlook some of the easy and obvious bugs. Just by having him walk through the install we have opened up several new bugs and come up with some new Idea’s on how to make things easier in the future. I hope by early next week we can have some of annoying bugs fixed and code pushed to launchpad. When that happens I would encourage others to give it a whirl and help us make Waffle Grid better performing, more user friendly, and more stable.

You can check out some of the bugs here.

Thanks to Ronald for his help on this!

WaffleGrid: Cream Benchmarks, stable and delivering a 3x boost
+0 Vote Up -0Vote Down

Lets get down to how the latest version of Waffle Grid performs.

Starting off simple lets look at the difference between the wafflegrid modes. As mentioned before the LRU mode is the “classic” Waffle Grid setup. A page is put into memcached when the page is removed from the buffer pool via the LRU process. When a page is retrieved from memcached it is expired so its no longer valid. In the New “Non-LRU” mode when a page is read from disk, the page is placed in memcached. When a dirty page is flushed to disk, this page is overwritten in memcached. So how do the different modes perform?

4GB Memcached, Read Ahead Enabled TPM % Increase No Waffle 3245.79 Baseline Waffle LRU 10731.34 330.62% Waffle NoLRU 10847.52 334.20%

You can see here that with 100% of

  [Read more...]
WaffleGrid: 0.5 Cream Release
+0 Vote Up -0Vote Down

I wanted to let everyone know that we are releasing Waffle Grid 0.5 code named Cream. This release fixes the nasty secondary index bug that plagued the butter release. I have been running tests on this code base for about a week straight with no errors. While I think this release is much more stable I would remind everyone this is still not a fully GA release. This release includes the ability to choose the mode of Waffle grid. By setting innodb_memcached_enable to 1, we will push pages to memcached when a disk read is done or when a page write is done, setting this to 2 will enable the classic LRU mode. If you decide to set this to 1 ( non-lru) I would recommend using the standard memcached, as with previous versions the LRU mode works better with our slightly altered memcached ( expire from memcached on get ). I will be posting benchmarks and more details within the next couple of days. Right now you can

  [Read more...]
Waffle: Progress and a Rearchtecture?
+1 Vote Up -0Vote Down

So I spent several hours over the last few days on the Secondary index bug. Out of frustration I decided to try and bypass the LRU concept all together and try going to a true secondary page cache. In standard Waffle a page is written to memcached only when it is expunged ( or LRU’d ) from the main buffer pool. This means anything in the BP should not be in memcached. Obviously with this approach we missed something, as Heikii pointed out in a comment to a previous post, it seems likely we are getting an old version of a page. Logically this could happen if we do not correctly expire a page on get or we bypass a push/lru leaving an old page in memcached to be retrieved later on.

So I was thinking why not bypass the LRU process? While I feel this is the most efficient way to do this, its not the only way. I modified innodb to use the default LRU code and then modified the page get to push to

  [Read more...]
Waffle: The Mystery Continues
+0 Vote Up -0Vote Down

So I spent the weekend looking at places where we may have missed something in the code for waffle. You can actually see some of the stuff I tried in the bug on launchpad about this, but the weird thing is the very last thing I tried. As I took a step back and looked at the problem ( secondary index corruption ) and our assumption that we “missed” something, I decided to find the place where pages are written to disk and to push to memcached from here as well as from the LRU. With the double write buffer enabled that place should be buf_flush_buffered_writes. By pushing to memcached here we should eliminate the page that falls through the cracks of the LRU. Basically this should help ensure memcached has an exact copy of the data that exists on disk. The result? It failed with the same secondary index failure. This means:


  [Read more...]
Waffle: limiting the space ids being pushed to memcached
+2 Vote Up -0Vote Down

If you read Yves blog post about waffle yesterday we are seeing some weird gremlins in the system and could use some scoobey doo detective work if you have some ideas. The strange thing is it only exhibits under high load. So it really seems like we may have missed some background cleanup process that accesses or removes pages from disk or the buffer pool without going through the functions we call waffle in (buf_LRU_search_and_free_block & buf_read_page_low ).

One of the idea’s I had was trying to narrow the scope of what’s being pushed and read form Memcached. Even though I am using file per table, system tablespace pages are still making it in and out of memcached. I thought if we missed something maybe it was here ( even though I could not find it in the code ). I mean cleaning up undo or internal data would seem like a logical place to

  [Read more...]
What’s up with WaffleGrid?
+0 Vote Up -0Vote Down

You probably haven’t noticed but I have not blogged since the UC. It is not because I am upset by the perspective of working for Oracle, I have simply been busy tracking down an issue we have with WaffleGrid. We discovered that under high load, with DBT2 on a tmpfs, we end up with error in a secondary index. In the error of MySQL, we have entries like this one:

InnoDB: error in sec index entry update in
InnoDB: index `myidx1` of table `dbt2`.`new_order`
InnoDB: tuple DATA TUPLE: 3 fields;
 0: len 4; hex 80000001; asc     ;;
 1: len 4; hex 80000bea; asc     ;;
 2: len 4; hex 80000005; asc     ;;

InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 32
 0: len 4; hex 80000001; asc     ;;
 1: len 4; hex 80000bea; asc     ;;
 2: len 4; hex 80000004; asc     ;;

TRANSACTION 14469, ACTIVE 1 sec, process no 7982, OS thread id 2995481488 updating or deleting
mysql tables in use 1, locked
  [Read more...]
Random notes from conversations at the UC
+2 Vote Up -0Vote Down

People always have conversations at the UC that spark ideas for new projects or features, this is my random collection of features or items I need to look into.  Most people can ignore it… but if your interested in what goes on in my head here ya go:

General Innodb:

  • I started looking into this before, but I was reminded of it again today…  it would be useful to setup separate innodb buffer pools like the key caches in MyISAM.
  • maybe an easy hack to put in a tiered LRU based on file per table.  What I mean by this is internally the internal BP in inno actually uses the space id ( file per table will have a new ID per table )…  this should make it feasible to ( maybe easy ) to move certain spaceid’s up or down the internal LRU list  This maybe an interesting feature, not exactly like pinning a table into the BP but close.    Basically, if a table is accessed
  [Read more...]
Counting down the days before UC Time!!
+0 Vote Up -0Vote Down

One last reminder, we ( Yves and I ) will be appearing all over the place at the UC. Here is a quick run down of our schedule:

  • Tuesday 10:50 am - 11:35 pm (Free MySQL Camp) I will be presenting “Learning from others’ MySQL Performance Mistakes” … a journey into the sometimes silly things we all end up doing.
  • Tuesday 2-3pm Yves and I are co-presenting “Distributed Innodb Caching with memcached” … this will be an intro into the Waffle Grid project, what it’s about and what it means to you.
  • Tuesday 4:25 pm - 5:10 pm (Free MySQL Camp) Yves will be presenting “NBD (MySQL Cluster) performance tuning and pitfalls”
  • Wednesday 2-3pm I will be presenting: “SAN Performance on a Internal Disk Budget: The Coming Solid State Disk Revolution” … the title sounded better when I
  [Read more...]
WaffleGrid is plugging along!
+0 Vote Up -0Vote Down

Yes we are still plugging away working on Waffle Grid, in fact I am testing the heck out of the plugin release this week. Some good news, great news, and bad news to report.

The good news on the testing is using the new Waffle Grid release, I am able to consistently get up close to 15K TPM, that’s up from 3K TPM without WaffleGrid ( 5x increase woohoo! ). This performance boost holds true through several tests, and based on this testing I am working on a set of recommended parameters for getting the most performance out of Waffle. I should have my recommendations for our User Conference presentation.

The great news is I have been able to get close to 20K TPM by disabling the read-ahead! This represents a huge boost in performance, over both the read-ahead enabled and the non-waffle tests. The bad news is in extended tests with the read-ahead disabled the plugin version takes a crap on

  [Read more...]
Wafflegrid memcached 1.3.2
+0 Vote Up -0Vote Down

Quick note I got memcached 1.3.2 working with waffle. I added a lot more code then I needed, and I think I may be able to get this down to a 1 or 2 line patch… In case your interested in a slightly bloated waffle version of 1.3.2 its here: https://code.launchpad.net/~yonkovim/wafflegrid/memcached-1.3.2

I will clean this up this week and push a better more streamlined version then.

Playing with Waffle & Mecached 1.3.2
+0 Vote Up -0Vote Down

So I got our Waffle specific code over into Memcached 1.3.2 last night…  not 100% sure why at this point but I am seeing a huge change in the the hit rate between 1.2.5 and 1.3.2.  Take a look:

Old 1.2.5:

10290.89 new-order transactions per minute (NOTPM)

Server: (11211)
pid: 16522
uptime: 2190
time: 1233504281
version: 1.2.5
pointer_size: 32
rusage_user: 23.753484
rusage_system: 127.611975
curr_items: 46800
total_items: 3487050
bytes: 769596712
curr_connections: 20
total_connections: 23
connection_structures: 21
cmd_get: 1752218
cmd_set: 1801081
get_hits: 1685969
get_misses: 66249
evictions: 1680669
bytes_read: 769596712
bytes_written: 769596712
limit_maxbytes: 805306368
threads: 1

New (1.3.2):
6695.67 new-order transactions per minute (NOTPM)

Server: localhost (11211)
	 pid: 9778
	 uptime: 2087
	 time: 1238184346
	 version: 1.3.2

  [Read more...]
The upcoming MySQL UC 2009
+0 Vote Up -0Vote Down

The MySQL UC 2009 is coming and it is time for my own little marketing. As Matt already annonced it a few months ago we (Matt and I) are doing a WaffleGrid presentation, Distributed InnoDB caching with Memcached, Tuesday at 2PM. I am also presenting at the MySQL Camp or unconference, NBD (MySQL Cluster) performance tuning and pitfalls, also Tuesday at 4:25PM.

WaffleGrid ported to the InnoDB plugin 1.0.3
+0 Vote Up -0Vote Down

For the interested, I just ported Wafflegrid to the newest version of the InnoDB pluging, v. 1.0.3. Meanwhile, I also corrected a small bug with the CRC_32 code that wrongly cause MySQL to report corrupted blocks after being retrieved from memcached. To get the new code, simply do:

bzr branch lp:~y-trudeau/wafflegrid/waffleGrid-Innodb-plugin

and replace the directory storage/innobase by what you will retrieve from launchpad. If you are new to WaffleGrid, you will also need a custom version of memcached and a modified build script for MySQL. All these can be obtained at:


You will also need to change the file memcached_constants.h in libmemcached with the following value:


This gives a 20x perf gain… Yes… WaffleGrid is still experimental.

Waffle Grid: Async IO Concerns
+0 Vote Up -0Vote Down

Last night I was on a plane to a client site and I was reviewing the waffle grid code for inclusion of multi-gets when calling read-ahead functions (potentially in other area’s as well)…  their I noticed something that may slow down our performance in waffle grid.  You see we read from memcached in the buf_read_page_low function.  This function is responsible for checking wether a page exists in the buffer and if not making an IO request for it via the function fil_io.  What I saw in buf_read_page_low was:  we check the buffer, then check memcached, then go get the page off disk… you follow?  Alright The get page from disk part is challenge.  Let me do a quick deep dive on the internal function calls.

You see buf_read_page_low is called by a few different functions in a few different ways, the ones I am concerned with are the functions that are pasing a sync of false ( you do not want

  [Read more...]
Patch to Disable Innodb’s Dynamic Stat Generation
+0 Vote Up -0Vote Down

Alright my stats patch appears to be working ( its a first step anyways ), eventually this is going to make it into waffle grid… but I thought it would be useful as a standalone patch.  Basically what this patch does is turn off the dynamic collection of stats by disabling the calls to dict_update_statistics_low & btr_estimate_number_of_different_key_vals for everything but the analyze statement. At this point its not really refined, but it does appear to work.  i.e. It will need more refining in the future and may not 100% work in every little case.  The short of it, this means that once you set your stats you should not have to worry about the cardinality of your tables changing dramatically.

Other databases ( even other storage engines ) allow you to control when and how your stats are gathered.  Lots of shops analyze their tables once a week using a large sample set with the idea that their

  [Read more...]
WaffleGrid new features
+0 Vote Up -0Vote Down

Recently, I have been traveling a lot to the west coast (I live in Quebec) so I had plenty of time to update WaffleGrid while flying. Here are the new features/ports:

  • Dynamic memcached servers list
  • Matt enhanced stats ported from the non-plugin InnoDB version
  • CRC32 checksum for each element sent and retrieved from memcached
  • Better thread protection
  • So now, if you update the innodb_memcached_servers with a statement like “set global innodb_memcached_servers = ’serverA,serverB:11212′;” then MySQL will disconnect from the current servers and reconnect to the new ones. The key prefix is incremented to prevent cache coherency problems.

    You might have noticed that Matt added many WaffleGrid statistic counters like shown here. Those counters have been ported to the InnoDB plugin version.

      [Read more...]
    What do we do here at bigdbahead.com over the weekend?
    +0 Vote Up -0Vote Down

    We have benchmark parties!  Yes , I spend my weekend glued to the LCD running benchmarks day and night.  While it may seem like the work of a madman or a lonely bunch of geeks without a date who are just trying to pass the time before their WOW quest begins…  its really rather fun.

    Check out this:  The RamSan 500 …. I have had access to this bad boy this week to perform all kinds of interesting test.  How fast is it?  50-50 read/write benchmarks I am pulling down 24K iops and I am not taxing the drive ( the benchmark system’s cpu is only a quad core with 2gb of memory )…  I will have a report for you later this week.

    What else?  Well my ec2 trials and tribulations continue…  sometimes I feel like I am trying to squeze performance out of an old 386 ( without the co-processor ).

    Of course I am

      [Read more...]
    Follow up on my Ec2 Latency Rant
    +0 Vote Up -0Vote Down

    I was testing up our latest Waffle release when I saw this horrid latency, average in the 12+ms range.  So I decided to dig a bit deeper today:

    i’ll start with a Single threaded test on the ec2 setup I was complaining about, this is running waffle 0.4 and memcached 1.2.5.

    mysql> set @@global.innodb_memcached_enable=1;
    Query OK, 0 rows affected (0.00 sec)mysql> use dbt2;
    Reading table information for completion of table and column names
    You can turn off this feature to get a quicker startup with -AseleDatabase changed
    mysql> select count(*) from stock;
    | count(*) |
    | 2000000 |
    1 row in set (17.79 sec)mysql> select count(*) from customer;
    | count(*) |
    | 600000 |

      [Read more...]
    Darn Ec2 Latency!
    +0 Vote Up -0Vote Down

    Memcached puts    141075
    Memcached hits    122696
    Memcached misses  18329
    Memcached Prefix:  29298
    Memcached Get Total Lat 12222 (us)
    Memcached Get Recent Lat 10925 (us)
    Memcached Miss Total Lat 8490 (us)
    Memcached Miss Recent Lat 5740 (us)
    Memcached Set Total Lat 142 (us)
    Memcached Set Recent Lat 65 (us)

    12ms?  Really, I mean come on…. what are we using 10baseT?

    at least it appears somewhat consistent.  Consistently bad, but consistent… and if you can not survive the odd spike to 35ms, what good are you.

    Waffle Grid: bugs & future features
    +0 Vote Up -0Vote Down

    As we have mentioned several times you probably know we are planning to migrate our development over to the innodb-plugin and start following the path of the storage engine.  That’s all well and good but what are the warts and bugs and things we are trying to fix?  Ahhh yes the seedy underbelly of Waffle Grid. The reason why we call it experimental.

    Big on my list is going to be ensuring data integrity.  Currently you can manually ( or via some weird network corruption ) overwrite a good innodb page with an invalid one, the result? Well lets check it out:

    Waffle Grid: 0.4 “Butter” technology preview
    +0 Vote Up -0Vote Down

    I have just uploaded a technology preview release of Waffle Grid code named “Butter”.  Yes Butter is good on Waffles.

    This release is a preview release, experimental, and should probably not be used in production unless you are brave…  I am including the full source for this release.  This includes patched memcached, libmemcached, and MySQL 5.1.30.  I bundled it this way to try and get folks to try it out and let us know what they think.  This is probably going to be the only time we package the full MySQL source as we are now working with the innodb-plugin.  You can still grab this as a patch off of launchpad.

    What’s in this release?  Well this is our first “release”  but some of the notable items since the last patch:

      [Read more...]
    Waffle Grid: Testing Waffle With Dolphin Interconnects
    +0 Vote Up -0Vote Down

    Let’s grab your attention with a shameless marketing ploy:  How would you like up to a 14X performance boost in your Mysql database performance? Now your hooked who wouldn’t want a 14x boost?  Well we got that using Dolphin interconnects and Waffle Grid! Read on!

    As you know over the last few weeks the good folks over at Dolphin Interconnect Solutions have lent me use of a couple of servers with their interconnects installed ( They also sent me a couple of cards to test with, but I ended up not being able to secure the hardware to put these through their paces at my home office ) .  I ran into several challenges in the testing, and we also found out a lot about Waffle Grid and where we have some open bugs ( its good to test Waffle at the fringe of performance and see what breaks).

    The oddest thing I ran into here was the way using the Dolphin

      [Read more...]
    Waffle Grid: Mini Update
    +0 Vote Up -0Vote Down

    I am currently working on putting together and pushing a 0.4 version of Waffle Grid. Which I think will be the last until we switch over to Plugin model. This version includes the use of libmemcached NoReply, all kinds of new stats ( latency stats for disk & memcached ), turning read ahead on/off, and a slew of fixes. I was able to push a lot of transactions using this, so It should be good for people to play around with and see what kind of performance they can get.

    Next up for us on the development side will be:

    • the planned port to the Innodb Plugin – this version should include a dynamic server list which will allow adding memcached server without a restart.

    • work on some scalability issues with  our current code ( namely we open 1 connection per innodb thread, per memcached server )

    • evaluating more performance enhancements, possibly a more

      [Read more...]
    WaffleGrid webinar
    +0 Vote Up -0Vote Down

    For the ones who are interested, there will be a webinar on WaffleGrid next Tuesday, February the 17th at 10:00 am PST (1:00 pm EST, 18:00 GMT). Matt will present benchmarks he did with Dbt2 and Sysbench using GbE, Dolphin, SSD while, for my part, I will present the WaffleGrid concepts, status and roadmap. I will also be a good way to learn how to get started with WaffleGrid if you want to start experimenting!

    Registration (http://www.mysql.com/news-and-events/web-seminars/display-279.html) is free!

    Waffle Grid: More fast interconnect madness
    +0 Vote Up -0Vote Down

    So as I mentioned earlier I am seeing some rather odd behavior when I testing Waffle Grid and really fast interconnects.  In a nutshell, with faster access to pages it appears innodb does not need to throw items out of the buffer pool as often.  Test after test shows with Dolphin Fast interconnects I get outstanding performance, but they end up putting less into memcached ( LRU’s blocks ).  The same tests performed over and over on 1gbe, get a fraction of the performance but put 3x the data to memcached.  Looking at the numbers the fast interconnect tests all share a lower number of read aheads then the 1gbe test.   I was really leaning towards the read-ahead as the culprit.  So I commented out the read-ahead.  The result.  The same.  The 1gbe is now faster then before, but the dolphin interconnect tests still blow it out of the water, and does 1/4 of

      [Read more...]
    WaffleGrid roadmap changes… a storage engine?
    +0 Vote Up -0Vote Down

    If you followed the recent blog posts by Matt, you have noticed the nice performance. This confirms that the Wafflegrid concept is a good one and we need to continue its development and improve it.

    You have also probably noticed there has been no recent WaffleGrid release, the reason is that we were thinking about how to distribute WaffleGrid. The current “patch” like approach is, let’s say, pretty poor. Another approach would be to work on the whole tree of MySQL or Drizzle but then, it would be pretty hard for someone to use InnoDB and WaffleGrid at the same time, not ideal either.

    Hey…. MySQL has a plugin interface for storage engines, why not just add a WaffleGrid storage engine. It would be much easier to add WaffleGrid to MySQL and Drizzle and we would also be far less impacted by new releases of the databases. So, the next release will be a branch of InnoDB

      [Read more...]
    Showing entries 1 to 30 of 48 Next 18 Older Entries

    Planet MySQL © 1995, 2014, Oracle Corporation and/or its affiliates   Legal Policies | Your Privacy Rights | Terms of Use

    Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.