Yes we are still plugging away working on Waffle Grid, in fact I am testing the heck out of the plugin release this week. Some good news, great news, and bad news to report.

The good news on the testing is using the new Waffle Grid release, I am able to consistently get up close to 15K TPM, that’s up from 3K TPM without WaffleGrid ( 5x increase woohoo! ). This performance boost holds true through several tests, and based on this testing I am working on a set of recommended parameters for getting the most performance out of Waffle. I should have my recommendations for our User Conference presentation.

The great news is I have been able to get close to 20K TPM by disabling the read-ahead! This represents a huge boost in performance, over both the read-ahead enabled and the non-waffle tests. The bad news is in extended tests with the read-ahead disabled the plugin version takes a crap on me and consistently dies about 22-24 minutes …

I have been benchmarking Waffle Grid using the new Innodb Plugin 1.03 the past couple of days. Let me say the plugin is fast. Which got me thinking, generally when you fix a bottleneck another area becomes a bottleneck… its a vicious cycle really. I figured why not benchmarks several different settings just to see what sort of improvement or detriment we get in Inno. This hopefully will lead to the next place to look for potential performance improvements. For the test I chose a somewhat IO bound setup and a CPU bound setup.

The IO bound setup was a 20W test, 768M buffer pool.
The CPU boud setup was a 20W test, 5GB buffer pool.

I decided to start with the Double Write Buffer. For those who are not familiar with the double write buffer check out the docs or …

So we have already looked at sysbench & dbt2 tests… now we have to look at the new Juice DB benchmark. Juice runs a series of queries generate its load, these queries are combined into a workload. I tested the v1010 with a mixed workload ( mix of short & long updates and selects ), a mixed simple workload ( mix of short running updates and selects ) , and a read only ( selects which are designed to hit the disk ) . Because this is still an evolving benchmark I am including results from an Intel MLC drive (note these boxes are vastly different).  Keep in mind this is not a completely fair comparison. The Intel drive is not the enterprise class drive, but even with the SLC drive I don’t think its a fair comparison. The price difference between these two solutions is ~$50/GB -vs- ~$12.5GB.

The setup for this test created about a 20GB database, with each of the 3 large tables coming in around 6 GB each. I tested primarily with a …

Quick note I got memcached 1.3.2 working with waffle. I added a lot more code then I needed, and I think I may be able to get this down to a 1 or 2 line patch… In case your interested in a slightly bloated waffle version of 1.3.2 its here: https://code.launchpad.net/~yonkovim/wafflegrid/memcached-1.3.2

I will clean this up this week and push a better more streamlined version then.

So I got our Waffle specific code over into Memcached 1.3.2 last night…  not 100% sure why at this point but I am seeing a huge change in the the hit rate between 1.2.5 and 1.3.2.  Take a look:

Old 1.2.5:

10290.89 new-order transactions per minute (NOTPM)

Server: 192.168.2.105 (11211)
pid: 16522
uptime: 2190
time: 1233504281
version: 1.2.5
pointer_size: 32
rusage_user: 23.753484
rusage_system: 127.611975
curr_items: 46800
total_items: 3487050
bytes: 769596712
curr_connections: 20
total_connections: 23
connection_structures: 21
cmd_get: 1752218
cmd_set: 1801081
get_hits: 1685969
get_misses: 66249
evictions: 1680669
bytes_read: 769596712
bytes_written: 769596712
limit_maxbytes: 805306368
threads: 1

New (1.3.2):
6695.67 new-order transactions per minute (NOTPM)

Server: localhost (11211)
         pid: 9778
         uptime: 2087
         time: 1238184346
         version: 1.3.2
         pointer_size: …

Continuing my series on the Violin Memory 1010 I am turning my attention to the DBT2 benchmark which simulates an OLTP workload. I started with my typical “waffle” workload which is a 20 warehouse setup ( about 2.5 GB ) with a 768M buffer pool and I compared it to a 5G buffer pool with the same setup.  The ultimate goal or the nirvana state of any system is to have the performance of the storage system be as fast as having everything all in memory. The closer we can get the better off we are. The sad thing is even with the fastest of flash solutions we see times in the 70-300 microsecond response time range,  which is very  far off the nano second response time delivered by memory. That being said lets see how close we can get to a fully cached database:

I am including the Intel #’s for perspective here and to show just how close we can get full in memory speeds. The fact is I am comparing a potentially …

Continuing my series of in depth looks at flash appliances, sans, and drives I spent a few weeks test driving the Violin Memory flash ( and DDR based ) solutions. Just from the specs the Violin Memory 1010 is impressive. According to the site the v1010 does 300K random reads per second and 200K random writes and has latency of less then 300 microseconds! That is pretty impressive!  But as I have stated before its difficult to test these limits with our current set of benchmarks.   For my test’s I did run this through the  ysbench fileio tests and dbt2 to get a feel for performance, but I was really eager to test the new juice db benchmark to really drive IO.  For the test Violin generously made available a 4 core (3.4Ghz ) server with 8GB of memory with access to a 360GB DDR based v1010 and then a 320GB DDR based v1010. Unlike the Ramsan I tested a …

Pushed about a ton of changes to the Juice benchmark since I announced it… lots of little fixes and a few big ones… really their are too many to talk about, and a lot of the new stuff is boring ( who cares about adding ramp up time, its just not that interesting to talk about ).

The big change is all about the Analyze.pl script. I am adding a ton to this script in order to try and give a concise and clear picture of what happened during the benchmark run. I am excited about it ( I am a geek though ) so I thought I would share what it looks like so far:

Total Test Runtime = 513.110480070114 seconds, limiting results to 300 seconds however
QNum:      4 ... QCount:      9 ... QTime:   0.028973 ... Max:   0.037164 ... FlatTime:   0.030530  ... Min5%:   0.015769  ... Max5%:   0.037164
QNum:      7 ... QCount:   2900 ... QTime:   0.001224 ... Max:   0.027039 ... FlatTime:   0.000658  ... Min5%:   0.000134  ... Max5%:   0.012549
QNum:      8 ... …

I was asked today about what is the cost of adding an index on a frequently updated column ( like a timestamp, count, or weight )… typically my answer is it depends. But for this question it was narrowed down to a specific case. An update on a secondary index based on a PK lookup. I decided to try and give an exact answer. I hacked the Juice DB Benchmark to attack my medium sized table ( which magically already had a count column in it ). I then cranked up the test. A few more details Query 23 updated a column without an index, queries 21,23,24 updated the d_count column. query 21 adds 5 to the count, query 22 adds 150, query 24 subtracts 1…. here are the results:

With a solo index on d_count:

Run Number:  86  threads:  8 Length :  340 LoadType: upd
Total Test Runtime = 375.245010137558 seconds, limiting results to 300 seconds however
QNum:     21 ... QCount:  78448 ... QTime:   0.003985 ... Max:   0.095937 ... FlatTime:   0.003673 …

How many fat marathon runners do you know? I have yet to see anyone run the Boston marathon who looks like me. What about a fat man competing in the 100 yard dash in the the Olympics? no? It seems to make sense then that maybe a fat/bloated index or primary key may not be able to compete for the performance crown.  As you may have read,  One of my common performance mistakes is using incorrect or bloated datatypes in your table design ( shameless plug: stop by the MySQLCamp unconference at the UC and hear me babel on about common PT mistakes). Unfortunately even wise datatype selection can result in suboptimal indexes. Raise your hand if you have heard someone tell you not to use a char/varchar as a PK in innodb?

Not everyone has. Two weeks in a row makes a trend, and a trend means I should probably blog about this. Ran into a pair of clients who are using char/varchar primary key fields in Innodb. I strongly urge my clients …