I am doing the final prep work for my upcoming UC presentation on SSD’s, and I thought I would throw this out their. Recently their has been a great deal of discussion on the write cache on the Intel x-25e and whether you need to disable it to prevent data loss on a power outage. Most disk caches are not protected by a battery backup and are disabled by default on most high end controllers. Who wants to potentially lose 16-64MB of data on an outage? So it seems like it would make sense that you should disable the cache on the Intel drives as well. But their is a problem. Vadim over at the MySQL Performance Blog recently published some benchmarks that show some rather slow results when the disk cache is disabled, in fact I have also noticed a significant slow down in these cases as

So we have already looked at sysbench & dbt2 tests… now we have to look at the new Juice DB benchmark. Juice runs a series of queries generate its load, these queries are combined into a workload. I tested the v1010 with a mixed workload ( mix of short & long updates and selects ), a mixed simple workload ( mix of short running updates and selects ) , and a read only ( selects which are designed to hit the disk ) . Because this is still an evolving benchmark I am including results from an Intel MLC drive (note these boxes are vastly different).  Keep in mind this is not a completely fair comparison. The Intel drive is not the enterprise class drive, but even with the SLC drive I don’t think its a fair comparison. The price difference between these two solutions is ~$50/GB -vs- ~$12.5GB.

The setup for this test created about a 20GB database, with each of the 3 large tables coming

Continuing my series on the Violin Memory 1010 I am turning my attention to the DBT2 benchmark which simulates an OLTP workload. I started with my typical “waffle” workload which is a 20 warehouse setup ( about 2.5 GB ) with a 768M buffer pool and I compared it to a 5G buffer pool with the same setup.  The ultimate goal or the nirvana state of any system is to have the performance of the storage system be as fast as having everything all in memory. The closer we can get the better off we are. The sad thing is even with the fastest of flash solutions we see times in the 70-300 microsecond response time range,  which is very  far off the nano second response time delivered by memory. That being said lets see how close we can get to a fully cached database:

I am including the Intel #’s for

Continuing my series of in depth looks at flash appliances, sans, and drives I spent a few weeks test driving the Violin Memory flash ( and DDR based ) solutions. Just from the specs the Violin Memory 1010 is impressive. According to the site the v1010 does 300K random reads per second and 200K random writes and has latency of less then 300 microseconds! That is pretty impressive!  But as I have stated before its difficult to test these limits with our current set of benchmarks.   For my test’s I did run this through the  ysbench fileio tests and dbt2 to get a feel for performance, but I was really eager to test the new juice db benchmark to really drive IO.  For the test Violin generously made available a 4 core (3.4Ghz ) server with 8GB of memory with access to a 360GB DDR based v1010 and then a 320GB DDR based v1010. Unlike the Ramsan I

This is part 3 in my RAMSan Series.

While I am confident the read-only test was a reasonably good test ( I just needed to push more ), my mixed load test was marred by issues.  It was really a quick attempt to get a heavy read/write workload.  I ran into issues with how I wrote this so I will spare you the details.  Some flash devices are notoriously poor performing in writes, so its important to at least briefly look at this.  What I will share are the IOPS & latency numbers from this test.  The mixed workload does updates & selects at this point, these are a mix of PK updates, secondary index updates, etc.  These typically are built to run faster and smaller the the read-only IO bound workload.

By the 11th interval the Ramsan was pretty much complete.  The peaks are whats interesting…  lets look

This is part 2 of My RAMSan Series.

In my normal suite of benchmarks I typically run dbt2 & sysbench oltp benchmarks next…  and I did run then, but to be honest they just weren’t that interesting.  They showed an improvement over my intel ssd results I ran on frankenmatt,  but it was difficult to provide an apples to apples comparison.   The server hardware was way different ( cpu, memory, controller, etc ).  Plus I typically run a test -vs- non-flash then a test with flash, and ran tests with varying degrees of memory… the test box had 2GB of memory and sparse internal disk, so my normal test cycles were already in jeopardy.  For what I ran   I was pushing CPU limits long before I was hitting the IOPS I saw above.  In fact in a 100W test I ended up peaking @ 1200 iops, while the CPU was @ 100%.

The challenge is building an effective solution that will easily maximize

Well its about time I posted this:)  This is part 1 of 3 in my Ramsan series.

For those who have paid attention to my blog, know I love talking  IO!  I also love performance.  Absolutely love it.  Love disk, disk capactiy, io performance, solid state..  So as I march towards my UC session on MySQL Performance on Solid State Disk my goal is to try and test as many high end solid state disk systems as possible.  All the vendors have been great, giving me access to some really expensive and impressive toys.  I finished up testing Texas Memory System’s flash appliance the RamSAN 500 this week and wanted to post some numbers and some thoughts. TMS makes RamSAN appliances that merge disk and RAM into a really fast SANS.     First I go a ways back with TMS, I deployed and Oracle Rac installation on one of their RamSAN 300’s several

So since last talking about a new benchmark I found a need for one.  Specifically the need to truly test IO bound workloads in mysql to flex fast storage subsystems.  Tackling a new benchmark is not as easy as I thought.  I am already on version 2 of my code and its still really a mess.  But it appears solid enough to try and give it a few test runs on some really fast disk subsystems.  So Armed with several really bad queries I put together a read-only workload that just hammers the disk and little else…  I am working a little under the gun as I only have access to the TMS Ramsan for 1 more night, and only have access to a Violin memory test box for a short while longer as well.  The problem I am running into with these systems is DBT2 and sysbench bottleneck the CPU before they really flex the disk.  I want to flex the disk damn it!

So how’s it going?  reasonably well. I am not happy

A quick one here, we often talk about effectively utilizing memory to get the most of your MySQL Server.  I wanted to remind folks to not forget about allocating memory to a tmpfs for the tmpdir.  Certainly this is not going to help everyone, but those who create lots of temporary table will find the performance boost most welcome. In fact in some cases you maybe better off allocating a little extra memory to a tmpfs then you would be the innodb buffer pool, but like everything it depends on your workload and environment.

What is tmpfs?  In a nutshell a filesystem ontop of a ramdisk  used for temporay file access.  Read the wiki page for more.   Sounds like a great thing for /tmp right? By the way It really urks me that most people leave /tmp as part of the root filesystem ….  shameful, read my common mistakes post for more complaining about

Let’s grab your attention with a shameless marketing ploy:  How would you like up to a 14X performance boost in your Mysql database performance? Now your hooked who wouldn’t want a 14x boost?  Well we got that using Dolphin interconnects and Waffle Grid! Read on!

As you know over the last few weeks the good folks over at Dolphin Interconnect Solutions have lent me use of a couple of servers with their interconnects installed ( They also sent me a couple of cards to test with, but I ended up not being able to secure the hardware to put these through their paces at my home office ) .  I ran into several challenges in the testing, and we also found out a lot about Waffle Grid and where we have some open bugs ( its good to test Waffle at the fringe of performance and see what breaks).

The oddest thing I ran into here was the way using the Dolphin

Ok, I am perplexed… i don’t say that often.   I have the privilege of testing out a couple of Dolphin Interconnects with Waffle Grid.   They are proving to substantially improve our transactions throughput, I mean we are getting 3x performance over 1gbe…. but what is perplexing is each run going over the faster interconnects results in 1/3 of the memcached sets/gets that occur when testing over 1gbe!  Same datasets, same tests, repeatable results.  See here:

1gbe:
         cmd_get: 771811
cmd_set: 784119

Dolphin:

         cmd_get: 239423
cmd_set: 271259

So instead of testing out the interconnect performance, I am really seeing better results from a higher cache hit rate.  Less items are hitting the LRU.   So what could it be?  Well We are doing way less read-aheads then normal:

One of our customers reported strange problem with MySQL having extremely poor performance when sync-binlog=1 is enabled, even though the system with RAID and BBU were expected to have much better performance.

The problem could be repeated with SysBench as follows:

On Dell R900 with CentOS 5.2 and ext3 filesystem we get 1060

Some good news.  I have been able to secure some additional hardware for testing over the next few months.  First up I received a pair of Dolphin interconnect cards which I will be putting through the paces sooner rather then later.  These cards will really help complete our Waffle Grid Testing.    Also I had a conversation last week with Texas Memory Systems who has agreed to let me loose on one of their RamSan 500’s  for a few days in February.   And Finally I have reached out and made contact with the folks at FusionIO and Violin Memory to try and setup some time to test with their products as well.  (Still trying to get my hands on one of the new Sun Storage Arrays as well as some Intel SLC drives to compare to the MLC

One typical question which frequently pops up is whenever it is better to use hardware upgrade or optimize software more. I already wrote about it, for example here.

Today I'll look at the same topic from the consultants view. When consultant should suggest hardware upgrade and when it is not in a simple checklist form.

How good is hardware ? Sometimes people use so crappy hardware it would be much cheaper for them to upgrade before purchasing much of professional services. Though in some cases people like their system to be optimal and so they want to run it on some old box even if it costs them more to optimize it. It may be valid choice allowing to take a hardware

As one of the co-founders of the Waffle Grid project, I beam with pride every time I get a stellar benchmark or every time I find a new use for the Waffle.  But as a professional I still have to be critical of all solutions I would recommend or deploy.  One of the big goals of Waffle Grid is to replace disk IO which should be slow with remote memory which should be much faster.  But what happens when the disk is no longer slow?  This leads me to ask myself, is Waffle Grid only good for servers with slower disk?  Or is this a solution that can also help systems with fast disk?  So which should you deploy SSD -vs- Waffle?  Are they competitors?  Or are they complementary technologies?

I am going to say this, in these tests latency is king.  The faster the drives can deliver data, the higher the benchmarks should be.  Basically if my interconnect can deliver faster then the drive can serve up data, I

Following up on my previous post Peter asked if he could see a 24 hour run on the Intel drive…  took me a few days because I am doing some testing on a few other things, but I kicked a run off yesterday before leaving the Vegas airport…  Here ya go:

These are roughly 10 minute TPM averages.  As you can see their is a definate decline in performance the longer the drive is active.   Its realtively small% wise, only about 6% off of peak…  but the delcine is easy to spot.

What’s this mean?  It you beat the hell out of the drive over a long contious period things slow down the longer the drive is in continuous use.   Just an FYI, the drive was 78% full during these tests.

As you may have recently seen there are some articles about scaling MySQL one 256-way system.

I though wow did they really make it work, considering how many bottlenecks remain in MySQL.

What article really tells us ?

First the number 256 - this is not number of Cores... this is number of hardware threads which is not exactly the same thing. Each T2 Plus CPU has 8 cores, which with 8 threads each giving 64 threads per chip or 256 threads all together.

Now what is about MySQL scaling to use these 32 cores with 256 threads ? Especially with the goal of "Do it with minimal tuning i.e as close as possible as out-of-the-box" ? Do we simply start MySQL server and change couple of

Thought I would throw this out here so I remember it, and it maybe useful for someone else.  During a session here at Sun’s CEC conference one of the presenter was talking about setting CPU affinity for hardware interrupts.  This made me think of our hack ( workaround ) for innodb scalability on a 16 core machine ( set the mysqld affinity to cpu’s 0-7 or 8-15 ).   In order to get the top performance out of a 16 core machine we probably want to ensure everything else goes to the cpu’s not set to be servicing mysql.   So a quick google search came up with this article on setting the affinity for IRQ’s in the linux kernel.   So with a 16 core box, I could set the affinity for Mysqld to cpu’s 8-15, while setting the affinity of the interrupts to something in the 0-7 range.

As promised, here are the DBT2 results for the Intel SSD drive:

As you see the Intel drive blew away all the competition here… even besting another dbt2 score I got from a nice new shiny 8 disk raid 10 system.

Doing dbt2 tests on on the intel drive today…  One of the strange things I ran into last time testing out my memoright drive was running the RPM version of the enterprise binaries -vs- the tarball version seemed significantly faster.  I had some other folks try it on other hardware and they could never replicate the performance slowdown,.  Basically before I was getting 4407 TPM in DBT2 from the RPM (5.0.60) in Centos 5, while the tarball (5.0.60) was only hitting 2505 TPM.   This was consistant.  Now I see that my most recent run of dbt2 against the Intel SSD acheived 3600 TPM/s, which is lower then the rpm, but higher then the tarball ( this was achieved via tarball ).   As i said this difference has not been verified independently, and it could be any number of odd factors at play on my hardware.

I need to go back and figure this out again…   But on a positve note, apples to apples the

Seeing my recent love affair with solid state drives I thought I would test drive one of the latest greatest drives out their the 80GB intel x-25m80GB.  Like a child on Christmas morning, I felt true excitement as the generic UPS envelop arrived on my porch today.

While it did not show up until late in the day, I can’t just let it sit their without starting to test it can I?

Benchmarks are running as I write this and I will provide the full breakdown of the drives performance as I finish up the tests.

But to wet your appetite, check this out:

50-50 read/write sysbench test:  1899 IO requests per second!!!  Thats huge!!!

Thats compare to the 284 IOPS I got on the memoright GT, a performance improvement of 6.6x, with a higher capacity 80GB -vs- 32GB, and a Lower cost $773 -vs- $680…  outstanding!!!

Here are the first unverified sysbench test runs:

I presented these at an internal MySQL professional services meeting about a month ago… its mostly a hodge podge of various benchmarks… but enjoy! The big difference in these benchmarks vs the other benchmarks is I am testing on the memoright GT drive, which is supposed to be one of the fastest SLC drives out their currently. Lets get right too it:

Looking at sysbench Random read/write iops:

The Memoright drive is noticeably faster then the Mtron drive in the synthetic random read/write tests. But the 100% write tests are substantially lower then the mtron drive which is just weird.

In the sysbench OLTP test, the memoright was slower then the mtron I tested earlier:

OLTP

So Yves knowing my affinity for all things solid state forwarded me this link, http://torvalds-family.blogspot.com/2008/10/so-i-got-one-of-new-intel-ssds.html , it seems Linus Torvalds picked up one of the new Intel SSD drives ( if anyone wants to send me one to test that would be cool ).  Whats interesting is he says the thing just rocks.  But how will this perfom in a database environment?  Not 100% sure, but I think it’s going to perform worse then the mtron or memoright drvies I have tested.  Why?  Well the drive is MLC not SLC.  Anandtech has a great review the Intel SSD, with an awesome explination with accompaning benchmarks on SLC -vs- MLC.  Most of these tests are performed in a windows

I picked up what some claim is the fastest of the current SSD drives this week, the Memoright GT.  From articles around the net I have seen performance speeds substantially faster then the mtron drive I tested earlier…. so I took the plunge.  The first results?  luke warm.  The random performance ( databases like mysql are all about random performance )  of the drive is better in mixed read/write tests over the mtron, but the mtron blew it away in random read performance.  This are are my first pass tests, so I may have something wrong …  so take them with a grain of salt:

The split tests were higher in all but 1, but the random read alone was 5x slower! more later.