We're trying to migrate to an 8x SSD RAID on Linux software raid on Linux 3.0.3.
Our previous server config was 96GB of memory which was rather expensive per month. Our goal is to migrate to SSDs which are about 1/2 to 1/3rd the price.
The problem we're having is that it's not keeping up with our replication load and is falling behind.
After a bunch of tuning we noticed the following IO load:
You can see that we're writing at a pretty solid rate... about 180MB/s at the peaks.
But I'm trying to figure out why we're falling of so significantly. As soon as reads come into the equation, performance completely falls. The server isn't doing much reading, it's just that it completely destroys our write throughput.
I've made a number of reasonable SSD performance changes.
We're on XFS with noatime and nobarrier.
The drives we're using are Micron p300s at 100GB each. We have 8x of them on an Adaptec RAID controller on a 16 core 2.8Ghz X5570.
It's a RAID 0 setup. The RAID stripe size is 256k. I played with smaller stripe sizes and this seemed like the best value.
The IO scheduler is now noop. We tried deadline as well and performance was about the same.
Here's a raw pastebin of our MySQL config.
http://pastebin.com/raw.php?i=Bn13Hhj9
The important changes we made are:
innodb_log_file_size=10G
innodb_read_ahead=none
innodb_flush_neighbor_pages = 0
innodb_doublewrite=0
innodb_io_capacity=160000
innodb_read_io_threads=64
innodb_write_io_threads=64
our buffer pool size is about 42GB ... the box has 48GB.
One obvious change I'd like to make is to use 4k pages. However, that's a days (at LEAST) worth of work because I have to do a full mysqldump of the entire database and re-import.
But honestly, that might just resolve the situations because, in theory, I would be writing/reading 1/4 of the data. But it still confuses me that the reads are causing such a massive write falloff.
Any feedback would be appreciated. It might also help me resolve the problem by triggering an idea.