This is the story of cold blocks and mismatched instances and how they will cause you pain and cost you money until you understand why.
Most of the clients that we support run on the Amazon cloud using either RDS or running MySQL on plain EC2 instances using (Provisioned IOPS) PIOPS EBS for data storage.
As expected the common architecture is running a master with one or more slaves handling the read traffic.
A common problem is that after the slaves are provisioned (normally created from an EBS snapshot) they lag badly due to slow IO performance.
Unfortunately what tends to be lost in the “speed of provisioning new resources” fetish is some limitations in terms of data persistence layer (EBS).
If you are using EBS and you have created the EBS volume from snapshot or created a new volume you have to pre-warm the EBS volume[Read more...]