The other night I was thinking about MySQL cluster and whether or not a high speed interconnect would be worth the investment. Doing some research lead me to the conclusion that ethernet latency isn't discussed very often except in niche mailing list threads. I had to do a lot of research to find even a basic discussion of the problem (Note. If you have any good resources on the subject please add a comment.)

In order to help other developers avoid mistakes I've made in the past I figure a good blog post on the subject was in order.

What's ethernet latency? When you send a packet to a remote machine over gigabit ethernet it's not just instant. There's overhead involved in sending the packet back and worth. The data has to go from user space into the kernel, out the network interface, onto a switch, and then back up the kernel and into userland …