Ever see this
TCP: drop open request from 10.209.23.142/43407
Well lets start with a more specific example:
Memcache is tightly coupled in your code: Every request caches
the response from the database so a lot of quick calls to
memcache is made. Then you start adding full HTML to memcache
instead of just caching the raw data; so now your load pattern is
bigger blobs of data still at a high request rate.
Now suddenly the memcache port hangs-you verify this by ssh to
the box and then telneting the the memcache box port 11211 and
see that ssh works (port 22) yet 11211 does not. As a result all
your front ends fall over because they are hanging on the
memcache port.
THIS IS NOT A MEMCACHE PROBLEM. Its a kernel problem. Default
installs of Linux set the TCP window buffer size to a desktop
setting and not a server setting.
So I run this script.
…[Read more]