I've been looking at a problem with our memory cache system at work for the last week and the problem finally clicked into place.
At work I developed this really awesome benchmarking library which exports statistics from various subsystems in our robot. It works really well - too well sometimes.
Its been telling me that one of my key caches is only functioning at 20% efficiency.
This is a memcached cache which means its broken out into many small servers running 256M or 512M or memory (really whatever they can spare).
Each of these servers was reporting that they are only 70% full so therefore I don't need to add any more memory right?
Wrong. I was look at the number of stored bytes vs the total maximum amount of bytes allocated to memcached.
The problem is …
[Read more]