This week I helped dealing with performance problems (part MySQL related and part related to LAMP in general) of system which does quite a bit of content delivery, serving file downloads and images - something a lot of web sites need to do these days. There were quite a bit of mistakes in design for this one which I though worth to note, adding some issues seen in other systems.
Note this list applies to static content distribution, dynamic content has some of its own issues which need different treatment.
DNS TTL Settings The system was using DNS based load balancing, using something like img23.domain.com to serve some of the images. I'm not big fan of purely DNS based load balancing and HA but it works if configured well. In this case however the problem was zero TTL set in DNS configuration. This obviously adds latency especially for "aggregate" pages which may require images to be pulled from 10 different image …
[Read more]