Sometimes, I make mistakes. It’s true. That can be difficult for us Systems Engineering-types to say, but I try to distance myself from my ego and embrace the mistakes because I often learn the most from them. ..Blah, blah, school of hard knocks, blah, blah…. Usually my mistakes aren’t big enough to cause any visible impact, but this one took the site out for 10 minutes during a period of peak traffic due to a confluence of events.

Doh!

Here is how it went down…

We have an issue where MySQL table statistics are occasionally getting out of whack, usually after a batch operation. This causes bad explain plans, which in turn cause impossibly slow queries. An ANALYZE TABLE (or even SHOW CREATE INDEX) resolves the issue immediately, but I prefer not get woken up at 4AM by long running query alerts when my family and I are trying to sleep. As a way to work around the issue, we decided to disable InnoDB automatic …

In early 2006 Paul Hurley (ideeli’s CEO) and I (Mark Uhrmacher, CTO) were thinking about a new business. We had the idea to create a community based around great deals for Women’s fashion products where we saw a great deal of potential for great content and product sales. Now, over five years later, we’ve realized much of that vision. Our business success has been chronicled over the years in several places (see here and here). Though we’re very proud of our achievements there, that isn’t what this blog is about.

Insatiable Demand is about a mostly untold story. Over the past five-plus years we’ve built a phenomenal technology platform and team. From two people and three servers to a 70 person team and a 100 instance production environment, …

We've published our first case study. The customer, ideeli, had a database that was struggling on standard MySQL and InnoDB. The big win was the upgrade to XtraDB. The business continued to grow quickly, and months later under much more traffic, the database is still outperforming their previous version.

I thought I'd write a few notes that didn't seem appropriate to include in the case study, because this was a fun project that might be interesting to readers.

As usual, it was all about diagnosing the problem correctly. I used a variety of tools to help with this, foremost among them "stalk" and "collect" from Aspersa. There were several problems, not just one, and they required different techniques to diagnose. This can be hard when the …