Michael Arrington of TechCrunch asks Twitter a few questions. I have only included a sample list below but you should read his blog post for all the questions:

• Is it true that you only have a single master MySQL server running replication to two slaves, and the architecture doesn’t auto-switch to a hot backup when the master goes down?
• Do you really have a grand total of three physical database machines that are POWERING ALL OF TWITTER?
• Is it true that the only way you can keep Twitter alive is to have somebody sit there and watch it constantly, and then manually switch databases over and re-build when one of the slaves fail?
  

A 'yes' answer to any of these questions by Twitter would be disturbing to say the least. However, …

Following a link from the High Scalability blog, I found this really great article about scalability practices, as told by Randy Shoup at eBay. Randy is very good at explaining some of the more technical aspects in more or less plain English, and it even helped me find some wording I was looking for to help me explain the notion (and benefits) of functional partitioning. He also covers ideas that apply directly to your application code, your database architecture (including a little insight into their sharding strategy), and more. Even more about eBay’s architecture can be found here.

How often do you think about the reasons why your favorite RDBMS sucks? Last few months I was doing this quite often and yes, my favorite RDBMS is MySQL. The reason why I was thinking so because one of my recent tasks at Scribd was fixing scalability problems in documents browsing.

The problem with browsing was pretty simple to describe and as hard to fix - we have large data set which consists of a few tables with many fields with really bad selectivity (flag fields like is_deleted, is_private, etc; file_type, language_id , category_id and others). As the result of this situation it becomes really hard (if possible at all) to display documents lists like “most popular 1-10 pages PDF documents in Italian language from the category “Business” (of course, non-deleted, …

The video of one of my three sessions, "Optimizing MySQL and InnoDB on Solaris 10 for World's Largest Photo Blogging Community", presented at MySQL Conference & Expo 2008 has been uploaded by Sheeri. I am very thankful to her for doing all the hard work and making it available.

There are a few slides that were edited out of video because of reasons beyond my control. However, you should still be able to enjoy most of the video.

There is one point related to this video that I would like to make: Based on my particular experience I was leading to believe that Solaris 10 Kernel had the same …

I just made my reservations to attend Velocity Conference in Burlingame, CA. Velocity is a new two day conference being organized by O'Reilly. I was happy to learn at Lunch today that one of my good friends from CafeMom will also be attending. Over at Facebook I see Don McAskill has RSVP'd for the event as well.

Jesse Robbins, chair for Velocity conference graciously provided a 20% discount coupon as a comment on my blog post.

The early registration is about to end, but I find it really interesting that many slots still mention TBC (to be confirmed). I would have expected the schedule to …

The slides from my tutorial yesterday are up at develooper.com. 189 slides this year! A handful of people have told me today (and more yesterday obviously) that they learned a lot and got new ideas for how to build things - yay!

I’m considering putting up a version of the slides with sound. Would anyone want a 3 hour quicktime movie of that?

Well, maybe split up into smaller bits, but you get the idea. You wouldn’t get the hand waving, but you would get a bunch more detail, obviously.

I actually had audio recorded, but I haven’t checked how it came out, yet, and I’d have to make a synchronized movie version of the slides (and …

We were using memcache in our application for a long time and it helped a lot to reduce DB servers load on some huge queries. But there was a problem (sometimes called a “dog-pile effect”) - when some cached value was expired and we had a huge traffic, sometimes too many threads in our application were trying to calculate new value to cache it.

For example, if you have some simple but really bad query like

SELECT COUNT(*) FROM some_table WHERE some_flag = X

which could be really slow on a huge tables, and your cache expires, then ALL your clients calling a page with this counter will end up waiting for this counter to be updated. Sometimes there could be tens or even hundreds of such a queries running on your DB killing your server and breaking an entire application (number of …

We got a couple of new servers at Solfo recently which showed me one of the reasons virtualization is so popular now: Servers are too fast!

The "standard issue" CPU is now a quad-2.5GHz CPU, so in each server we have 20 GHZ CPU and 32GB ram (at less than $50 per gigabyte it's too cheap to not just fill it up and be done upgrading). Just a few years ago the CPUs we were getting were "only" dual 2GHz, for ~8GHz CPU per box. That's a big increase!

In each "tier" of the application (app servers, db servers, search servers) our main reason for having more than one or two servers is redundancy / high availability - never lack of CPU and rarely because we need more memory.

Here's from one of our webservers (virtualized with Xen with 6 of the 8 CPUs on the "real" hardware).

The big exception is the MySQL servers where we get constrained by I/O so we need a single …

Once in a while I hear people talking about using thousands or tens of thousands of tables in one MySQL database and often how it “doesn’t work”. There are two things to say to them:

• Are you nuts?!
• Sure it works

I’ll elaborate a little on both …

Why’d you do that? Are you nuts?!

In most cases when extraordinarily many tables are brought up the “correct answer” is: Fix your schema to not duplicate the same table layout for each customer/user/site/…!

Once in a while though there are good reasons to have way too many lots of tables. Even “takes too long to fix the application now” can be a good enough answer sometimes.

My use case was to use the many tables as an extra “index” for a situation that a regular index couldn’t cover. Tens of thousands of tables, here we come.

…

In exactly two months - April 14th - I'm doing a scalability tutorial at the MySQL conference.

It'll, of course, be based on the material I've been throwing around since 2001, but once again it's greatly expanded, extended and improved! It should be lots of fun.

I've yet to hear anyone tell me that they wasted their time with this particular session and I don't think people are just being polite because I'm often told that people find it tremendously useful. So go sign up already. And don't pay any attention to Brian Aker and …