More of information about how we handle database stuff can be found in some of my talks.

Lately I hear people questioning database software choices we made at Wikipedia, and I’d like to point out, that…

Wikipedia database infrastructure needs are remarkably boring.

We have worked a lot on having majority of site workload handled by edge HTTP caches, and some of most database intensive code (our parsing pipeline) is well absorbed by just 160G of memcached arena, residing on our web servers.

Also, major issue with our databases is finding the right balance between storage space (even though text is stored in ‘external store’, which is just set of machines with lots of large slow disks) – we store information about every revision, every link, every edit – and available I/O performance per dollar for that kind of space needed.

As a …

A few months ago I wrote a tool that verified MySQL and Drizzle protocol compatibility, along with testing for all sorts of edge cases. In analyzing protocol command interactions in mysqld, I found that the MySQL server will happily read an infinite amount of data if you exceed the maximum packet size while using a special sequence of protocol packets. The reasoning behind this behavior is so that the server can be polite and flush your data before sending a “max packet exceeded” error message, but perhaps there should be a limit to one’s politeness. What’s more interesting is that you can do this during the client handshake packet without authorization, so anyone could do this to any open MySQL server. The appropriate thing to do here would be to set some maximum limit of data to read and force a connection close when it is reached, otherwise your bandwidth and CPU could be consumed (essentially a DoS attack).

This portion of code …

I recently had a run-in with a very popular PHP ecommerce package which makes me want to voice a recurring mistake I see in how many web applications are architected.

What is that mistake?

The ecommerce package I was working with depended on caching.  Out of the box it couldn't serve 10 pages/second unless I enabled some features which were designed to be "optional" (but clearly they weren't).

I think with great tools like memcached it is easy to get carried away and use it as the mallet for every performance problem, but in many cases it should not be your first choice.  Here is why:

• Caching might not work for all visitors - You look at a page, it loads fast.  But is this the same for every user?  Caching can sometimes be an optimization that makes the average user have a faster experience, but in reality you should be caring more that …

As part of my IPv6 setup I wanted to check what support I could expect for my current hardware. This post relates the results I got by checking on each one.

• Linux PC – IPv6 is supported in Linux
• MacBook – IPv6 is supported in MacOS X
• Windows 7 NetBook – IPv6 is supported in Windows 7
• Draytek Vigor 2820N – Draytek support in Spain confirmed to plans to implement IPv6 on this box, but they have 1 IPv6 aware router so future models should hopefully have support.
• HP Officejet Ro L7680 Printer – this printer has a network connection. A support request on HP’s website generated an automatic message saying my printer was no longer supported.  So no chance …

In order to use IPv6 you probably need to setup things so that your sites can be reached over IPv6. I manage my own DNS server for my domain so decided to make my DNS server reachable over IPv6.  This did not sound too complicated.

This required 2 things:

• Ensure that Bind 9 was configured to receive and respond to queries on this address
• Tell my registrar about the IPv6 address of my domain

I checked my DNS software, Bind (v9), and it did not seem to be responding to DNS requests to the IPv6 address. A quick look around indicated that I had to add an extra block in the options section of named.conf

options {
 directory "/etc/named";

 // required to listen for ipv6 queries
 listen-on-v6 { any; };

....

My registrar is gkg.net and I’ve been using them for a …

We use nginx and its features a lot in Scribd. Many times in the last year we needed some pretty interesting, but not supported feature – we wanted nginx X-Accel-Redirect functionality to work with remote URLs. Out of the box nginx supports this functionality for local URIs only. In this short post I want to explain how did we make nginx serve remote content via X-Accel-Redirect.

First of all, here is why you may need this feature. Let’s imagine you have a file storage on Amazon S3 where you store tons of content. And you have an application where you have some content downloading functionality that you want to be available for logged-in/paying/premium users and/or you want to keep track of downloads your users perform …

One common question I get is how to use partitioning instead of MERGE tables. The process I use involves using stored procedures to create and drop partitions. This article will go over the stored procedures I use; special thanks to Roland Bouman for taking a look and giving great feedback to optimize this process.

First, a simple table, not partitioned (yet):

use test;
DROP TABLE IF EXISTS my_part;
CREATE TABLE IF NOT EXISTS my_part (
  id int NOT NULL,
  creationDate datetime NOT NULL,
  PRIMARY KEY (id,creationDate)
) ENGINE=InnoDB;

In real, life there is more to the table than just id and creationDate. The most important part is that the partitioned field(s) need to be part of the primary key.

Now, add the partition definition. This can be done in the CREATE statement, but I have found that it is easier for me to think …

Welcome to Log Buffer, the weekly roundup of database industry news.

For your reading pleasure this week we have Log Buffer #196:

Charles Hooper blogs about an in-depth investigation on what can cause Oracle to ignore a hint.

Doug Burns reminds his readers that there are only two weeks left to …

Howdy folks,

slides are up for my talk, "Database Scalability Patterns", which I gave this week at OSCon 2010. You can get them from the OSCon page, from slideshare, or just watch it below

Database Scalability PatternsView more presentations from Robert Treat.

MyTOP is a console-based (non-gui) tool for monitoring the threads and overall performance of a MySQL created by Jeremy Zawodny.

For months now I have been adding updates to mytop. When I started using version 1.6 it worked but didn’t return some data fields. After fixing these bugs I began to ideas for improvements. Here is a quick list of what I have done.

• New ‘!’ command to force fix replication errors.
• Display rows sorted per second.
• Added new ‘Cmd’ column to display the state of the query along with the statement.
• New ‘M’ command to change the [Mode] to ‘status’.  This was in the code but covered up by another command.

If you find any bugs or would like to see your own updates added …