Following a link from the High Scalability blog, I found this really great article about scalability practices, as told by Randy Shoup at eBay. Randy is very good at explaining some of the more technical aspects in more or less plain English, and it even helped me find some wording I was looking for to help me explain the notion (and benefits) of functional partitioning. He also covers ideas that apply directly to your application code, your database architecture (including a little insight into their sharding strategy), and more. Even more about eBay’s architecture can be found here.

Startups are pretty fascinating. I work for a startup, and one of my good friends works for another startup. I’ve also worked for 2 other startups, one during the first “bubble”, and another one a few years later. Oh my, how the world of web startups has changed in that time!

1999: You must have funding

The first startup I was ever involved in was a web startup. It was an online retailer. They were starting from nothing. My friend (a former coworker from an earlier job) had saved for years to get this idea off the ground. He was able to get a few servers, some PCs for the developers he hired, and he got the cheapest office space in all of NYC (but it still managed to be a really cool space, in a way that only NYC can pull off), and he hosted every single service required to run the web site in-house. If I recall correctly, he had a web and database server on one machine, and I believe the primary DNS server was on an old …

If you’re an advanced MySQL person, you might already know these, in which case, please read anyway, because I still have some questions. On the other hand, f you’re someone who launched an application without a lot of database background, thinking “MySQL Just Works”, you’ll eventually figure out that it doesn’t, and in that case, maybe these tips will be of some use. Note that I’m speaking specifically about InnoDB and MyISAM, since this is where most of my experience is. Feel free to add more to this content in the comment area. 

InnoDB vs. MyISAM

Which one to use really depends on the application, how you’re deploying MySQL, your plans for growth, and several other things. The very high-level general rule you’ll see touted on the internet is “lots of reads, use MyISAM; lots of writes, use InnoDB”, but this is really an oversimplification. Know your application, and know your data.  If all of …

How do you configure python for use with mysql? You require the MySQLdb module for connecting to mysql server using the python code. This module is used for firing queries to the database server and handling of result sets from python code.

First of all, check if the module is available or not :

jayant@jayantbox:~$ python Python 2.5.2 (r252:60911, Apr 21 2008, 11:17:30) [GCC 4.2.3 (Ubuntu 4.2.3

Since the recovery from my recent outage, I’ve noticed that none of the normal feed sites where my posts normally show up caught the last post, so this is a test post to see what’s going on, if it was a temporary glitch, or what.

If you didn’t see the post linked above, please read it if you’re happy with your web host. I’m looking for a new one :-/

Ok, this blog is currently hosted by 1and1.com, and I think that needs to change, which is sad, because up until recently, I’ve been pretty happy with the performance. However, I recently had an issue, and some things came to light about my package that I wasn’t aware of.

First of all, the maximum time a process can run is about 6 seconds. Second of all, the maximum number of processes you can have running at once is 12. When I asked if this was a limitation of my *package*, the answer I got was that it was a limitation of all shared hosting accounts.

This blog started throwing 500 errors some time yesterday. I called support late last night and some lady said that she was running a tool that should fix my issue, and to call back in 30 minutes if the problem persists. Well, I’m sick as a dog, and it was late, so I went to bed. This morning, the problem was still there. I called support 3 times today, got cut off twice, and …

Google has just announced their alternative to Amazon’s s3 called ‘App Engine’. 

I think that if this is successful it will provide a shift in some of the basic web development economics and practices, even more than Amazon’s s3 has.

why?
- Small hosting providers (ones that offer a shell account for $12/month) will be marginalized. why pay for something when you get it for free?

- M&A. It will create a 3rd platform to develop on. you currently have LAMP and Windows. The google app engine provides a 3rd. The major difference is you can’t buy it. If we acquire a company who runs on this platform we have 2 choices. continue paying google for the infrastructure, or redevelop it onto LAMP. of course this suits google as their integration costs are lessened. Google might provide a ‘open source’ version of their infrastructure.. but I doubt it.

- …

While recently migrating Tschitschereengreen.com from Symfony to Django plus changing the database backend from MySQL to PostgreSQL, there were mainly two tasks more time-consuming than I’ve had thought beforehand:

SQL dump

The old database used a latin1 encoding for the database fields and utf-8 as the server and client connection encoding. With these settings, even trying to get a correctly encoded database dump from phpMyAdmin is a bad idea.

Using mysqldump with an explicitly specified character-set is much better:

mysqldump ? ?default-character-set=latin1 …

I have my own idea for a Summer of Code Project; an implementation of mysqldump, but in Python. I see it as a good choice because the spec is already there for you - you just have to make it work. There's also a lot of concepts to learn in writing it (consistent snapshots, dealing with a potential combination of character sets).

My intention of proposing this isn't as a replacement to the existing mysqldump, but rather as a community maintained alternative. There are some features missing in mysqldump that I could add myself if it was in a language I am friendlier with. Two of these would be --slave-data, and parallel dumping[1], but that is not a complete list.

[1] Yes, I'm aware of maatkit. I want to use the mysqldump interface though.

It's been nearly a year since the last post, so you might naturally wonder if I am dead or stopping development of MySQLdb. Actually I've been sick for the last year and a half or so, and hadn't really been motivated enough to do anything. Here's what's been going on:

John Eikenberry has taken over development of ZMySQLDA, since I pretty much don't do anything with Zope these days. He's made a couple of releases on the way to a 3.0 release. As far as I know, ZMySQLDA is still only useful with Zope 2 as Zope 3 has a different architecture and comes with MySQL support directly.

Monty Taylor from MySQL AB has volunteered to help out on MySQLdb. I believe the way this is going to work out is he's going to be doing maintenance on the 1.2 branch, and get some minor bug fixes out there. In addition, he has a good start on a native (i.e. written in Python) …