Closing keynote of the 2007 MySQL Conference is Pasha Sadri from Yahoo! talking about pipes.

Pasha starts with a story about looking for an apartment near a park (this happened to him a few years ago). It was very tedious to go looking on Craigslist every hour clicking on every apartment and looking to see if the apartment was near a park.

Made a 50-line Perl script to merge database between Craigslist and Yahoo! local to get data sets. This turned into Yahoo! pipes which provides a visual editor to take data sources and create a set of controls over that data source and output the data in a certain way.

If you haven't seen pipes it is hard to describe, but very cool.

For design, pipes relied heavily on the idea of Unix pipes. The goal to …

Listening to Dathan Pattishall talk about flickr at the 2007 MySQL User Conference. Dathan worked at AuctionWatch in 1999, then in 2003 worked at Friendster, now at Flickr.

Flickr was unable to keep up with demand. Replication was not working, too much slave lag. They came up with some requirements. Needed to support a write intensive site with multiple masters. There should be no single point of failures. Need to have real-time maintenance and be able to serve pages extremely fast.

At AuctionWatch they put folks on separate boxes. At Friendster they had an algorithm that spread folks across many machines. At Flickr they use federation, which is made up of shards, a global ring, and logic to connect shards.

Shards are a slice of a main database. Flickr uses active master-master replication but externalizing the auto increment process.

The global ring is a …

This morning as I was driving over to the MySQL conference I thought about something Guy Kawasaki said yesterday. He said that one indicator of a sketchy VC-seeking group is if their proposal includes a large chunk of money for database licenses ("a million dollars"). The idea was that if you're building a new product there's a better chance of funding if you use something like MySQL and don't have to spend a lot of the investment on your database.

Flash back to mid to late 90s. From my limited experience this was opposite. The company I worked for was using Oracle for one reason, because it made them look serious when talking to funders. It meant that we were serious about scalability and poised to handle the heavy click loads. (The ironic thing is that we weren't. The application fell flat on it's face during even minimal …

With palm's recent announcement that it will have a Linux-based Treo by year end and all the hoopla around the upcoming apple OS X palmtop, this seems like a really good time to be in the FOSS world.

I haven't seen the specs of either OS yet, of course. But I can guess at the hardware that they will both debut on, and I'm fairly confident that both devices will be more powerful than the first Linux box I ever worked on. I'm hoping that the end result will be a full enough set of posix libraries to move the LAMP stack with little trouble. "Little trouble" being a relative term, of course.

read more

I've heard a lot about Ruby on Rails, and finally had occasion to try to use it. This tutorial was very well laid out, and entirely true - maybe a half dozen commands and a couple dozen lines of code later, there was a nice set of "scaffolds" for browsing & editing my data model.

It didn't "guess" my data model, but I prefer to tell a framework how to handle relationships. Telling rails was as simple as adding a handfull of commands like "has_one :parent", "belongs_to :category". And then you get drop-downs for foreign key values and column sorting virtually free.

read more

In August 2006 I made a post about the MySQL blackhole engine and its possible use for replication scenarios.

Back then the primary focus was to reduce the amount of data being transferred between master and possible many slaves during normal business. This speeds up processing on the slaves, because they have considerably less data to first store to their relay logs and then immediately discard again.

What still takes a lot of time is the initial setup of a slave system. This is because there is a potentially large SQL based dump file to be executed on maybe relatively low-end systems (e. g. Celeron CPUs and single IDE hard drives). While this can be made to work reliably with a set of scripts to automate the process of downloading a zipped SQL file from a local server and running it against the clients' databases, it …

In August 2006 I made a post about the MySQL blackhole engine and its possible use for replication scenarios.

Back then the primary focus was to reduce the amount of data being transferred between master and possible many slaves during normal business. This speeds up processing on the slaves, because they have considerably less data to first store to their relay logs and then immediately discard again.

What still takes a lot of time is the initial setup of a slave system. This is because there is a potentially large SQL based dump file to be executed on maybe relatively low-end systems (e. g. Celeron CPUs and single IDE hard drives). While this can be made to work reliably with a set of scripts to automate the process of downloading a zipped SQL file from a local server and running it against the clients' databases, it …

In a previous post I mentioned about patches needed to accomadate the new DST changes for Sql Server. (I know, this is one of Dannyman’s favorite subjects ;))

For MySQL, obviously you need to patch the host server first. Then you need to find out if MySQL needs separate work. Here is a note I gathered. Let me know if I am wrong on this or if there is a better way.

1. Get into mysql and do \s to find out the version of your MySQL.

If it is prior to 4.1.3, don’t worry about it.

Else

2. Do select @@global.time_zone;

If the result is SYSTEM, don’t worry about it.

Else

You need to load time zone info, usually at /usr/share/zoneinfo into your mysql database, by running something like:

# …

I just thought I might link to an article on RegDeveloper called MySQL is the company's SQL now...

Go read it yourself, it is rather short, but interesting nevertheless. I especially like the sentence at the bottom: It remains to be seen whether the company fully understands, and can cope with, the culture shock it is about to suffer; as callers change from 'friends' to 'customers'.

This is what I have been thinking about too, when they started releasing enterprise builds more often than community builds (binary builds that is). In the past MySQL always suggested to use the binaries provided by them to rule out any effects that may be introduced by a 3rd party build process or patches. That made much sense to me.

However now they tell you that you can check the source …

I just thought I might link to an article on RegDeveloper called MySQL is the company's SQL now...

Go read it yourself, it is rather short, but interesting nevertheless. I especially like the sentence at the bottom: It remains to be seen whether the company fully understands, and can cope with, the culture shock it is about to suffer; as callers change from 'friends' to 'customers'.

This is what I have been thinking about too, when they started releasing enterprise builds more often than community builds (binary builds that is). In the past MySQL always suggested to use the binaries provided by them to rule out any effects that may be introduced by a 3rd party build process or patches. That made much sense to me.

However now they tell you that you can check the source …