I’ve always been a big Flickr fan, and user, and love them even more now that their 2GB limit is gone. So this was a most interesting talk, and I think photographers and Flickr users alike will find it interesting what’s behind Flickr. Dathan also has a very interesting blog.

Federation at Flickr: Doing Billions of Queries Per Day, Dathan Pattishall

Database guy - query, performance, troubleshooting, application building, etc. Previously worked at Friendster in 2003.

Master-Slave lag - unable to keep up with the demand that Flickr was having. So they have multiple masters, the had multiple single points of failure (SPOF).

Everything had to be real time. Write intensive, so more than 1 master needed. No more slave lag. Serve pages fast, with many queries. Get rid of the SPOFs and be redundant.

Need ability to make live …

Yesterday my talk about "Scaling the world's largest photo blogging community" went very well and I couldn't be more happier. There were a lot of questions from the audience at the end which made me really happy as it was a clear sign that my presentation wasn't flying over their head :)

Thank you to all those who attended. I will be posting the slides to my talk later tonight (it sucks that blogger doesn't has file upload).

Charles Lee is presenting the session "Using Hibernate to Ease the Migration to Open Source Databases."

Most of the talk so far is Hyperic related.

EJBQL is a poor query language and has bad transaction handling. It also has pessimistic locking and lack of database tools. Entity Beans are proxy objects. Managers interface with entity beans.

Why migrate to Hibernate?

HQL and Criteria based queries supported by Hibernate are fully featured. It does Lazy fetching and doesn't loads objects until you need them. There is a straight forward transaction demarcation. it has secondary cache integration and is a pretty popular framework.

Hibernate POJOs and Transactions: Hibernate POJOs are actually detached from the framework and don't have to inherit the framework API. They are actual objects that can be passed around. Managers look up the POJOs and POJOs travel through the …

Listening to Dathan Pattishall talk about flickr at the 2007 MySQL User Conference. Dathan worked at AuctionWatch in 1999, then in 2003 worked at Friendster, now at Flickr.

Flickr was unable to keep up with demand. Replication was not working, too much slave lag. They came up with some requirements. Needed to support a write intensive site with multiple masters. There should be no single point of failures. Need to have real-time maintenance and be able to serve pages extremely fast.

At AuctionWatch they put folks on separate boxes. At Friendster they had an algorithm that spread folks across many machines. At Flickr they use federation, which is made up of shards, a global ring, and logic to connect shards.

Shards are a slice of a main database. Flickr uses active master-master replication but externalizing the auto increment process.

The global ring is a …

It seems Google couldn’t wait to have their code evaluated for merging into the main source tree, and decided to release it to the general public as patches to 4.0.26 for the community to evaluate.

What have we added and enhanced? The high availability features include support for semi-synchronous replication, mirroring the binlog from a master to [...]

Came across this post this morning:

http://www.futhark.ch/index.php?action=comment&category=mysql&id=150

The Title is "Are you prepared for the attack of the MySQL worm?"

With all of the real security issues that exist on the Internet I dislike seeing people dream this stuff up.

The author mentions have a piece of SQL inject itself into an event (which you don't need, this could be done with a stored procedure and a sleep() call), and a Federated table.

Let us talk about why this is a non issue:

Databases do not live on the Internet. They live behind firewalls where access is strictly controlled. Databases are not mail servers, web servers, or finger servers. People do not allow others to access their databases directly.

…

Yes, you heard it here first. Monty is the world’s first MySQL Fellow. For outstanding work, being the co-Founder of MySQL AB, and for continuing excellent work. Standing ovation, might I add.

Some quick comments from the clash of the DB egos:
Heikki Tuuri - he worked for Solid, then went on to university, and learned everything about databases. He attributes it to stubbornness. Monty said he should make InnoDB open source, and he obeyed!

Mikael Ronstrom - “entrepreneurial ego in a large company”. He had a vision of what he wanted to do. Guess he was also a bit stubborn…

Jim Starkey had the most interesting hat ;-) And he probably made a couple of bucks from it…
“Does the United States still have anything to say in databases?”

Starting over is important, because you learn new things… Jim Starkey has written about 5 databases in his lifetime.

Nitro: really …

Day 3 Photos

Okay - so it’s the morning - and the battel of the database egos session has just finished. Some photos up already!

Bdale is just about to start now… maybe some photos soon :)

Tony (my fiancee) says it best — he’s an amazing writer. For some history, I work for a dating site that caters to gay men:

MySQL is database software. Whenever a computer program or system (like, say, a gay man’s online dating service) needs to randomly access, store, and keep track of a bunch of data “stuff” (like, say, a bunch of fruits, their personal information and, uh, “vital statistics”), it puts it into and maintains a database.

MySQL (http://www.mysql.com) is a very popular, very good database system. It’s the one Sheeri uses at her job, and my company is currently test-driving a new way to put together web sites that relies on a MySQL database.

An interesting feature of it is that it’s what’s called “open source” software. That means that the community of users is also largely the community of developers. Anyone using the software who …

Curiouser and curiouser....

IBM today announced a technology and marketing partnership that will have IBM selling MySQL's database, among other things:

The agreement calls for IBM and MySQL to develop software that will make MySQL compatible with programs that run IBM's System i line of business computers, including IBM's i5 operating system DB2 database. IBM will also sell MySQL's service products.Just when I thought IBM was getting stodgy and provincial in its views on open source, it does something like this. This is news because:

1. IBM's DB2 competes with MySQL, and

2. MySQL is GPL (though dual-licensed under commercial terms), and IBM has tended to be somewhat Apache-centric, its Linux (and Eclipse - yes, I know that IBM does more than BSD-style licensing) …