I am sitting in Dorion's session about scaling the real time web. Technorati tags grew from 0 to 100 million in a year.

Technorati has about 10TB of core data in MySQL over about 20 machines and they use replication. With replication they add 100TB and 200 machines more. Currently growing at about 1TB per day in total.

A service oriented architecture to separate physical and logical access is used. Technorati uses commodity hardware and Open Source software.

Scaling Technorati Tags : Launched on Janurary 10 of 2005.

Tags are partitioned by entity (tags and posttags). Tags database is separate. Post information in one set of databases and tags information in another set of databases. A caching layer is also present. Overtime Technorati has blended the use of InnoDB and MyISAM based on use.

Currently around 120 million tags in a single table distributed using replication. …

Second keynote of Thursday morning (last day) at MySQL UC 2006 is Mitch Kapor talking about the Wikipedia. Mitch starts with a survey of the crowd. Most people in the crowd use the wikipedia, 15% of the crowd edits the wikipedia.

The typical response from a person that doesn't know about the wikipedia says "The Wikipedia can't possibly work..."

but it does

The mainstream media is very skeptical. Anyone can edit any article at any time. This radical openness is more open than open source projects. There is no review and anyone can post. The radical openness provides opportunities that have never been tried before.

Ideas that are challeneged by the Wikipedia:

• 
  
• someone has to be in charge - even the sysadmins are volunteer and do not keep a …

by Mitch Kapor

Wikipedia uses MySQL as their backend. Wikipedia is known among geeks, but hasn’t quite hit society at large, but probably will soon. What lessons can we learn from Wikipedia? People who hear about the concept of wikipedia say “It can’t possibly work — an encyclopedia written by volunteers, that is completely open?”

But we know it does. It’s increasingly becoming the web page of choice for a wide range of factual topics. [as a side note, I go there when I hear of something and know nothing about it, and I get a good overview]

The mainstream media has been skeptical about Wikipedia, and makes stories about it. People sometimes will put untrue facts in Wikipedia, so mainstream media siezes on it as proof that Wikipedia doesn’t work. But we know that that is the exception, not the rule.

So how and why does it work, if it’s so counter-intuitive? Most people have erroneous assumptions …

Hey, hey, this is well worth a read: MySQL to Offer Linux Bundled with Database as Free Download.

So, the MySQL Forge has been announce and has been live for a while… Even the Forge Wiki has content that isn’t created by me! Ha, the community at work…

And Ubuntu, now supported by MySQL. There’s a relationship there. Mark Shuttleworth gave a great keynote, talked about the relationship with upstreams, the new MySQL relationship they have, and how he was really excited to be here. Thawte I believe, ran on MySQL :-)

If there’s one thing to take away from his talk it’d have to be the fact that he’s also a VC and a lot of people ask him for money. Don’t go to a VC and tell them, “Hey, I’ve got Project Y, and …

“No, you can’t run a Migration BoF, and it won’t make sense without a few of our migration experts,” he exclaimed with a petulant expression.

Well, it turns out the migration BoF was actually quite good at the MySQL UC. It turns out, that everyone attending thought that it was a migration from 3.23 or 4 or 4.1 to MySQL 5. My understanding of this was that we’d actually prefer folk to come in from $database and figure they’d want to migrate to MySQL.

And fair enough, the only experience I’ve had over time was migrating from Postgresql right to MySQL. But the idea of a BoF is to let everyone talk, share experiences, and so on. So a few resources we discussed, and I think were generally useful (and hopefully now, there will be more 5.0 users):

• Upgrading from MySQL 4.1 to 5.0 - from our great …

The first Thursday morning kenote at MySQL Users Conference is Mark Shuttleworth. Mark has done a ton of stuff (Thawte, space flight), but he's here today to talk about Ubuntu Linux, which he founded. Ubuntu is a fairly universal term across several African tribes that translates to "human-ness" the way you look after other people expresses best who you are.

10 years ago MySQL leveled the playing field. Mark was able to get into development as a kid in a garage many years ago because of MySQL and other open source tools.

Mark says you never want to be trying to make or be the next X, you want to pioneer in new ways.

The new deal of free software has different economics. Software is free globaly, provides free updates, and is supported by MANY commercial companies. There's a new culture with community governance that is …

LinuxWorld Magazine (written by Paul Sterne and Nicholas Herring) has an interesting piece on open source venture capital investments. The report focuses on the question: is now the right time to invest in open source? The answer to some is "Of course!!!" But the data might suggest we're in an inflationary period, with smart money waiting for valuations to come down.

From the article:

Starting in the mid '90s, a few brave pioneers like Benchmark invested in an open alternative to proprietary software and made a fortune. By the end of the decade, everyone wanted a piece of the action. A second wave of VCs rushed in at ridiculous valuations and got their clocks cleaned. In 1999 and 2000, over-capitalized, over-valued open source companies burnt through hundreds of millions of dollars. Shame on the dumb money that gives efficient markets a bad name. Then, …

So last night, during a break in the quiz show (where Prokrasti Nation had a good showing, as did the other teams — Recreational Evil, Peeps, and Safe Hex) we bid on the T-shirt that had the signatures of all the speakers at the conference. All the proceeds were to go to the EFF, so it’s a good cause.

They announced it was cash only, so I looked in my wallet. $33. Well, the bidding quickly went over that, and when it reached about $100 they said it didn’t have to be cash only. Around $300 Brian Aker said that they’d give whoever won credits in a new command, SHOW CONTRIBUTORS. Well, when they said that I knew I HAD to have my name in the source code.

I mean, dude, my NAME in the SOURCE CODE!!! But then again, this is an open source application, I could just spend some time and write a patch.

I’ve been saving for my wedding next June (14 months away) so when I bid …

At the MySQL Users Conference I met with Paul McCullough, the developer behind the new PBXT transactional engine.  Paul is a super sharp developer and also a very nice guy and he obliged me by posing in a photo as part of my "storage engine stalker series" of photos including Ken Jacobs of Oracle, Jim Starkey author of Falcon, and Paulo Lubet from Solid. 

It will be interesting to see in the coming months how PBXT and other engines flourish in the community.  We'll do our best to provide more engines, samples, documentation and ideas on the MySQL Forge as well as the …

It's been well mentioned by myself that of late the time I have been able to devote to MySQL has become restricted. Part of this was due to me taking an exciting new job just before Christmas. When I took the decision to close MySQLDevelopment.com down this was in part due to the fact I was using Oracle rather than MySQL and my exposure to real world situations was limited.

However the day after I made the decision and announced it here my boss asked me to take over the running of a MySQL/PHP based website. The good news at least is that it means it might give me something to actually comment on with regard to the blog, even if it doesn't afford me the time to continue with the site.

But it's not just good news for me it's also good news for you out there too, the company I work for are looking to take on some new developers. We are looking ideally for people with PHP and Oracle experience but are willing to train good PHP …