We’re working with web site preparing for massive growth. To make sure it handles large data sets as part of the process we work on generation test database of significant size as testing your application on table with 1000 rows may well give you very dangerous false sense of security.

One of the process web site had was creating of summary tables which was done by executing some huge group by query, doing some stuff with results and then populating tables. This all worked well for small tables… but not for larger ones.

First problem was PHP script generating the table took 10GB of RAM and was swapping development server which had just 4GB of Ram (and plenty of swap space) like crazy. Why ? Because by default mysql_query uses mysql_store_result C library call and buffers all result set in the process memory. Not good if there are over 50 millions of rows. Note this limit is not controlled by …

As the 2007 Community Advocate of the Year, I’m taking the “MySQL 5 Wishes” meme and changing it a bit. I hope y’all don’t mind:

1) Everyone has a different level of familiarity. The community does well with this when writing articles, for instance cross-referencing older articles, linking to documentation, the MySQL Forge, etc. Not much background information other than “MySQL usage” is assumed.

However, where we fall down is when we aggregate some writings and call it documentation. The worst form of this is a tool that grows organically, from “look, here’s a script!” to a full-blown tool/patch/add-on. Sourceforge stinks for trying to make documentation, so most folks just link to their posts tagged “mytool” or whatever the name is.

Using some marketing skills would be wonderful — make a page for folks who have never seen one post about it. Voila, you get your code going from something that …

After a relaxing holiday in France with some friends I now return to coding! I can feel my tan wearing off already as I enclose myself in my room, fingers tapping away on the keyboard, screen glaring its artificial light onto skin which for a few sweet days had basked in the glow of real sunlight!

But that's for losers.

Anyway, the problem right now is the size of the steps I'm taking. The initial traffic analyzer design was simple enough because it didn't attempt to interpret the MySQL packet payloads in any significant way. It discarded unprintable characters and dumped the rest to the console or to a file. Right now, however, I've assigned myself the task of actually interpreting the data I'm receiving and this is not quick or easy.

The problem lies in the passive nature of the system. As with the design of any complex piece of software, you have to think long and hard about what can go wrong. In the …

The 52nd edition of Log Buffer is up, edited by Dominic Brooks and published on his blog, OraStory. On deck, Daniel Fink. Boy, these Oracle people sure know how to blog! We haven’t heard from a MySQL blogger since Ronald Bradford did Log Buffer #47, from an PostgreSQL blogger since Robert Treat’s #27, [...]

I'm looking for SQL testing harnesses, or any sort of testing
harnesses that are frameworks. I've got a need for one for MySQL.

Open source required :)

Anyone have any idea on what the current hot project is in this area?

Something like Siege
or possibly something with a big more infrastructure then Pulse.

I spent a half-hour this morning talking with Lance Walter, VP of Marketing for Pentaho, a leading open source Business Intelligence vendor. I wanted to see if Pentaho's experience in the market matches up with what other open source application companies are seeing.

Indeed. The good news of open source goes well beyond any one particular vendor.

Question: I hear good things about Pentaho all the time. Can you give me a high-level update?

Sure. First off, you may have seen the news that we did a big competitive replacement of Crystal Reports at Boyne Resorts the largest family run four-season resort company in North America. We also just closed a deal …

I love Marten Mickos, and it is quotations like this in a Computerworld interview that reinforce my respect for him. Asked whether MySQL would ever go partially proprietary in order to get a higher download-to-sale conversion rate, Marten replied:

We've had that debate many times. I think we might win a few new customers, but we would lose 2 million users. We're not ready for that kind of compromise. We also look at other companies who have built closed-source products on top of open-source ones. They don't seem successful.

I think we are well protected against predatory behavior by our competitors. When you download MySQL, it's just GPL code. But the code is owned by us. We have the copyright, we determine what goes into it, we put in the bug fixes. There's nobody else with that core skill.

Secondly, …

One interesting solution to scalability and HA is to implement a circular master-master replication setup directly on the frontend servers. Obviously there is a reasonable limit as to how many servers you can have in such a ring, since the lag with which changes propagate will increase linearly as you add more servers to the ring. However according to an article by Giuseppe a 10 server ring is reasonable. Especially if you make your sessions sticky on a per frontend server basis, you actually have an elegant solution against the old issue of replication lag, where a user does not see changes he has made in subsequent requests, since his changes were written to a remote master server which have not propagated to the slave he is reading in the subsequent request. Latency should also improve as you do not have to open up a remote connection in the …

So, how do you get more free time? Simple, you quit your job. I did. Yes, seriously.
Well, I was employee#25 at MySQL AB (from August 2001), that's nearly 6 years - I reckon I've done my time ;-)

What's next? That's not really the point, I haven't taken another job elsewhere.
For starters I'm taking a few months off, spending more time with my wife and daughter, getting more exercise, and catching up on some other stuff.

Here's a three interesting interviews with Marten Mickos, Brian Aker and Gary Whizin, all from MySQL.  Marten is the CEO, Brian is Director of Architecture and Gary is Director of our Enterprise tools team. 

It's an interesting juxtaposition to see the range of topics across all three.  Marten talks about the business model, Brian's focused on storage engines and Gary is all about building the right team.  But the common thread across all three is how important open source is to what we're doing at every level. Here's a quick test... see if you can match the quotes below with the right person.  No matter how varied the …