By tim

In building our Research data mart, which includes data on book sales trends, job postings), blog postings, and other data sources, Roger Magoulas has had to deal with a lot of very messy textual data, transforming it into something with enough structure to put it into a database. In this entry, he describes some of the problems, solutions, and the skills that are needed for dealing with unstructured data.

Roger wrote:

• When integrating our research data mart with a legacy sales transaction system, I was asked to help tune a data mart with appx 3mm rows that joined to a few large dimensions; an aggregate query was not completing. I was able to tune the query down to …

I had lunch with a company's IT staff yesterday, and asked them, "Which open source projects/products do you use?" They listed off several projects (Tomcat, etc.), but I had to prompt for the most obvious one: "And Linux? I assume you're running Linux?"

"Linux? Oh, of course, I guess we had forgotten that's open source."

This was a hugely telling remark. For this company (and, frankly, for most of the industry), "open source" is short-hand for many things, and sometimes connotes a cutting-edge, not fully baked product. This is accurate in some respects, and wholly inaccurate in others (no one could sanely call Apache, MySQL, etc. "not fully baked").

In this company's mind, Linux is just there. It works, requires no special justification, etc. It just is.

Give JBoss and MySQL a year or two, and they'll be the same. Along with hordes of other open source projects.

For those of you who, like I, have been …

Every year the user conference gets better and better. I’m not sure if it’s the actual conference or just that I know so many more people than I did the year before so I’m that much more excited to see them all again. I was a speaker this year which is something like being a C celebrity. The attendees at the conference were split into a few very distinct groups. High order geeks, geeks with questions, and business people. The sessions seemed to be setup to appeal to one of these three groups. Of the sessions I attended my favorites were the row based replication session (which inspired Row based replication and application developers) and Timour Katchaounov’s session on new features of the 5.0 optimizer.

It was interesting walking around the conference meeting different people from various backgrounds and …

PHP 5.1.3 was just released, including the latest PDO and PDO drivers. If you're still running older releases of PHP you should probably upgrade. If you don't want to upgrade the PHP core then you'll be pleased to know that I've also released the PHP 5.1.3 versions of PDO, PDO::MYSQL, PDO::PGSQL, PDO::SQLITE and PDO::ODBC to PECL--these are the versions that I've had time to personally test.

These are identical to the code in PHP 5.1.3, with the exception of PDO::ODBC, which includes two additional bug fixes that didn't make it in time.

Enjoy!

On a related note, I noticed that people were downloading PHP 5.1.2 and then trying to build older PDO drivers from PECL into it, and then filing bugs when it didn't work. Don't do that; it really doesn't make sense. As a general rule, PHP 5.1 and higher bundle PDO, so if the version of PHP you are using is newer than the latest PECL package, stick with the version of PDO that …

The subject says it all, if you are heading to the mysql meetup, go to the pizza place up the street.

ROME, Italy -- MySQL AB today announced that the Food and Agriculture Organization of the United Nations (FAO) has selected MySQL as its open source server-side database standard. The agency -- one of the UN's largest -- plans to migrate up to 80% of its technical information systems to MySQL.

Sometimes it’s a terrible idea to do set-based operations in SQL. There are real-world constraints, especially in heavily used OLTP or large databases, that require doing things a tiny bit at a time. In this article I’ll show you how to write a job that can purge data from a huge table without impacting critical processes, filling up the transaction log, or causing deadlocks. I have released a tool that does a fantastic job of archiving and purging MySQL tables, as part of MySQL Toolkit.

Last Sunday I was heading out to a party when I overheard the conversation of "You shouldn't buy an iPod, they have a horrible interface". Now this always holds my interest since I do believe that the interface to my iPod could be a lot better. I stepped into the conversation and heard similar complaints as mine as to why the iPod was the wrong interface, and I also got an earfull on the "Geek Interface".

What is the "Geek Interface"? It is the interface that all geeks want.

It has a thousand buttons.
It will fetch RSS for you.
It reads your email.
It will control your remote vacuum cleaner/indestructible fighting robot.
It probably has an interface to a database (or this could be just because I work for MySQL).

Every geek wants one of these. They want it to be wearable, they want it to look cool.

The "geek …