We’ve been doing some tests with medium sized data sets lately.  We extracted around half a year of data (514M rows) from a warehouse where we’re doing a database partitioning and clustering test.
Below is an example where we copy +500M rows from one database to another one that is partitioned. (MS SQL Server to MySQL 5.1).  This is done using the following transformation.  In stead of just using one partitioned writer, we used 3 to speed up the process. (lowers latency).

Copying 500M rows is just as easy as copying a thousand, it just takes a little longer…

It would have completed the task a lot faster if we wouldn’t have been copying to a single table on DB4 at the same time. (yep, again 500M rows) This slowed down the transformation to the maximum speed of DB4.  That being said, if you still had any doubt about Pentaho Data Integration being able to copy large volumes of data, …

The presentation How to Be a Consultant has been posted to the Midlands PASS Chapter website. The presentation was given by Midlands PASS Vice President and Treasurer, Ben DeBow, of CounterLogic.com. There's a lot of great information on succeeding as an independent consultant from one who has been doing just that for a while now. It is not specific to any technology but rather focuses on the business side of being a consultant.


Technorati Tags: SQL Server | Microsoft SQL Server | …

Dirk Riehle has posted a paper on the The Economic Motivation of Open Source Software: Stakeholder Perspectives.

Let us take this article apart.


..should use open source software to grow their user communities and build an ecosystem around their products and services..


A good product is a good product. If we look at this from the stand point of the web has Livejournal been any more successful then Google because it open sourced its framework? I don't think so. Both exist because they delivered value. I've never been sure about why Brad open sourced LJ in the beginning but I don't believe it was because he thought the masses would contribute in mass to its evolution (and I speak as someone who has sent in patches and have had them accepted in the code). Open source had little to do …

What a perfect day. I'm in London today, and went to the Arsenal vs. Bolton match with Mark Shuttleworth, founder of Ubuntu. Mark isn't a big football fan, but he indulged my Arsenal fixation and even treated me like a rational human being, which I decidedly am not when it comes to football. Arsenal won 2-1. All is right in the universe.

But where the day got really interesting was over dinner at Tamarind, one of my favorite restaurants anywhere. I've long respected Mark, but over dinner I found him engaging both as a person and as a technology visionary. Here is a very real, good person who has happened to be phenomenally successful as an entrepreneur, without letting it turn him into an obnoxious Muppet.

Some of the insights I gleaned throughout the course of our dinner:

• Mark has the potential to fundamentally change the …

I would not normally blog this, but I thought it was interesting to see this appear in the MySQL forums:


We're looking for a consultant to help us write a custom storage engine that distributes data amongst many machines but abstracts this from clients who still issue regular MySQL queries.

You can look at it as a stripped down version of MySQL Cluster or a beefed up version of the FEDERATED storage engine.

Candidates must have strong knowledge of clustering and MySQL internals.



http://forums.mysql.com/read.php?8,147823

When I was in Japan last week I got to meet another one of the Storage Engine vendors. Who had joined their company to do this work?

One of my X interns.

There is a good and growing eco-system forming around this work. On Friday of …

InformationWeek's Charlie Babcock has stumbled on a great truth--the only way to build a business is to have customers. This is something that Matt and I have stressed a number of times. Revenue solves all problems :>

Says Rod Johnson of Spring/Interface21:

Instead, open source products "should be funded by their customers," the only way to ensure they that have a future, Johnson says. It's no accident that the best known enterprise open source programs, such as the MySQL database and the JBoss application server, have companies with paying customers behind them.

Link: What Makes An Open Source Project Successful?

…

I was asked in comments "nice, but its only read only right?".

Not true at all!

For example, lets create this table:


mysql> CREATE TABLE `website` (
-> `filename` varchar(125) NOT NULL DEFAULT '',
-> `contents` text,
-> PRIMARY KEY (`filename`)
-> ) ENGINE=HTTP DEFAULT CHARSET=latin1 CONNECTION="http://localhost/"
-> ;
Query OK, 0 rows affected (0.00 sec)


Now, lets insert a page:


mysql> insert into website VALUES ("index.txt", "This is a document on a website");
Query OK, 1 row affected (30.01 sec)


Read the page from curl:

[brian@zim ~]$ curl http://127.0.0.1/index.txt
This is a document on a website



…

From IM:
"So can anything be done with XML?"

Create a table:

CREATE TABLE `krow` (
`a` varchar(125) NOT NULL DEFAULT '',
`b` text,
PRIMARY KEY (`a`)
) ENGINE=HTTP DEFAULT CHARSET=latin1 CONNECTION='http://krow.livejournal.com/data/'

Now... can I extract anything?

mysql> select EXTRACTVALUE(b, '/rss/channel/title') from krow WHERE a="rss";
+--------------------------------------------------+
| EXTRACTVALUE(b, '/rss/channel/title') |
+--------------------------------------------------+
| Brian "Krow" Aker's Idle Thoughts |
+--------------------------------------------------+
1 row in set (1.29 sec)

Basically I use the new XML methods in 5.1 to extract out realtime data from 5.1.

I publicly committed mod_methods:
…

Tim O'Reilly, ever a few steps ahead of the rest of us, has some thoughtful musings on what's behind Google's new 411 service. Tim doesn't cast this as an open source move, but rather a Web 2.0 move designed to build up a treasure trove of data against which to build better speech recognition:

But it also seems to me that there's a hidden story here about the speech recognition itself. I was talking recently to Eckart Walther of Yahoo!, who used to be at Tellme, and he pointed out that speech recognition took a huge leap in capability when automated speech recognition started being used for directory assistance. All of a sudden, there were millions of voices, millions of accents to train speech recognition systems on, and much less need for the individual user to train the system.

This is reminiscent of a comment that Peter Norvig, Director …

I'm finishing up my presentation for the upcoming COSS.fi event next week (4/19/07), and was exploring an idea - how do you compete when the code is open? - when two things happened in quick succession.

One, I was reading the Bible and came across this interesting verse from Genesis:

...Cursed is the ground for thy sake; in sorrow shalt thou eat of it all the days of thy life;

Thorns also and thistles shall it bring forth to thee; and thou shalt eat the herb of the field;

In the sweat of thy face shalt thou eat bread, till thou return unto the ground.... [Genesis 3:17-19]

In other words, it's good to have to wade through difficulty. That's where growth comes from. There are no magic shortcuts to prosperity.

Second, I read Mike Olson's response to some criticism that I and Stephen O'Grady had heaped upon his …