Pretty Sick Slick

The last week I was under the weather and a year ago that would have meant that development of Pentaho Data Integration (PDI) would pretty much stop. These days I’m happy to say that this is absolutely not true anymore. In fact, hundreds of commits where done in the last week.

MySQL bulk load

To pick one example, Samatar Hassan added a job entry that allows you to configure a MySQL Bulk load job entry:

This job entry loads data as fast as possible into a MySQL database by using the LOAD DATA SQL command. It’s not as flexible as the Text File Input step, but it sure is fast. In certain cases, it might actually be up to ten times as fast. In short: …

An interesting but ambitious idea would be to refactor the mysqld code in a major way: Not just a minor tweek but would take a lot of the concepts, libraries and design of mysqld and put them together in a cleaner way with the benefit of hindsight, wishlists and the goals of mysqld. Very possible to do but not at all easy.

I somehow introduced a syntax error into MySQL Table Checksum in the 1.0.1 release and didn’t notice till just now. I’ve replaced the file in the Sourceforge release, but it may take a few hours for it to propagate to all mirrors. My apologies.

I’ve been designing an algorithm to resolve data differences between MySQL tables, specifically so I can ‘patch’ a replication replica that has gotten slightly out of sync without completely re-initializing it. I intend to create a tool that can identify which rows are different and bring them into sync. I would like your thoughts on this. Background and requirements I see this as the next step in my recent series of posts on MySQL tools and techniques to keep replication running reliably and smoothly.

Matt Asay proposes the following definition to the answer what consititudes an "open source company" that I blogged about yesterday: "An open source company is one that, as its core revenue-generating business, actively produces, distributes, and sells (or sells services around) software under an OSI-approved license."

I see a lot of merit in this definition. However it does shut out companies like EnterpriseDB that do proprietary extensions while feeding a lot of code back to the open source parent. Of course you can point to the fact that the product they sell is not open source.

This however is the only code based business model around BSD projects. Without picking favorites, I personally do appreciate the fact that BSD style projects produce an ecosystem that …

I have never been that interested in strict definitions of concepts and this never-ending discussion of what is and isn't open source is an interesting philosophical argument but not an entirely practical one.

There is no question that the GPL offers many benefits to companies/people who create products and release them under that license. But it's not for everyone, and there is often a necessary evolution (as Matt and SOG note) where the benefit of adoption under MPL or Apache or whatever are potentially more important than what the GPL affords.

Matt is excited about going to GPL and I back Alfresco's choice 100%. I think it makes sense for the business and where their product lives in the enterprise stack. But GPL doesn't make sense for everything (not that I think Matt believes it does, but I have had a number of people …

Right now I run into a quite common problem (see this thread on Devshed forums), people trying to load date and time related values from a flat file into MySQL and all they are getting is mangled data.
The reasons are two, first of all they are not running in a "strict" server SQL mode, which should spit errors instead of insering meaningless values (this poll shows some interest in it), the second one is related to the server datetime settings ...
As a workaround people are often tempted to mess with the input file, which is impractical when reloading large dump files, here is a quick glance at my solution, hope this helps but the usual caveats are …

SHOW PROFILES is finally available! It has been a long struggle, but finally this important community patch by Jeremy Cole has been integrated into a publicly distributed MySQL source tree (thanks to Chad Miller excellent integration work).
What's SHOW PROFILES? Is a feature to let you peek inside MySQL internals, which tell you what your queries were doing during their execution.
This feature is also remarkable because it makes a significant difference between Community and Enterprise releases.
The binaries will be available soon, but in the meantime, you can build the source code and give it a try.
BuildingTo build from source, follow the instructions at Installing from the Development Source Tree using the …

I wrote previously about looking for a more powerful search solution, and I mentioned that Xapian wasn’t quite so convenient in indexing my data. I then chose to experiment with sphinx a little more, and proceeded to create a number of search engines and indexed a number of data sources in order to decide which direction to go. Unfortunately, while sphinx was convenient and still provides an excellent backend for basic search indexes, I’m revisiting Xapian once again based on it’s more-than-anticipated flexibility. I was brief in my explanation of Xapian however, and didn’t mention some of the more important and powerful aspects of it.

Xapian provides an API

Xapian is primarily an API for search indexing/data retrieval. They do provide a handy utility called Omega (available here) for indexing static pages and a plethora of other mime-types. However, I’m in …