Dear Kettle users,

Most of you usually use a data integration engine to process data in a batch-oriented way.  Pentaho Data Integration (Kettle) is typically deployed to run monthly, nightly, hourly workloads.  Sometimes folks run micro-batches of work every minute or so.  However, it’s lesser known that our beloved transformation engine can also be used to stream data indefinitely (never ending) from a source to a target.  This sort of data integration is sometimes referred to as being “streaming“, “real-time“, “near real-time“, “continuous” and so on.  Typical examples of situations where you have a never-ending supply of data that needs to be processed the instance it becomes available are JMS (Java Message Service), RDBMS log sniffing, on-line fraud analyses, web or application …

I'm about to take a week off from my new gig as COO at Zendesk and it got me reflecting on the company and my decision to join.  I stayed with MySQL through the Sun acquisition and left when Oracle acquired Sun.  Although I have a lot of respect for Oracle, it seemed to me the only interesting jobs would be those that report directly to Larry Ellison.  So I took some time off to travel, worked as an EIR at Scale Ventures for a few months and began thinking about what I wanted to do next.

I turned down offers from companies and investors to come in and "repeat the MySQL playbook" in Big Data or NoSQL or apps or whatever.  I think Open Source can be a fantastic …

I’ve just sent an email to the Maatkit discussion list to announce a planned change to how Maatkit (and Aspersa) are developed. In short, Percona plans to create a Percona Toolkit of MySQL-related utilities, as a fork of Maatkit and Aspersa. I’m very happy about this change, and I welcome your responses to that thread on the discussion list.

Related posts:

1. Aspersa, a new opensource toolkit
2. Four companies to sponsor Maatkit development
3. How …

The Dallas / Fort Worth Unix User Group asked me to present on Open Source BI tools on July 7th. They meet 7PM at IBM Innovation Center at 13800 Diplomat Drive (see website for details) and will serve pizza! All are welcome, see you there!

…

By far, the most popular way for PDI users to load data into LucidDB is to use the PDI Streaming Loader. The streaming loader is a native PDI step that:

• Enables high performance loading, directly over the network without the need for intermediate IO and shipping of data files.
• Lets users choose more interesting (from a DW perspective) loading type into tables. In particular, in addition to simple INSERTs it allows for MERGE (aka UPSERT) and also UPDATE. All done, in the same, bulk loader.
• Enables the metadata for the load to be managed, scheduled, and run in PDI.

However, we’ve had some known issues. In fact, until PDI 4.2 GA and LucidDB 0.9.4 GA it’s pretty problematic unless you run through the process of patching LucidDB outlined on this page: …

Our last LucidDB release was now, just more than 12 months ago on June 16, 2010. We were really really trying to beat the 1 year mark for our 0.9.4 release but we just couldn’t. A tenet of good, open source development is early and often and we need to do better. Since the 0.9.3 release we’ve:

• Built out an entire Web Services infrastructure
• Developed a wicked cool Admin user interface
• Developed cool connectors to Hive, CouchDB
• Built a whole ton of extensions (auto indexing, DDL generation, improved load routines)
• Scriptable …

Update

Since this article was written, HPCC has undergone a number of significant changes and updates. This addresses some of the critique voiced in this blog post, such as the license (updated from AGPL to Apache 2.0) and integration with other tools. For more information, refer to the comments placed by Flavio Villanustre and Azana Baksh.

The original article can be read unaltered below:

Yesterday I noticed this tweet by Andrei Savu: . This prompted me to read the related GigaOM article and then check out the HPCC Systems …

Today, more and more proprietary software vendors are choosing to go Open Source. Doing this enables them to leverage the community benefits of Open Source, shorten the sales cycle, and gain a competitive advantage over other proprietary products.

However, for those firms considering a switch to Open Source, there are some hard decisions to make with regard to their product architecture. Should they provide only a single Open Source product, and earn revenue from add-on services like support and consulting (RedHat)? Or should they adopt the Open Core model, offering their product under both Open Source and proprietary licenses (MySQL)? Or …

1. OMG Text -- a plugin for CSS framework Compass for directional text shadows. (via David Kaneda)
2. Build a Cheap Bitcoin Mine -- some day it will be revealed that the act of generating a bitcoin token is helping the Russian mafia to crack nuclear missile launch codes and Afghan druglords built the Bitcoin system to destabilize the US dollar.
3. Polycode -- a free, open-source, cross-platform framework for creative code. You can use it as a C++ API or as a standalone scripting language to get easy and simple access to accelerated 2D and 3D graphics, hardware shaders, sound and network …

“Our experience from PNUTS also tells that these systems are hard to build: performance, but also scaleout, elasticity, failure handling, replication. You can’t afford to take any of these for granted when choosing a system. We wanted to find a way to call these out.” – Adam Silberstein and Raghu Ramakrishnan, Yahoo! Research. ___________________________________ A [...]