|Showing entries 1 to 12|
Dear Kettle users,
Most of you usually use a data integration engine to process data in a batch-oriented way. Pentaho Data Integration (Kettle) is typically deployed to run monthly, nightly, hourly workloads. Sometimes folks run micro-batches of work every minute or so. However, it’s lesser known that our beloved transformation engine can also be used to stream data indefinitely (never ending) from a source to a target. This sort of data integration is sometimes referred to as being “streaming“, “real-time“, “near real-time“, “continuous” and so on. Typical examples of situations where you have a never-ending supply of data that needs to be processed the instance it becomes available are JMS (Java Message Service), RDBMS log sniffing, on-line fraud[Read more...]
With OpenSQL Camp and FrOSCon being over for almost a week now, it's time to come up with a short summary. I traveled home on Monday morning and then took Tuesday off, so I had some catching up to do...
As for the past years, FrOSCon rocked again! According to the closing keynote, they had around 1.500 (unique) visitors and I had a great time there. I really enjoyed meeting all the old and new faces of the various Open Source communities. The lineup of speakers was excellent, Jon "maddog" Hall's keynote about "Free and Open Source Software in the Developing World" was quite insightful and inspiring.
Most of the time I was busy with[Read more...]
Greetings from Sankt Augustin, Germany! I've arrived by train today and just returned from the FrOSCon venue, which will start tomorrow. The organizers are still busy with the preparations, but things already seem to be in good shape.
It was a mild and sunny evening today. Hopefully it will be the same tomorrow again, so we can enjoy a relaxed BBQ outside! The social event at FrOSCon is always a nice opportunity to meet and talk with fellow open source enthusiasts, users and developers.
And finally some good news for those of you who can't make it to FrOSCon this year: there will be live video streams from selected lecture rooms! So you will be able to attend the OpenSQL Camp sessions virtually - just head over to http://live.froscon.org/ and select room "HS6". It'll be interesting to see how this will work out.
Some of you may have noticed that blob streaming has been merged into the main Drizzle tree recently. There are a few hooks inside the Drizzle kernel that PBMS uses, and everything else is just in the plug in.
For those not familiar with PBMS it does two things: provide a place (not in the table) for BLOBs to be stored (locally on disk or even out to S3) and provide a HTTP interface to get and store BLOBs.
This means you can do really neat things such as have your BLOBs replicated, consistent and all those nice databasey things as well as easily access them in a scalable way (everybody knows how to cache HTTP).
This is a great addition to the AlsoSQL arsenal of Drizzle. I’m looking forward to it advancing and being adopted (now much easier that it’s in the main repository)[Read more...]
Being a serious Perl addict since 20 years, I of course start missing a lot of nice functions for manipulating strings, and the most immediate one is join, so I wrote a C++ IOStream manipulator to join the elements of an arbitrary sequence and output them to an
In this case, since the I/O Manipulator takes arguments, it has to be written as a[Read more...]
CREATE TABLE notes_tab (And assume we have a file called blob_eg.txt with the contents "This is a BLOB Streaming upload test".
n_id int PRIMARY KEY,
After reporting a minor bug, I managed to build and run it with some sample scripts. I'm using Kubuntu Feisty, and had some initial problems, but it was actually pretty straightforward. I'll repeat the steps anyway, in case anybody else have problems.
svn co http://svn.mysql.com/svnpublic/mysql-proxy/ mysql-proxy
Since I work with replication, these things got me thinking on what the impact is for replication and how it affects usability, efficiency, and scale-out. Being a RESTful guy, I started thinking about URIs both when[Read more...]
|Showing entries 1 to 12|