Speaking of conferences, in general, and OpenSQLCamps in particular, there is one a week from now, and I will be speaking! It is organized as a single room track at Froscon, Germany, by Felix Schupp (Blackray/Softmethod) and Volker Oboda (Primebase). The content is mostly a collection of database related talks originally submitted via the main Froscon call for papers. (In other words, unlike many previous camps, the schedule is all set.)

I'm a little excited about this one, because for the first time in my career as speaker I will be giving the keynote. The title of my talk is

How I learned to use SQL and how I learned not to use it

read more

Percona has announced Percona Live MySQL Conference and Expo 2012. Kudos for their vision and entrepreneurship. I have seen comments praising their commitment to the community and their willingness to filling a void. I have to dot a few i's and cross some t's on this matter.

That was not the only game in town.

By the end of June, there were strong clues that O'Reilly was not going to organize a conference. The question of who could fill the void started to pop up. The MySQL Council started exploring the options for a community-driven conference to replace the missing one. The general plan was along the lines of "let's see who is in, and eventually run a conference without the big organizer. If nobody steps up, the IOUG can offer a venue
  [Read more...]

I will be speaking at August Penguin 2011 (אוגוסט פינגווין), on August 12th in Ramat-Gan, Israel.

August Penguin is the annual meeting of Hamakor society: an Israeli society for Free Software and Open-Source Code (read more here).

I'll be holding a non-technical talk about MySQL, titled "MySQL and the Open Source Sphere". In this talk I will be presenting my impressions of the nature of open source development of MySQL and surroundings: the core server, the various forks, patches, 3rd party tools, companies involved, etc. So this is a general "get to know who's

  [Read more...]

Dear Kettle users,

Most of you usually use a data integration engine to process data in a batch-oriented way.  Pentaho Data Integration (Kettle) is typically deployed to run monthly, nightly, hourly workloads.  Sometimes folks run micro-batches of work every minute or so.  However, it’s lesser known that our beloved transformation engine can also be used to stream data indefinitely (never ending) from a source to a target.  This sort of data integration is sometimes referred to as being “streaming“, “real-time“, “near real-time“, “continuous” and so on.  Typical examples of situations where you have a never-ending supply of data that needs to be processed the instance it becomes available are JMS (Java Message Service), RDBMS log sniffing, on-line fraud

  [Read more...]

In my announcement for common_schema I have failed to deliver the following message:

I will be happy to receive contributions to common_schema, and I will be happy to have contributors on this project

What kind of contributions are wanted?

So, I'm mostly interested right now in:

• Views: providing more insight on metadata (data types, schemata, volumes, transactions, locks, connections, etc.)
• Views: SQL code generation (e.g. generate my FOREIGN KEYs)
• Functions: DETERMINISTIC, NO SQL, preferably no control flow inside, making

  [Read more...]

I'm about to take a week off from my new gig as COO at Zendesk and it got me reflecting on the company and my decision to join.  I stayed with MySQL through the Sun acquisition and left when Oracle acquired Sun.  Although I have a lot of respect for Oracle, it seemed to me the only interesting jobs would be those that report directly to Larry Ellison.  So I took some time off to travel, worked as an EIR at Scale Ventures for a few months and began thinking about what I wanted to do next.

I turned down offers from companies and investors to come in and "repeat the MySQL playbook" in Big Data or NoSQL

  [Read more...]

Today I have released common_schema, a utility schema for MySQL which includes many views and functions, and is aimed to be installed on any MySQL server.

What does it do?

There are views answering for all sorts of useful information: stuff related to schema analysis, data dimensions, monitoring, processes & transactions, security, internals... There are basic functions answering for common needs.

Some of the views/routines simply formalize those queries we tend to write over and over again. Others take the place of external tools, answering complex questions via SQL and metadata. Still others help out with SQL generation.

Here are a few highlights:

• Did you know you can work out

  [Read more...]

I’ve just sent an email to the Maatkit discussion list to announce a planned change to how Maatkit (and Aspersa) are developed. In short, Percona plans to create a Percona Toolkit of MySQL-related utilities, as a fork of Maatkit and Aspersa. I’m very happy about this change, and I welcome your responses to that thread on the discussion list.

Related posts:

Aspersa, a new opensource toolkit

Four companies to sponsor Maatkit development

How Maatkit benefits from

  [Read more...]

By far, the most popular way for PDI users to load data into LucidDB is to use the PDI Streaming Loader. The streaming loader is a native PDI step that:

• Enables high performance loading, directly over the network without the need for intermediate IO and shipping of data files.
• Lets users choose more interesting (from a DW perspective) loading type into tables. In particular, in addition to simple INSERTs it allows for MERGE (aka UPSERT) and also UPDATE. All done, in the same, bulk loader.
• Enables the metadata for the load to be managed, scheduled, and run in PDI.

However, we’ve had some known issues. In fact, until PDI 4.2 GA

  [Read more...]

Following up on MySQL security: inconsistencies, and on MySQL bug #61596, I was thinking it may take a long time till the non-existent ROUTINE_PRIVILEGES view is implemented. Here’s my own implementation of the view.

I’ve followed the somewhat strange conventions used in the *_PRIVILEGES tables in INFORMATION_SCHEMA, where the IS_GRANTABLE is a separate column, although in 2nd 1st normal form.

I present it here as a query, using session variables, rather than a view definition:

SELECT STRAIGHT_JOIN
  CONCAT('\'', User, '\'@\'', Host, '\'') AS GRANTEE,
  NULL AS ROUTINE_CATALOG,
  Db AS ROUTINE_SCHEMA,
  Routine_name AS

  [Read more...]

Our last LucidDB release was now, just more than 12 months ago on June 16, 2010. We were really really trying to beat the 1 year mark for our 0.9.4 release but we just couldn’t. A tenet of good, open source development is early and often and we need to do better. Since the 0.9.3 release we’ve:

• Built out an entire Web Services infrastructure
• Developed a wicked cool Admin user interface
• Developed cool connectors to Hive, CouchDB
• Built a whole ton of extensions (auto indexing, DDL generation, improved load routines)
• Scriptable

  [Read more...]

Update

Since this article was written, HPCC has undergone a number of significant changes and updates. This addresses some of the critique voiced in this blog post, such as the license (updated from AGPL to Apache 2.0) and integration with other tools. For more information, refer to the comments placed by Flavio Villanustre and Azana Baksh.

The original article can be read unaltered below:

Yesterday I noticed this tweet by Andrei Savu: . This prompted me to read the related GigaOM article and then check out the  [Read more...]

Today, more and more proprietary software vendors are choosing to go Open Source. Doing this enables them to leverage the community benefits of Open Source, shorten the sales cycle, and gain a competitive advantage over other proprietary products.

However, for those firms considering a switch to Open Source, there are some hard decisions to make with regard to their product architecture. Should they provide only a single Open Source product, and earn revenue from add-on services like support and consulting

  [Read more...]

“Our experience from PNUTS also tells that these systems are hard to build: performance, but also scaleout, elasticity, failure handling, replication. You can’t afford to take any of these for granted when choosing a system. We wanted to find a way to call these out.” – Adam Silberstein and Raghu Ramakrishnan, Yahoo! Research. ___________________________________ A [...]

I wrote this up a while ago and decided that I didn’t want to lose it in a shuffle of documents during my transition to a new workstation. It’s the basics of setting up Heartbeat (LVS) + DRBD (block replication between active/passive master servers) + MySQL. This should give you the basics of a H/A system without the benefits of SAN but also without the associated cost. The validity of this setup for H/A purposes is highly dependent on your workload and environment. You should know the ins and outs of your H/A solution before deciding to blame the system for not performing as expected. As with all production systems you should test, test, test and test some more before going live.

When I get around to it later I’ll post my How-To for setting up RHCS + SAN + MySQL. You can download the DRBD document PDF here:

  [Read more...]

So I’ve been doing a fair number of automated load tests these past six months. Primarily with Sysbench, which is a fine, fine tool. First I started using some simple bash based loop controls to automate my overnight testing, but as usually happens with shell scripts they grew unwieldy and I rewrote them in python. Now I have some flexible and easily configurable code for sysbench based MySQL benchmarking to offer the community. I’ve always been a fan of giving back to such a helpful group of people – you’ll never hear me complain about “my time isn’t free”. So, let me know what you want in an ideal testing environment (from a load testing framework automation standpoint) and I’ll integrate it into my existing framework and then release it via the BSD license. The main goal here is to have a standardized modular framework, based on sysbench,

  [Read more...]

I split my time last week between the IOUG’s Collaborate conference in Orlando, Florida and O’Reilly’s MySQL Conference & Expo in California. The contrast was stark. For me as a MySQLer, Collaborate was a dud. On the other hand, the MySQL conference O’Reilly puts on is superb. It is vital to MySQL as a project and as a community, and it follows that it’s vital to MySQL’s business success. Oracle needs to participate to make it a success in the future.

MySQL at Collaborate had good speakers and content, but no one there is interested in MySQL. MySQL is just from a different world — it is a curiosity at an Oracle conference. Also, as a speaker, sponsor, and attendee, Collaborate was a giant frustration. I can’t recommend it to anyone. (These comments do not reflect on the work that MySQL community members did in

  [Read more...]

Today I have delivered my talk, openark-kit: MySQL utilities for everyday use, at the O’REILLY  MySQL Conference 2011.

The slides are uploaded to the O’Reilly site, and I’m attaching them here as well. Feel free to download the PDF: openark-kit-mysqlconf11.pdf

I wish to thank all who attended my talk!

 

This year was the first year the public could nominate candidates for these awards. We got in many nominations and this really helped the panel to get an overview of everything that is noteworthy in the MySQL ecosystem right now. Thank you to everyone who participated by sending in nominations.

In the first category the panel received and debated over a dozen suggestions. There are so many great people in this community, it is easy to think of people who truly deserve to be awarded. There were however two persons who clearly stood out with a track record of years and years of broad and really key contributions to the MySQL community. These two persons also clearly stood out both in number of nominations and number of votes. Therefore, the...

read more

I always remember the saying that tomorrow is the first day of the rest of your life.  It refers to taking each day as it comes and making the best of it, and not concerning yourself with what you can no longer change - the past. This has new resonance for me today as tomorrow I start on a new job, as Webmaster for SkySQL Ab.

For the last five and a half years I've worked for MySQL in all its forms.  Initially MySQL AB, then the Database Group in Sun Microsystems, and finally the Open Source business unit in Oracle. (For anyone still at Oracle, forgive me if I get the names wrong, I don't think I ever fully worked out what our group's official name was).  For most of that time I've loved every minute of it, working with great people on a great product and bringing it to the world (or at least the World Wide Web

  [Read more...]

For Open Source projects whose software architecture allows it, inviting developers to extend the core product through add-on modules and plug-ins is a great way to raise interest and awareness and thus kickstart or foster an adoption/contribution cycle. In such a setting, Open Source vendors and their business partners should consider building and maintaining an online marketplace or exchange for add-ons, which will serve as a highly effective distribution and sales channel.

Distribution and Sales Channel

Such a modules marketplace

  [Read more...]

I have been working with MySQL replication for quite a while. I have dealt with simple replication setups and I have experimented with complex ones. Five years ago I wrote an article about advanced MySQL replication, which was mostly a dream on what you could do with imagination and skill, but the matter from that article is still not even remotely ready for production. Yet, since that article, I have been approached by dozens of people who wanted to know how to make the multiple master dream become reality. To all of them, I had to say, "sorry, this is just a proof of concept.Come back in a few years, it may become possible". It still isn't.
Despite its latest great technological advance, MySQL native replication is is very poor of topologies. What you can do with MySQL native
  [Read more...]

Topology

Participating hosts

* 1x BIG-IP VE
* 1x Debian Squeeze “Provisioning Server” serving DHCP, TFTP & DNS
* 3x Nortel/Avaya 1120E hard phones flashed to SIP1120e04.01.13.00
* 2x Debian Squeeze + digium asterisk packages:

$ grep asterisk /etc/apt/sources.list
deb http://packages.asterisk.org/deb squeeze main
deb-src http://packages.asterisk.org/deb squeeze main

DUNDi

*CLI> module reload pbx_dundi.so
*CLI> dundi show peers
*CLI> dundi show mappings
*CLI> dundi lookup 4012@extensions bypass
*CLI> dundi set debug on

SIP

*CLI> sip show peers

dialplan

*CLI> dialplan show RegisteredDevices

Work in progress

Yesterday, I installed a trixbox virtual machine using the

  [Read more...]

I finally did the migration to Drupal 7 for the Footnotes module this weekend. See Release notes and project page for more information.

With this release I also announced my intent to hand over the module to a new maintainer. Since I'm now increasingly active with affairs in the MySQL community, both hacking as well as other community tasks, it is prudent to not let old projects dangle without attention but to formally hand them over to fresh minds.

read more

Dear Kettle friends,

Some time ago while I visited the nice folks from Human Inference in Arnhem, I ran into Kasper Sørensen, the lead developer of DataCleaner.

DataCleaner is an open source data quality tool released (like Kettle) under the LGPL license.  It is essentially to blame for the lack of a profiling tool inside of Kettle.  That is because having DataCleaner available to our users was enough to push the priority of having our own data profiling tool far enough down.

Kasper worked on DataCleaner pretty much in his spare time in the past.  Now that Human Inference took over the project I was expecting more frequent updates and

  [Read more...]