Pivotal launches. SkySQL and Mony Program merge. And much, much more

Our report on the changes in the MySQL ecosystem is now available for 451 clients and non-clients alike at bit.ly/451mysql

— Matt Aslett (@maslett) April 25, 2013

For 451 Research clients: VMware expands Serengeti’s horizons with updated Hadoop virtualization project bit.ly/17muQFI

— Matt Aslett (@maslett) April 26, 2013

For 451 Research clients: SkySQL, Monty Program merge to support MariaDB following formation of MariaDB Foundation bit.ly/10dsdjf

  [Read more...]

I’m sitting here in the lounge at SFO thinking back on the last week, the majority of which has been spent meeting my new workmates and attending the Percona MySQL conference.

For me it has been as much of a family reunion as it has been about seeing the wonderful things going on in MySQL.

Having joined Continuent last month after an ‘absence’ in NoSQL land of almost 2.5 years, joining the MySQL community again just felt like coming home after a long absence. And that’s no bad thing. On a very personal level it was great to see so many of my old friends, many of whom were not only pleased to see me, but pleased to see me working back in the MySQL fold. Evidently many people think this is where I belong.

What was great to see is that the MySQL community is alive and

  [Read more...]

Switching roles

To get a taste of the power of Tungsten Replicator, we will show how to switch roles. This is a controlled operation (as opposed to fail-over), where we can decide when to switch and which nodes are involved.

In our topology, host1 is the master, and we have three slaves. We can either ask for a switch and let the script select the first available slave, or tell the script which slave should be promoted. The script will show us the steps needed to perform the operation.

IMPORTANT! Please note that this operation is not risk free. Tungsten replicator is a simple replication system, not a complete management tool like Continuent Tungsten. WIth the replicator, you must make sure that the applications have stopped writing to the master before starting the switch, and then you

  [Read more...]

Intro

Tungsten Replicator is an open source tool that does high performance replication across database servers. It was designed to replace MySQL replication, although it also supports replication from and to Oracle and other systems. In this article, we will only cover MySQL replication, both simple and multi-master.

Preparing for installation

To follow the material in this article, you will need a recent build of Tungsten Replicator. You can get the latest ones from http://bit.ly/tr20_builds. In this article, we are using build 2.0.8-167.

Before starting any installation, you should make sure that you have satisfied all the prerequisites. Don't

  [Read more...]

In the first part of this article we examined the types of conflicts and their causes. In this part, we will analyse some of the methods available to deal with conflicts.

Pessimistic locking (or: conflicts won't happen)

Applicability: synchronous clusters with 2pc

We've covered this topic in the previous article, but it's worth repeating. If you use a synchronous cluster, you don't have conflicts. For example, MySQL Cluster ensures consistent data with updates coming from different nodes. However, MySQL Cluster is not a replacement for a MySQL server, and it has severe limitations.

Optimistic locking

Applicability: synchronous clusters without 2pc (Galera)

Conflicting transactions proceed on different

  [Read more...]

What is a conflict?

Readers of this blog know that one of my favorite tools, Tungsten Replicator, can easily create multi-master replication topologies, such as all-masters, star, fan-in. While this is good news for system designers and ambitious DBAs, it also brings some inconvenience. When you allow updates to happen in more than one master, you risk having conflicts. You may have heard this term before. For the sake of clarity, let's define what conflicts are, before analyzing each case in detail.

You have a conflict when several sources (masters) update concurrently the same data in asynchronous replication.

It's important to stress that this happens with asynchronous replication. In a truly synchronous cluster, where all data is kept consistent through

  [Read more...]

The Percona Live MySQL Conference and Expo 2013 is almost 1 month away. It's time to start planning, set the expectations, and decide what to attend. This post will give a roundup of some of the sessions that I recommend attending and I look forward to.

First, the unexpected!

After much talk and disbelief, here they come! Oracle (http://www.mysql.com) engineers will participate to the Percona Live conference. This is wonderful! Their participation was requested by the organizers, by the attendees, and by community advocates, who all told the Oracle management how important it is to be in this conference. Finally, they have

  [Read more...]

Stating the problem.

In my job, I do a lot of testing. And no matter how much organized we try to be, we end up with fewer machines than we would need to run all the tests that we want.

For some tasks, we can run MySQL Sandbox, and get the job done. But sometimes we need to make sure that applications and systems work well across the network, and we need to install and run systems on separate servers.

However, when you test replication systems, and every cluster takes three or four servers, you run our of available hosts very quickly. So you decide to use the clusters that are dedicated to automated testing to also run your own manual tests. Soon you realize that the tests that you are running manually are clashing with the automated ones, or with the ones that your colleagues are running.

A simple solution

  [Read more...]

On Thursday, I will travel to Boston, MA, to attend the Northeast LinuxFest, which includes also an edition of the Open Database Camp. The events will be at one of my favorite places on earth: The Massachusetts Institute of Technology, a.k.a. the MIT. Every time I speak at an event there, I feel at home, and I look forward to be there once more.

The Open Database Camp is organized, as usual, with the formula of an un-conference, where the schedule is finalized on the spot.

There are a few ideas for sessions. I have proposed two of the topics I am most familiar with:

  [Read more...]

Intro

While working on one of the MySQL to Oracle replication projects for Continuent, I needed to implement an open-source utility for transforming MySQL schema to an Oracle dialect (DDL statements that create specific schema on Oracle) to save from otherwise tedious work. This article introduces ./ddlscan tool, which does that and is extensible to do much more.

Ingredients

Here's what you'll need:

• Your favorite DBMS with some tables. Currently supported MySQL, Oracle and PostgreSQL.
• Latest Tungsten Replicator build. Not even needed to install, enough to untar.
• Velocity template of your

  [Read more...]

MySQL 5.6 is probably the version of MySQL with the biggest bundle of new features. You may want to try it soon, since it's now released as GA, and I would like to offer some practical experience on how to use some of the advanced features.

Since replication is my main interest, I will focus on some of the new features in this field, and I will compare what's available in MySQL 5.6 with Tungsten Replicator.

The focus of the comparison is usability, manageability, and some hidden functionality. Parallel replication has been available with Tungsten Replicator for almost two years, and Global Transaction Identifiers for much longer than that. With MySQL 5.6, it seems that the MySQL team wants to close the gap. While the main feature (parallel execution threads) is available and performing well, there are some

  [Read more...]

Tungsten Replicator 2.0.7 was released today. In addition to a large number of bug fixes, this release adds several improvements for multi-master management, and support for Amazon RDS (as a slave).

While the Release Notes show a long list of improvements, I would like to focus on some of them that improve the handling of multi-master deployments.

When we released version 2.0.6, we added the first revision of the cookbook recipes in the build. That was still a green addition, which caused several bug reports. But since then, we have integrated the cookbook in our internal testing, making these recipes more robust and reliable. We are also planning to improve

  [Read more...]

This article is the third in a series on data fabric design and introduces the fabric connector service design pattern.  The previous article in this series introduced the transactional data service design pattern, which defines individual data stores and is the building block for data fabrics based on SQL databases.  The fabric connector builds on transactional data services and is another basic building block of fabric architecture.

Description and Responsibilities
Fabric connectors make a collection of DBMS servers look like a single server.  The fabric connector presents what appears to be a data service API to applications.  It


  [Read more...]

Data management is undergoing a revolution.  Many businesses now depend on data sets that vastly exceed the capacity of DBMS servers.  Applications operate 24x7 in complex cloud environments using small and relatively unreliable VMs.  Managers need to act on new information from those systems in real-time. Users want constant and speedy access to their data in locations across the planet.

It is tempting to think popular SQL databases like MySQL and PostgreSQL have no place in this new world.  They manage small quantities of data, lack scalability features like parallel query, and have weak availability models.  One reaction is to discard them and adopt alternatives like Cassandra or MongoDB.  Yet open source SQL databases have tremendous strengths:  simplicity, robust transaction support, lightning fast operation, flexible

  [Read more...]

We have started a new series of webinars at Continuent that we call Tungsten University.  They provide education on Tungsten clustering and replication in handy one-hour chunks.  These are not sales pitches.  Our goal is to provide accessible education about setting up and operating Tungsten without any marketing fluff.

The first Tungsten University webinar entitled "Configure & provision Tungsten clusters" will take place on Thursday January 17th at 10:00 PST.  It will show you how to set up a cluster in Amazon EC2.  There will be a repeat on January 22nd at 15:00 GMT.  We usually record webinars, so you can look at them later as well. 
You do not have to be a customer to


  [Read more...]

The recent release of the MariaDB client libraries has prompted questions about their purpose as well as provenance.  Colin Charles posted that some of these would be answered in the very near future.  I have a couple of specific questions about the MariaDB JDBC driver, which I hope will be addressed at that time.  
1.) What is really in the MariaDB JDBC driver and how exactly does it differ from the drizzle JDBC driver?  What, if any, relation is there to Connector/J code?  There is a 
  [Read more...]

The  MariaDB Foundation announcement spawned some interesting commentary about the state of open source databases.  One recent headline cited the "beleaguered MySQL community." Beleaguered is a delightful adjective.  The OED tells us that it means beset, invested, or besieged.  Much as I like the word, I do not think it is an accurate or useful description of the MySQL community.  This article and others like it miss the point of what is happening to MySQL and its users.

Let's start by disproving that the notion that the MySQL community is

  [Read more...]

Percona hosted another excellent Percona Live conference this past December 3-4 in London.  It was my pleasure to deliver 3 talks including the first keynote following Peter Zaitsev.  Percona does a great job of organizing these conferences--this year's London conference was well attended and in an excellent location in Kensington.  My thanks to the entire Percona team for putting this together.

Here are the slides for my talks in case you would like to see them.

Keynote:  Future-Proofing MySQL for the World-Wide Data Revolution -- Covering the greatly exaggerated death of MySQL and design patterns for robust MySQL systems that can last for decades

Talk:  





  [Read more...]

On Monday afternoon, Neal Armitage and I will be speaking at Percona Live in London. It will be a three hours tutorial about Tungsten replicator.

The contents of this tutorial are mostly new. We have released recently a new and easier way of installing different topologies, in the shape of cookbook scripts, which are distributed with the replicator tarball.

Using this cookbook, any user can simply install multiple topologies, from the simple master/slave to all-masters, fan-in, and star.

There are recipes for showing the replication cluster, switching roles between master and a chosen slave, taking over MySQL replication, installing direct slaves with parallel replication, testing each topology, and

  [Read more...]

Tungsten Replicator version 2.0.6 was released today.

You can get both the binaries and the source code at the project's downloads page.

This release contains many bug fixes, and various improvements. All of them are listed in the Release Notes. The most interesting ones are the improvement in multi-master topologies. Using this release with star topologies you will get less traffic than before, because we have reduced some duplication of transaction history logs that were sent between servers.

And speaking of multi-master topologies, this release includes the cookbook recipes mentioned in this blog

  [Read more...]

Attached are the slides for my MySQL Connect talk Evaluating MySQL High-availability alternatives, which I will present today at 14:30 at the MySQL Connect conference.

A bit unusually I'm posting the material ahead of the talk. The point of the talk is about evaluating each alternative from your own perspective. With that in mind, if you're at the talk with your own laptop, feel free to browse the slides at your own pace from here.

read more

The fall conference season is starting.  I will be doing a number of talks including a keynote on "future proofing" MySQL through the use of data fabrics.  Data fabrics allow you to build durable, long-lasting systems that take advantage of MySQL's strengths today but also evolve to solve future problems using fast-changing cloud and big data technologies.  The talk brings together ideas that Ed Archibald (our CTO) and I have been working on for over two decades.  I'm looking forward to rolling them out to a larger crowd.

Here are the talks in calendar order.  The first two are at MySQL Connect 2012 in San Francisco on September 30th:

  [Read more...]

There have been a number of excellent articles about the pros and cons of automatic database failover triggered by Baron's post on the GitHub database outage.  In the spirit of Peter Zaitsev's article "The Math of Automated Failover," it seems like a good time to point out that database failure is usually not the biggest source of downtime for websites or indeed applications in general.  The real culprit is maintenance.

Here is a simple table showing availability numbers out to 5 nines and what they mean in terms of monthly down-time.

Normal 0 false false



  [Read more...]

Github had a recent outage due to malfunctioning automatic MySQL failover.  Having worked on this problem for several years I felt sympathy but not much need to comment.  Then Baron Schwartz wrote a short post entitled "Is automated failover the root of all evil?"  OK, that seems worth a comment:  it's not.  Great title, though.

Selecting automated database failover involves a trade-off between keeping your site up 24x7 and making things worse by having software do the thinking when humans are not around.  When comparing outcomes of wetware vs. software it is worth remembering that humans are not at their best when woken up at 3:30am.  Humans go on

  [Read more...]

This month I have joined the team at Continuent. No stranger to the MySQL ecosystem, Continuent provides replication and clustering technology for managing data between MySQL, Oracle, PostgreSQL, Vertica and a growing list of data stores.

I have known many of the team at Continuent for some time, and will again be joining Giuseppe Maxia from our days at MySQL Inc/AB starting back in 2006.

I am looking forward to taking the hard work out of administration of MySQL systems with the simplicity of Continuent Tungsten, simplifying tasks including automatic failover, multi-master and geo cluster redundancy to a single command.

Catch me speaking at the upcoming

  [Read more...]

In late 2011 I attended a lecture by John Wilkes on Google compute clusters, which link thousands of commodity computers into huge task processing systems.  At this scale hardware faults are common.  Google puts a lot of effort into making failures harmless by managing hardware efficiently and using fault-tolerant application programming models.  This is not just good for application up-time.  It also allows Google to operate on cheaper hardware with higher failure rates, hence offers a competitive advantage in data center operation.

It's becoming apparent we all have to think like Google to run applications successfully in the cloud.  At Continuent we run our IT and an increasing amount of QA and development on Amazon Web

  [Read more...]

Synchronous data replication over long distances has the sort of seductive appeal that often characterizes bad ideas.  Why wouldn't you want every local credit card transaction simultaneously stored on the other side of the planet far away from earthquake, storms and human foolishness?  The answer is simple: conventional SQL applications interact poorly with synchronous replication over wide area networks (WANs).

I spent a couple of years down the synchronous replication rabbit hole in an earlier Continuent product.  It was one of those experiences that make you a sadder but wiser person.  This article digs into some of the problems with synchronous replication and shows why another approach, asynchronous multi-master replication, is currently a better way to manage databases connected by

  [Read more...]

Intro

Oracle is widely use to support back-end systems.  On the other hand, MySQL is the "go-to" data management solution for the web-facing part of many businesses.  If you have both Oracle and MySQL in-house, you may already also have the need to share data between them.  In this article I'll describe software that my colleagues and I have been working on to move data from Oracle to MySQL in real-time without costing an arm and a leg.

Tungsten to the Rescue!

Latest Tungsten Replicator has many features, most of which are open-source, but the recent one for me is particularly exciting - thanks to the





  [Read more...]