This article is the third in a series on data fabric design and introduces the fabric connector service design pattern.  The previous article in this series introduced the transactional data service design pattern, which defines individual data stores and is the building block for data fabrics based on SQL databases.  The fabric connector builds on transactional data services and is another basic building block of fabric architecture.

Description and Responsibilities
Fabric connectors make a collection of DBMS servers look like a single server.  The fabric connector presents what appears to be a data service API to applications.  It


  [Read more...]

Cloud databases have a memory problem.   Continuent has been doing a lot of Amazon deployments lately, and it is becoming apparent that memory utilization in those environments is more than just an inconvenience.  In this article I would like to discuss the memory problem that we see in customer implementations and some new features of Tungsten Enterprise that help alleviate it for MySQL.

The Cloud Memory Problem and Database Arrays

As I discussed in a recent article about prefetch, the amount of RAM allocated to the InnoDB buffer pool is one of the principle determinants of MySQL performance.  The speed difference between using



  [Read more...]

Here are some of the fake O'Reilly book covers I mentioned in a prior post.  These have been optimized for use as black & white Kindle screensaver wallpaper images.  If you haven't done so already, you can install a Kindle screensaver hack with a couple of downloads. 

Update: I've embedded a slideshow from PicasaWeb, but it requires Flash.  If you don't see it you can click on the links below to go directly to

  [Read more...]

Multi-master database systems that span sites are an increasingly common requirement in business applications.  Yet the way such applications work in practice is not quite what you would think from accounts of NoSQL systems like Cassandra or SQL-based systems like Oracle RAC.  In this article I would like to introduce a versatile design pattern for multi-master SQL applications in which individual schemas are updated in a single location only but may have many copies elsewhere both locally as well as on other sites.  This pattern is known as a system of record architecture.  You can build it with off-the-shelf MySQL and master/slave  [Read more...]

I'm about to take a week off from my new gig as COO at Zendesk and it got me reflecting on the company and my decision to join.  I stayed with MySQL through the Sun acquisition and left when Oracle acquired Sun.  Although I have a lot of respect for Oracle, it seemed to me the only interesting jobs would be those that report directly to Larry Ellison.  So I took some time off to travel, worked as an EIR at Scale Ventures for a few months and began thinking about what I wanted to do next.

I turned down offers from companies and investors to come in and "repeat the MySQL playbook" in Big Data or NoSQL

  [Read more...]

This title is not nearly as snappy as Yeats' line from "The Second Coming," but it will have to do.  Conflict resolution has been a holy grail for us on Tungsten--we had people asking for it when MySQL was still MySQL AB, lo these many years ago.   Well, it's finally starting to happen after about a year of work.  

Let's start with a simple multi-master configuration.  To replicate bi-directionally between two masters, you typically run Tungsten replicator on each master with two replication services on each master.  The first local service reads the master log.  The second remote service is a slave of the other master.  It looks like the following picture:





  [Read more...]

My previous post focused on some of the problems of doing multi-tenant MySQL.

One of the reasons why I started hacking on Drizzle was that the multi-tenancy options for MySQL just weren’t very good (this is also the reason why I run my blog in a VM and not a shared hosting solution).

What you really want is to be able to give your users access to a virtual database server. What you don’t want is to be administering a separate database server for each of your users. What you want are CATALOGs.

A CATALOG is a collection of SCHEMAs (which have TABLEs in them). Each CATALOG is isolated from all the others. Once you connect to a catalog, that’s it. They are entirely separate units. There are no cross-catalog queries or CHANGE CATALOG commands. It is as

  [Read more...]

Last month my colleague Giuseppe Maxia described how to operate Tungsten parallel replication. Since then we have been doing a good bit of benchmarking on both synthetic as well as real production loads. In this article I would like to follow up with some tips about how you can goose up parallel replication performance.  These apply to Tungsten Replicator 2.0.1, which you can find here. 

The first way to get good performance with Tungsten is to have the right workload. As explained in an earlier article on this blog, Tungsten parallel replication works

  [Read more...]

A special extended edition of Tech Messages for 2010-12-22 through 2011-01-04:

• WS-C3560V2-XXXX-X, WS-C3750V2-XXXX-X Series RPS cover and rare power reset – Replacement Recommended
  The RPS cover for wall-mount of the Acc Kits in the BOM was not correct; it should be 22P instead of 14P. The issue is a safety issue for those who choose to wall-mount the RPS. It's found on WK48P/NZ48P that the magjack connector's shielding were short to the test via 2.5v to ground. Adding capton tape underneath Belfuse/Pulse connector to avoid the short via ECO as a short term solution will respin PWB to correct issue.
• WordPress › 3.0.4

  [Read more...]

I only recently found out about GigaOm's upcoming Net:Work conference.  It's held December 9 at UCSF Mission Bay conference center.  While the name of the conference is a bit ambiguous, the actual area of focus is very clear: how will we collaborate in the 21st century?  

The impact of smartphones, tablet computing, social networks, Software-as-a-Service and Cloud computing is just starting.  As a result, I think there are tremendous opportunities for startup companies to disrupt existing markets with more modern, lightweight applications

  [Read more...]

I wrote a guest column for GigaOm on how open source software, cloud and software as a service are helping to bring about the consumerization of IT: namely bringing simplicity where complexity reigned.  I cited some examples including New Relic, Box.net and Apple.

Open source has gone a long way toward putting power back in the hands of developers, who can download, install and deploy software without having to go through any kind of convoluted sales or budget approval process.  You want MySQL?  You can download and install in 

  [Read more...]

Single-threaded apply is one of the big downsides of MySQL's built-in replication, as Baron Schwartz pointed out a couple of days ago.  While a master can process dozens of updates at once, slaves must apply them one after the other on a single thread.  Add in disk I/O, and the result is very slow performance indeed.  The obvious answer is parallel apply, namely writing multiple non-conflicting updates to the slave at once.

I have spent the last few months implementing parallel apply for Tungsten 2.0, which we are now testing at customer sites.  In this article I would like to describe how Tungsten's parallel apply works as well as some of the lessons that have become apparent through the implementation.



  [Read more...]

Disaster recovery (DR) is not the first thing most DBAs think of when putting up a new database application.   However, it's one of the top issues for people using the data--what happens if the site goes down and everything disappears?   So even if DR is not the first issue in every deployment, it is a very high priority as soon as your application is the least bit successful.

At the database level DR has a fairly simple solution:  keep copies of data on a backup site that is up-to-date at all times.  This article explains the architecture for MySQL DR with Tungsten and a couple of key features that make it work, namely floating IP addresses and global transation IDs.  We will dig into those at the end.

First a bit of introduction.  Tungsten manages clusters of off-the-shelf database connected by



  [Read more...]

Have you ever run into a problem where MySQL replication did 95% of what you needed but not the remaining 5% to solve a real problem?  Hacking the binlog is always a possibility, but it typically looks like this example.  Not a pretty sight.  Wouldn't it be easier if replication were a bunch of building blocks you could recombine to create custom replicator processes? 

Tungsten 1.3 has a new feature called pipelines that allows you to do exactly that.  A pipeline consists of one or more stages that tie together generic components to extract, filter, store, and apply events, which is Tungsten parlance for transactions.  Each stage has a processing thread, so multi-stage pipelines can process data independently and without blocking.  The stages also take care of

  [Read more...]

I would like to announce a couple of new Tungsten versions available for your database clustering enjoyment.  As most readers of this blog are aware, Tungsten allows users to create highly available data services that include replicated copies, distributed management, and application connectivity using unaltered open source databases.   We are continually improving the software and have a raft of new features coming out this year.  

First, there is a new Tungsten 1.2.3 maintenance release available in both commercial as well as open source editions.  You can get access to the commercial version on the Continuent website, while the open source version is available on SourceForge. 

 The Tungsten 1.2.3



  [Read more...]

I have been a little bit stubborn on my concession that cloud computing is really going to be the wave of the future. Though lately I have been playing with a lot of cloud related technologies and am hooked. Spin up 10 (or 100) Linux servers in a minute from a web page with no real systems management skills and then you

  [Read more...]

Software-as-a-Service (Saas) is one of the main growth areas in modern database applications.  This topic has become a correspondingly important focus for Tungsten, not least of all because new SaaS applications make heavy use of open source databases like MySQL and PostgreSQL that Tungsten supports.

This blog article introduces a series of essays on database architectures for SaaS and how we are adapting Tungsten to enable them more easily.  I plan to focus especially on problems of replication and clustering relevant to SaaS—what are the problems, what are the common design patterns to solve them, and how to deploy and operate the solutions. I will also discuss how to make replication and clustering work better for these cases—either using Tungsten features that already exist or features we

  [Read more...]

A lot of people are into the whole cloud computing scenario these days. However, no one has talked about offering DBA-like services in the cloud, all automated, so that startups don’t have to get their own DBAs.

Enter FathomDB. They are poised to offer databases as a service (maybe they’ll charge per database - so you can in theory run both WordPress and Mediawiki, if you prefix wp_ and mw_ in your table creation, for example). They are using MySQL. They’ve also taken the worry of running a database out - they will backup, they will setup (so you don’t have to issue GRANT commands :P), and they will also monitor your databases for you.

But what really takes the cake? The fact that they will also offer performance advisors. This totally reminds me of the MySQL

  [Read more...]