Slides from my Percona Live talk evaluating the new spatial features in MySQL 5.6 and MariaDB 5.5 are now online. This is new material I have never presented before. It is based on work I have done in my job at Nokia HERE.com location services. So even if at this conference it draws less attention than my HA talks, it is actually what I'm most proud of to present.

TL;DR summary is that PostgreSQL has lots of features but MySQL has much better ease of use and performance. (I copy paste this standard sentence into any PostgreSQL vs MySQL evaluation I do :-) The MongoDB info is basically outdated, as the new 2.4 release introduces completely new implementation based on GeoJSON, new indexing, neither of which I tested.

read more

High Performance MySQL, 3rd Edition has been selling very well. It’s translated into many languages. O’Reilly sends me a hard-copy of the translations, and I have a whole section on my bookshelf dedicated to them. It’s really satisfying to look at it.

Today I’m happy to announce that we’re moving forward with a new batch of translations. Demand has been so strong that we want to make the book accessible to as wide an audience as possible. Plus, I get a fat check every time O’Reilly sells the translation rights.

The new languages will include Australian, l337 (“Leet”), Jive, Ebonics, Elmer Fudd, Blissymbols, and Esperanto. Here’s a sample before-and-after paragraph:

Isolating

  [Read more...]

In the first part of this article we examined the types of conflicts and their causes. In this part, we will analyse some of the methods available to deal with conflicts.

Pessimistic locking (or: conflicts won't happen)

Applicability: synchronous clusters with 2pc

We've covered this topic in the previous article, but it's worth repeating. If you use a synchronous cluster, you don't have conflicts. For example, MySQL Cluster ensures consistent data with updates coming from different nodes. However, MySQL Cluster is not a replacement for a MySQL server, and it has severe limitations.

Optimistic locking

Applicability: synchronous clusters without 2pc (Galera)

Conflicting transactions proceed on different

  [Read more...]

From my previous posts, we have some numbers (excluding NDB) for the size of MySQL, so what about PostgreSQL? Here, I used PostgreSQL git trunk and classing things in the contrib/ directory as plugins. I put the number of lines of code in the src/backend/storage directory down as storage engines LoC but did not count it as non-kernel code.

Version Total LoC Plugin LoC Storage Engines LoC Remaining (kernel) MySQL 5.5.30 858,441 2,706 171,009 684,726 (79% kernel) MySQL 5.6.10 1,049,344 29,122 236,067 784,155 (74% kernel) MariaDB 5.5 1,142,118 11,781 304,015 826,322 (72% kernel) Drizzle trunk 334,810 31,150 130,727 172,933 (51% kernel) PostgreSQL trunk 648,691 61,934 17,802 586,757  [Read more...]

This article is the third in a series on data fabric design and introduces the fabric connector service design pattern.  The previous article in this series introduced the transactional data service design pattern, which defines individual data stores and is the building block for data fabrics based on SQL databases.  The fabric connector builds on transactional data services and is another basic building block of fabric architecture.

Description and Responsibilities
Fabric connectors make a collection of DBMS servers look like a single server.  The fabric connector presents what appears to be a data service API to applications.  It


  [Read more...]

Teradata results. Funding for DataXu. The chemistry of data. And more.

For 451 Research clients: Oracle launches major update to MySQL open source database bit.ly/TSONAt

— Matt Aslett (@maslett) February 8, 2013

For 451 clients: Analyzing the chemistry of data bit.ly/TSOV2R By @451wendy Treating sensitive data like dangerous chemicals

— Matt Aslett (@maslett) February 8, 2013

Teradata: Q4 net income $112m on revenue up 10% to $740m, FY net income $419m on revenue up 13% to $2.7bn. bit.ly/14FNS8L

  [Read more...]

This article is the second in a series on data fabric design and introduces the transactional data service design pattern.  The previous article in this series introduced data fabrics, which are collections of off-the-shelf DBMS servers that applications can connect to like a single server.  They are implemented from data fabric design patterns, which are reusable arrangements of DBMS servers, replication, and connectivity.  With this article we begin to look at individual design patterns in detail.

Description and Responsibilities
The transactional data service is a basic building block of data fabric architectures.  A transactional data service is a DBMS server that processes transactions submitted by applications and stores data safely.


  [Read more...]

Data management is undergoing a revolution.  Many businesses now depend on data sets that vastly exceed the capacity of DBMS servers.  Applications operate 24x7 in complex cloud environments using small and relatively unreliable VMs.  Managers need to act on new information from those systems in real-time. Users want constant and speedy access to their data in locations across the planet.

It is tempting to think popular SQL databases like MySQL and PostgreSQL have no place in this new world.  They manage small quantities of data, lack scalability features like parallel query, and have weak availability models.  One reaction is to discard them and adopt alternatives like Cassandra or MongoDB.  Yet open source SQL databases have tremendous strengths:  simplicity, robust transaction support, lightning fast operation, flexible

  [Read more...]

Postgres XC explained 

Users demand faster processing and as the processing and reports get faster so do the expectation for even faster performance and scalability. Now days its unthinkable not to deliver the performance  and availability.
All of those challenges require a constant search/development of new and improves solutions. Enter Postgres XC just when you were thinking that RDBMS is dead it comes back with the vengeance. So what is exactly Postgres XC and why should anyone care? 
Postgres XC shares a common name with the Postgres RDBMS but this is where the similarities stop, Postgres

  [Read more...]

451 Research’s 2013 Database survey is now live at http://bit.ly/451db13 investigating the current use of database technologies, including MySQL, NoSQL and NewSQL, as well as traditional relation and non-relational databases.

The aim of this survey is to identify trends in database usage, as well as changing attitudes to MySQL following its acquisition by Oracle, and the competitive dynamic between MySQL and other databases, including NoSQL and NewSQL technologies.

There are just 15 questions to answer, spread over five pages, and the entire survey should take less than ten minutes to complete.

All individual responses are of course

  [Read more...]

The  MariaDB Foundation announcement spawned some interesting commentary about the state of open source databases.  One recent headline cited the "beleaguered MySQL community." Beleaguered is a delightful adjective.  The OED tells us that it means beset, invested, or besieged.  Much as I like the word, I do not think it is an accurate or useful description of the MySQL community.  This article and others like it miss the point of what is happening to MySQL and its users.

Let's start by disproving that the notion that the MySQL community is

  [Read more...]

Apart from my consulting as part of ScaleIn, I also invest to bootstrap companies with really disruptive ideas; and in the process met few database specific companies who are already[...]

Here is the typical “Big” data architecture, that covers most components involved in the data pipeline. More or less, we have the same architecture in production in number of places[...]

With MySQL it was for a long time the case that a lot of sub queries would actually perform poorly, because of poor execution plans. (This is no longer the case in MariaDB 5.5 or the upcoming MySQL 5.6.) Because of this, any MySQL DBA knows the rule of thumb that sub-queries should basically be avoided and you can usually get the same result by using JOINs instead.

I've now learned why PostgreSQL DBAs like sub queries so much. PostgreSQL - being the most advanced open source database - apparently does the exact opposite optimizations as MySQL: it requires you to rewrite simple queries into complex subqueries to get what you want. (Update: Mark Callaghan points out that MySQL - while it does create indexes automatically for foreign keys - actually has the same problems with the query plan as Postgres has in this post. See comments for details.)

read more

ConFoo is one of the most important web developer-oriented conferences. ConFoo 2013 will be held on February 25 through March 1 in Montreal, Canada.

We just opened call for papers and we are looking for the best PHP, Java, Ruby, DotNet,HTML5 experts who are willing to share their knowledge with the Canadian community. Candidates can submit proposals until September 23. Consult the call for papers page for details and to start submitting. That page also explains what expenses ConFoo can cover for speakers. You can also get advice on how to write proposals.

The call for papers is public, meaning that all proposals get published on the website for others to

  [Read more...]

My three previous blog posts I already wrote from Froscon. In this post I still want to go back and mention some people I met and discussions I had.

The MySQL side

There were of course many MySQL people, with both SkySQL and Oracle sponsoring. It was great to meet Carsten from Oracle, who has joined the MySQL Sales Engineer team in Europe (he moved from an OpenOffice position). That's my former team, so it was great to see a new face!

Going there the person I was most looking forward to meet was Hana Hütter, formerly a MySQL account manager for Central Europe, and now doing the same at SkySQL. My first ever MySQL sales gig was with Hana, and Ralf Gebhart who is also now with SkySQL but was not at Froscon. While Ralf was there only that first time to teach me how to be a Sales Engineer, with Hana we then continued to sell MySQL into telecom companies

  [Read more...]

In my quest to understand spatial GIS functionality, I have come to the ultimate goal: evaluation the actual database products themselves.

PostgreSQL / PostGIS

PostGIS is a variant of PostgreSQL with spatial extensions. The main reason for maintaining the GIS feature set outside of PostgreSQL proper seems to be licensing: the spatial extensions are LGPL GPL licensed.

PostGIS is widely recognized as the most mature and feature-rich GIS implementation for SQL databases (and perhaps any database), matched only by the costly Spatial extension for the Oracle Enterprise database. (See comparisons in the links below.)

read more

While the underlying index should be opaque to the user of a DBMS with spatial features, the API used to define spatial types and operate on them is of course more visible. The relevant standard in this space is often referred to as "OpenGIS", however the Open Geospatial Consortium in fact defines a long list of standards. The standard relevant to SQL databases is known more precisely as "OpenGIS Implementation Specification for Geographic information - Simple feature access - Part 2: SQL option" aka "Simple feature access".

It is not meaningful to recite the standard at length in my blog, my focus is instead on actual implementations that I will blog about later. The following points are however worth noting:

read more

I work for a company that is the leading supplier of automotive maps, and wants to be the leading supplier of online maps. So it was only a matter of time that I needed to learn more about how spatial extensions work in different open source databases. Let's start from the beginning, understanding various spatial data structures that are used in implementations...

Links are provided to Wikipedia articles - which are both comprehensive, yet easy to understand - for those who want to get a deeper understanding of each structure. All Wikipedia articles on spatial indexes are listed here: http://en.wikipedia.org/wiki/Spatial_index#Spatial_index

B-tree

read more

The recently released TokuDB brings many features. One of those features is support for XA Transactions. InnoDB already has support for XA Transactions.

XA Transactions are transactions which span multiple databases and or applications. XA Transactions use 2-phase commit, which is also the same method which MySQL Cluster uses.

Internal XA Transactions are used to keep the binary log and InnoDB in sync.

Demo 1: XA Transaction on 1 node:

mysql55-tokudb6> XA START 'demo01';
Query OK, 0 rows affected (0.00 sec)

mysql55-tokudb6> INSERT INTO xatest(name) VALUES('demo01');
Query OK, 1 row affected (0.01 sec)

mysql55-tokudb6> SELECT * FROM xatest;
+----+--------+
| id | name   |
+----+--------+
|  3 | demo01 |
+----+--------+
1 row in set (0.00 sec)

  [Read more...]

Since the famous conjecture by Eric Brewer and proof by Nancy Lynch et al., CAP has given the world countless learned discussions about distributed systems and many a well-funded start-up.  Yet who truly understands what CAP means?  Even a cursory survey of the blogosphere shows profound disagreement about the meaning of terms like CP, AP, and CA in real systems.  Those who disagree on CAP include some of the most illustrious personages of the database community.

We can therefore state with some confidence that CAP is confusing. Yet this observation itself raises deeper questions.  Is CAP merely confusing?  Or is it the case that as with other initially accepted but

  [Read more...]

This is a cross-post from the MySQL Performance Blog. I thought it would be interesting to users of PostgreSQL, Redis, Memcached, and $system-of-interest as well.

For about the past year I’ve been formulating a series of tools and practices that can provide deep insight into system performance simply by looking at TCP packet headers, and when they arrive and depart from a system. This works for MySQL as well as a lot of other types of systems, because it doesn’t require any of the contents of the packet. Thus, it works without knowledge of what the server and client are conversing about. Packet headers contain only information that’s usually regarded as non-sensitive (IP address, port, TCP flags, etc), so it’s

  [Read more...]

What's in a datatype then? A MySQL SMALLINT? A C int? An Oracle BLOB? One thing is for sure, they are not very well standardized, not even within the same environment, and much less so across them. And what does it means, really? When should I use a BLOB, when a BINARY VARCHAR and when to use a long long?

A datatype defines many attributes:

• What data I can store in it: Only numbers? Printable alaphanumeric characters? Unicode? Binary data? An object?
• What I can do with a value of that particular type and how does it behave? Can I concatenate two values? Add them (that is NOT the same as concatenate! The same operator (+) may be used, but it's not the same thing!)? Save it to a file?
• How is it stored and represented in technical terms. And this can be argued, a C int doesn't define how it is stored on disk, but a MySQL INT

  [Read more...]

Selena and I gave a talk on the various issues of running databases “in the cloud” at the recent linux.conf.au in Ballarat. Video is up, embedded below:

If you are interested in NoSQL databases (or maybe not) perhaps you have seen the anonymous "warning" about using MongoDB.   It concludes with the following pious request:

  Please take this warning seriously.

Now there are a lot of great resources about data management on the web but the aforementioned rant is not one of them.  If you plan to write technical articles and have people take them seriously, here are a few tips.

Sign your name.  Readers are more impressed when they see you are not afraid to stand behind your words. 

Explain what problem you were trying to solve.  Otherwise uncharitable readers might think you just started pumping information into a new database without thinking about possible consequences and now

  [Read more...]