The last 24 months has seen the introduction of Map/Reduce functionality into the data processing arena in various forms.  Map/Reduce is a framework for developing scalable data processing functionality, and was popularized by Google (see this earlier post).

Pure players like Hadoop are starting to find their own niche, helped by organizations such as Cloudera.  However there has been a number of for & against arguments relating to Map/Reduce functionality inside the database.

These arguments are now really serving a moot point.  Customers have recognized value in Map/Reduce prompting some (b)leading edge database vendors to introduce such …

A bit of fun for a Sunday.  Below is the list of my top 10 interesting companies in Data Management right now.  Interesting to me means doing new stuff and being somewhat disruptive, or have a “watch and see” quality about them.  Note this is about companies not data management applications. 

While I find a bunch of other data management applications interesting (PNUTS, Cassandra, Redis etc) these aren’t really encapsulated in a company with a go to market strategy.

10gen - They are making interesting noises not sure about delivery yet
Amazon – SimpleDB is neat, but not a grown up data platform yet
Aster Data – Doing funky things with Map/Reduce
GroovyChannel – Are they nuts, they have to change …

A great post from Jonathan Ellis on "Why you won't be building your killer app on a distributed hash table"

Scroll down to see my comment.  I agree a simple DHT is not a suitable solution for mainstream data management issues, but also I think that there is a gap between RDBMS and DHT.

• Introduction
• The Problem with the Relational Database (Part 1 ) –The Deployment Model

I met with a friend of mine in New York recently who runs a credit card processing end-point company.  They specifically built their business around a non-relational database platform and feel they would have major issues had they chosen to build their business around a traditional …

• Introduction
• The Problem with the Relational Database (Part 2) – Predictability

This is the first detail post in a series I am doing focusing on the issues that exist today with the Relational Database.  This first post is on the deployment model.  It could be argued that this isn’t directly related to the “relational database” but rather is an implementation model problem.  I disagree with this as many characteristics of the relational database lead to the deployment model …

• The Problem with the Relational Database (Part 1 ) –The Deployment Model
• The Problem with the Relational Database (Part 2) – Predictability

The relational database has been the core mechanism for structured data storage and retrieval for the past 30 years.  My career so far has focused around the relational database, whether it be from a development, administrator or investment perspective.  In all this time the RDB has been the best generic option available for developers …

Image via CrunchBase

Last week I spent some time talking to Michael “Monty“ Widenius (co-founder of MySQL) about his decision to leave Sun.  The bulk of the forward looking aspects of this conversation are going into a post I am preparing for Read Write Web.  This should be out sometime next week. 

While I didn’t want to focus what happened in the past too much in this article I thought I would post some of my own personal assessments I took from my conversation with Monty here.  They are as follows:

• As almost everyone who meets Michael concludes, Michael is a nice guy.  He is transparently open and a pure believer of a common …

Recently, a lot of new non-relational databases have cropped up both inside and outside the cloud. One key message this sends is, "if you want vast, on-demand scalability, you need a non-relational database".

If that is true, then is this a sign that the once mighty relational database finally has a chink in its armor? Is this a sign that relational databases have had their day and will decline over time? In this post, we'll look at the current trend of moving away from relational databases in certain situations and what this means for the future of the relational database.[more]

You may not realize it, but the data analytics market is buzzing. There are new vendors emerging, new products popping up, new deals being done, and several new strategies being pursued. Vendors are predominately chasing big data, with battles lines being drawn by solution providers that cater to between roughly 100 TB and 10 PB data sets. The battle was inevitable because the world is producing data at a phenomenal rate, and we have an increasing need to analyze them within shorter time frames. In this post we analyze one of these vendors, Kickfire.

Yet while the big names in town are capturing the headlines, in reality only a small percentage of businesses today need to be able to analyze petabytes of data. Today, the rest of us are more likely to deal with analytic data sets in the 50 GB to 3 TB range.

Kickfire is interesting because it has decided to let the other vendors fight it out for the massive data …

So it’s that time of year again when everyone puts out their predictions for the year ahead.  I think predictions are a bit of a waste of time because to be interesting predictions have to be big, but a year really isn’t all that long so actual changes over the course of 2009 are likely to be just small progressions.  So instead I have been thinking about the top issues that we face heading into 2009 and here is my Top 10 list for issues in Data Management.  In this post I avoid offering solutions to these issues, while I have several ideas on solutions these can be the subject of subsequent posts.

10 - Limits on Scalability

While scalability is on my list it is at number 10 because against popular belief, scalability is only an issue for a very small number of data based applications.  Almost all data based applications in use today can be scaled without major issue by increasing the underlying …