Restaurant Scalability

Could pro-waitering serve up some lessons on web scalability? Observing peak hour dining at a New York restaurant gave us some insight.

I was dining at a restaurant the other day with friends. It was a warm and cozy place, nicely decorated with a long, narrow dining room.  The food was scrumptious, yet we were getting increasingly frustrated by the service as the night went along.

With some waiting experience behind me, I could immediately see the problem. The waiters, probably through lack of experience, were making the mistake of doing one thing at a time.  They would go to a table, respond to one

  [Read more...]

Often I think about challenges that organizations face with “Big Data”.  While Big Data is a generic and over used term, what I am really referring to is an organizations ability to disseminate, understand and ultimately benefit from increasing volumes of data.  It is almost without question that in the future customers will be won/lost, competitive advantage will be gained/forfeited and businesses will succeed/fail based on their ability to leverage their data assets.

It may be surprising what I think are the near term challenges.  Largely I don’t think these are purely technical.  There are enough wheels in motion now to almost guarantee that data accessibility will continue to improve at pace in-line with the increase in data volume.  Sure, there will continue to be lots of interesting innovation with technology, but

  [Read more...]

Reading yesterday that the NSA has submitted a proposal to Apache to incubate their Accumulo platform.  This, according to the description, is a key/value store built over Hadoop which appears to provide similar function to HBase except it provides “cell level access labels” to allow fine grained access control.  This is something you would expect as a requirement for many applications built at government agencies like the NSA.  But this also is very important for organizations in health care and law enforcement etc where strict control is required to large volumes of privacy sensitive data.

An interesting part of this is how it highlights the acceptance of Hadoop.

  [Read more...]

Apache and syslog-ng

While logging to a database back-end has its benefits, the setup as it stands leaves us wanting. Some applications, such as Apache, do not log via syslog-ng by default. The good news is that this can be easily remedied, and there are a couple of different ways of doing this. First, the less good way:

Method #1: Changing the Apache configuration file.

First, we need to setup syslog-ng appropriately by creating a new source for apache, such as the following:

source s_apache {
 unix-stream("/var/log/apache2/apache_log.socket"
 max-connections(512)
 keep-alive(yes));
 };

log { source(s_apache); destination(d_pgsql); };

  [Read more...]

Database Administration and Management is as important under MySQL as it is under other enterprise database platforms such as Oracle or SQL Server.  Be proactive with your database operations, and avoid outage or loss of your most crucial data.

• MySQL Management and administration
• MySQL Performance, Optimization & tuning
• Remote Database Management
• Database monitoring & metrics collection
• MySQL Security Auditing
• Migrating applications and data to MySQL
• Migrating MySQL to the Amazon cloud
• Migrating MySQL to Amazon RDS
• Amazon RDS management & administration
• MySQL scalability
• Amazon RDS scalability
• MySQL High Availability
• MySQL Disaster Recovery
• Database Management & replication for highly available architectures
• Database security in the cloud

Skype or call us in NYC at +1-212-533-6828

MySQL Tuning

This is another section that is broader than one would first imagine.  There’s a reason that many large organizations employ dedicated database administrators.  That said, this doesn’t prevent the average sysadmin from making some changes to enhance performance on their database.

The easiest way to start on this is to utilize a script to automatically check your configuration options and make suggestions based on status variables MySQL sets.  I’ve had good luck with a script called mysqltuner.pl.  You can visit the project page at GitHub here:

  [Read more...]

Read the original article at 5 Tips for Better Database Change Management

Deploying new code that includes changes to your database schema doesn't have to be a process fraught with stress and burned fingers. Follow these five tips and enjoy a good nights sleep.

1. Deploy with Roll Forward & Rollback Scripts

When developers check-in code that requires schema changes, that release should also require two scripts to perform database changes. One script will apply those changes, alter tables to add columns, change data types, seed data, clean data, create new tables, views, stored procedures, functions, triggers and so forth. A release should also include a rollback script, which would return tables to their previous state.

This idea of

  [Read more...]

1. This page or area of the website is very slow, why?

There are a lot of components that make up modern internet websites, and a lot of places to get stuck in the mud.  Website performance starts with the browser, what caching it is doing, their bandwidth to your server, what the webserver is doing (caching or not and how), if the webserver has sufficient memory, and then what the application code is doing and lastly how it is interacting with the backend database.

With all this complexity, it's no wonder so many sites struggle.  Typically these types of analysis start with some load testing, to stress test your setup, so you can watch for leaks.  Then

  [Read more...]

Amazon EC2 and cloud computing offer great promise for startups to ramp up their online presence quickly.  Navigate those challenges with an strong partner.  We bring 20 years experience to the table with each new client.

• Scaling Web Applications
• MySQL High Availability in Amazon EC2
• Amazon Multi-AZ Deployments
• Amazon RDS Deployments
• Migrating to Amazon EC2
• Migrating to MySQL
• Managing Backups and Disaster Recovery in the Cloud
• Horizontal Scalability of MySQL on EC2
• Horizontal Scalability on Cloud Hosted Servers
• Evaluating Cloud Providers
• Evaluating MySQL

  [Read more...]

Conor O'Mahony over at IBM wrote a good post on a favorite topic of mine “The Future of the NoSQL, SQL, and RDBMS Markets”.  If this is of interest to you then I suggest you read his original post.  I replied in the comments but thought I would also repost my reply here.

-----------------------------------------------------------------------------------------------

Hi Connor, I wish it was as simple as SQL & RDBMS is good for this and NoSQL is good for that.  For me at least, the waters are much muddier than that.

The benefit of SQL & RDBMS is

  [Read more...]

My friends over at IA Ventures are looking both for an Analyst and for an Associate to their team.  If Big Data, New York and start-ups is in your blood then I can’t think of a better VC to be involved in. 

From the IA blog:

"IA Ventures funds early-stage Big Data companies creating competitive advantage through data and we’re looking for two start-up junkies to join our team – one full-time associate / community manager and one full time analyst. Because there are only four of us (we’re a start-up ourselves, in fact), we’ll need you to help us investigate companies, learn about industries, develop investment theses, perform internal operations, organize

  [Read more...]

In life there are really two major types of data analytics.  Firstly, we don’t know what we want to know – so we need analytics to tell us what is interesting.  This is broadly called discovery.  Secondly, we already know what we want to know – we just need analytics to tell us this information, often repeatedly and as quickly as possible.  This is called anything from reporting or dashboarding through more general data transformation and so on.

Typically we are using the same techniques to achieve this.  We shove lots of data into a repository of some from (SQL, MPP SQL, NoSQL, HDFS etc) then run queries/ jobs/ processes across that data to retrieve the information we care about.  

Now this makes sense for data discovery.  If we don’t know what we want to know, having lots of data in a big pile that we can slice and dice

  [Read more...]

It is a constant, yet interesting debate in the world of big data.  What scales best?  OldSQL, NoSQL, NewSQL?

I have a longer post coming on this soon.  But for now, let me make the following comments.  Generally, most data technologies can be made to scale - somehow.  Scaling up tends not to be too much of an issue, scaling out is where the difficulties begin.  Yet, most data technologies can be scaled in one form or another to meet a data challenge even if the result isn’t pretty. 

What is best?  Well that comes down to the resulting complexity, cost, performance and other trade-offs.  Trade-offs are key as there are almost always significant concessions to be made as you scale up.

A recent example of mine, I was looking at scalability aspects of MySQL.  In particular, MySQL Cluster

  [Read more...]

The MySQL database is a crucial part of a wide variety of products, particularly web applications. Naturally, it is very important to monitor the health status of MySQL.  However, there is constant disagreement on which of the many MySQL status variables provide the best overview on MySQL health status and indicate that something is not right with a server.

It certainly depends on what your application does – tuning read performance is different than optimizing write operations and everything changes when you have a cluster. The average user can use small subset of variables while advanced user want to get more detailed picture of the situation. So there cannot be one set of “magic variables” to quietly optimize every situation. However, it is possible to have

  [Read more...]

 

MySQL is a powerful open-source database.  With more and more database driven applications, people have been pushing MySQL to its limits.  Here are 101 tips for tuning and optimizing your MySQL install.  Some tips are specific to the environment they are installed on, but the concepts are universal.   I have divided them up into several categories to help you with getting the most out of MySQL:

 

MySQL Server Hardware and OS Tuning:

1.

  [Read more...]

Well as predicted, with Aster Data recently being picked up by Teradata most of the key new generation MPP distributed analytics vendors have been acquired (Aster Data, Vertica, Netezza & Greenplum).  This had to happen and was expected to happen.  The MPP Analytics startup “revolution” is over and these technologies will now be integrated into the mainstream.

So what’s next?  As we now, if you are a massive multi-national software company it is a lot less risky to incrementally innovate and leave the development of “game changing” technologies to startups that can be acquired after

  [Read more...]

There are so, so many big data platforms in play at the moment it can be confusing for developers to know where to start.  For startups it used to be simple, MySQL, but dust clouds were created when all the NoSQL platforms started to crash the party 18 months or so ago.  But I do see the dust begin to settle and we are starting to see some market “leaders” appear.  A very unscientific approach is to list the technologies I hear about in the “big data startup” world on a daily basis.  These are, in no particular order:

• MySQL - yes it is still very much hanging in there despite the Oracle acquisition.  MySQL has been helped by technologies such as AWS RDS and Xeround making it more digestible for big data startups who want

  [Read more...]

I have been busy travelling recently but thought I would jot down a couple of NoSQL myths that are fresh in my head from my recent discussions.

• Twitter use Cassandra internally but have not migrated their tweet store, despite their earlier plans to.  For now tweets are still stored in MySQL.
• Despite the widely accepted view that the use of Cassandra led to Diggs issues a couple of Digg engineers have apparently discounted this.
• Despite the widely accepted view that NoSQL databases all use eventual consistency this is not so.  HBase, for example, offers full consistency.
• Despite the widely accepted view that NoSQL is only

  [Read more...]

“NoSQL”, for lack of better name, is a generic term that describes any data management system that does not use SQL as a query interface.  Generally this means any data management system that is non-relational, but the term also has also been stretched as far to include the boundaries of what constitutes a data management system at all (such as Hadoop).

Early on (a couple of years back in NoSQL time) when the term was coined I think the positioning was much more aggressive, but more recently this has been softened so now NoSQL is commonly quoted as meaning of “Not only SQL” or “next generation databases” (whatever that means).  The common message you get now is something along the lines of NoSQL systems are

  [Read more...]

With IBM intending to acquire Netezza the predicted consolidation in the distributed analytics market is well underway.  Recent deals include EMC/Greenplum Teradata/Kickfire and now IBM/Netezza.  A good breakdown of this deal is on Curt’s blog.  There is still more to go of course with one of the crown jewels, Vertica, still ripe for the picking. 

What this indicates is that MPP analytics has moved from the innovative edge into the mainstream market and now the more risk

  [Read more...]

Back in 2008 Stonebraker & DeWitt published a paper and associated blog post titled “MapReduce: A major step backwards”.  Their key points being Map Reduce is:

A giant step backward in the programming paradigm for large-scale data intensive applications

A sub-optimal implementation, in that it uses brute force instead of indexing

Not novel at all — it represents a specific implementation of well known techniques developed nearly 25 years ago

Missing most of the features that are routinely included in current DBMS

Incompatible with all of the tools DBMS

  [Read more...]

I will be at VLDB 2010 next week.  If anyone on this blog is attending and wants to catch up to discuss start ups and innovation in DB, NoSQL, Big Data etc drop me a line and I will try to meet up.

Cassandra is one of the most interesting NoSQL platforms at the moment.  And by most interesting what I really mean is the most clearly justifiable.  Some NoSQL platforms offer new data models, improved query interfaces and/or good single node performance through relaxed consistency models.  As a database guy however, the justification for throwing out the RDBMS baby and bathwater is still difficult at this point as NoSQL platforms tend to be highly focused in one aspect of data management, and very immature in all other areas.  Cassandra is somewhat different as it is more mature in a number of key areas (albeit still immature in others).  Areas that can make Cassandra more justifiable for the right project, when compared with a more  [Read more...]

I work in all markets of the database industry, from web & startup through the largest and most established enterprises.  And to be completely honest, the name Ingres has not come up in conversation very much at all.  10 years ago maybe more often, but recently not all that much.  But Ingres has been quietly ticking away.  Despite being largely off the radar, they still have a sizable and loyal customer base, global offices and a focused & dedicated management team.  And importantly they have an open source business model which actually appears to be working.

I wrote last year that their "behind the

  [Read more...]

I have noticed a definite increase in NoSQL buzz over the last few months.  This is partly confirmed by Google Trends, this service shows data relating to how search topics rank:

The last couple of months has seen a dramatic rise in both the number of searches and also the number of news items relating to NoSQL. 

But the traditionalists need not yet fret, interest in NoSQL is yet but a blip on the data management radar, as demonstrated by this compairson between NoSQL and MySQL search rankings:

  [Read more...]

Image by Aranda\Lasch via Flickr

One of my favorite terms at the moment is “Big Data”.  While all terms are by nature subjective, in this post I will try and explain what Big Data means to me.

So what is Big Data?

Big Data is the “modern scale” at which we are defining or data usage challenges.  Big Data begins at the point where need to seriously start thinking about the technologies used to drive our information needs.

While Big Data as a term seems to refer to volume this isn’t the case.  Many existing technologies have little problem physically handling large volumes (TB or PB) of data.  Instead the Big Data



  [Read more...]

Image via Wikipedia

Oracle has published their promises which have reportedly gone a long way to appeasing the EU, so the likely outcome is the takeover of Sun will be approved in January.

My own personal opinion has been the anti-competitive stance really didn’t hold much water.  Reading Oracle’s promises, none appear very extreme (largely agreeing to maintain the status quo) which would lead you to question why it has taken so long to sort out.  But importantly for getting this resolved they are a concession by Oracle and a win for the EU.



  [Read more...]

Cassandra is fast emerging as one of the key NoSQL databases.  While we often express that the point of NoSQL is to offer more choice than an “RDBMS” hammer for every nail, there are practical reasons why a small number of stack technologies gain dominance and others circle on the sidelines.

Cassandra has already ticked many of the boxes needed to shoot it into the stratosphere as a widely used, default database platform.  Especially so in the web world where high scalability, high availability, open source and being proven by a bigger fish all matter.  Specifically Cassandra has:

• The ability to scale across many nodes
• The ability to scale to many hundreds of

  [Read more...]

Last week I spent some time speaking with Kevin Weil, head of analytics at Twitter. Twitter, from a technology perspective, has had a bit of a hard time due to their stability issues in their early days.  Kevin was keen to point out that he feels this was due to the incomparable growth Twitter was experiencing at the time and their constant struggle to keep up.  Kevin was also keen to show that Twitter prides themselves on striving for engineering excellence, the creation & contribution to new technologies and generally assisting in pushing the boundaries forward.  Our conversation naturally centered on analytics at Twitter.

Twitter, like many web 2.0 apps, started life as a MySQL based RBDMS application.  Today, Twitter is still

  [Read more...]

Image by Getty Images via Daylife

Like many I was disappointed, yet not surprised, that the EC formally logged their objection to Oracle’s acquisition of Sun on account of MySQL a few days back.  And we also hear today that Oracle will be stating their position in Brussels on the 25th of this month.  To me this case has odd from the onset and as it goes on it is just getting odder.  And of course this all seems to be occuring at immense cost to Sun, Oracle and MySQL themselves.

There are several reasons why this is odd.  One of the key

  [Read more...]