I see that “Big Data” has become the new buzzword with a spike of hype around it. Everyone’s jumping on it. Companies are eager to promote their products as “Big Data,” just as they were eager to be associated with Web 2.0, Service-Oriented Architectures, and all the rest. Predictably, there’s basically zero agreement on what it means.

I’ve seen “Big Data” mentioned in the context of 1TB, which I think is rather moderate sized. But worse yet, I’ve seen 100GB labeled Big Data. I’ve even seen 5GB labeled Big Data. No links — I don’t want to draw attention to them.

I don’t know what Big Data is, but the stick-of-gum-sized flash drive in my pocket holds 16GB. It’s pretty Small. I mean, I forget it’s even there — it’s definitely not Big. I don’t

For a long time I though the AutoMySQLBackup project had died.  Then, after I created many updates, in November of last year I saw some life. Meanwhile, I have continued to update the code and track what people have requested.   Low volume project are sometimes hard to keep alive. People think the project is dead and don’t want to use code that will not see updates.  Then if you make updates the authors may not come alive often enough to make the changes.

I have updated AutoMySQLBackup again. This time I have included the ability to make backups at the table level by including the table name, with the database name, in the list of databases to be backed up.  Adding a ‘.’ between the database name and the table name (data.table) will backup only that table in the database.  (IE

WHAT TIME IS IT?

This post started with a simple question: “Does the function NOW() get executed every time a row is examined?” According to the manual,  “Functions that return the current date or time each are evaluated only once per query …. multiple references to a function such as NOW() … produce the same result. …. (However,) as of MySQL 5.0.12, SYSDATE() returns the time (the row is) executes. “

• CURDATE() returns the current date.
• CURTIME() returns the current time.
• UTC_DATE() returns the current UTC date.
• UTC_TIME() returns the current UTC time.
• NOW() return the current date and time.
• UTC_TIMESTAMP() returns the current UTC date and time.
• SYSDATE() returns the

HandlerSocket is cool. But, it turns out there are a few issues.

Justin Swanhart points out HandlerSocket currently lacks atomic operations . Since HandlerSocket uses different connections for reading and writing, you can’t increment/decrement a value without creating a race condition.

Still, the idea of skipping SQL interpretation and just reading the data you know you want is a great one.  Writing data might even be better. But being able to use both SQL and NoSQL could be really wonderful.  What if we could use complex queries to update complex tables and pluck values out as needed.  For example, queries to analyze current weather conditions and produce forecasts that we could then retrieve via a location key? What about updating current condition data

In my last post I installed the HandlerSocket plugin into MariaDB and tested it.  Like the last post these examples are done with Linux CentOS 5.5.

HandlerSocket some with Perl code for the Net:: module group. When you build and install the HandlerSocket plugin it does not build the Perl module. I looked and it is not included in CPAN.

If you have already downloaded (git) and installed the plugin, you can install the Perl module by:

cd perl-Net-HandlerSocket
perl Makefile.PL
make
make test
make install

Most of the sample apps I’ve found do little more the prove it works.  I found the protocol (API) is very simple.  This makes it fast but may give you some trouble coding for it.  I created test table and my own version of the example code.

CREATE TABLE `user` (
 `user_id` int(10) unsigned NOT NULL,
 `user_name` varchar(50),

There have been a few great blog posts recently from MySQL bloggers about Postgres, and vice versa, with good comments and follow-on from the real experts in both systems. I think this is wonderful. Learning how other databases solve hard problems is highly educational, especially because ACID databases face some of the hardest problems in computing. Making MySQL better is good for PostgreSQL. The reverse is just as true. And we should also be learning from SQLite, and CouchDB, and others who have overcome tough technical hurdles, built successful companies, created thriving and enthusiastic communities, or whatever their success has been.

Related posts:

Web Operations - Keeping the Data On Time

VIEWING THE DATA

You’ve seen in the last few postings how to find, import and use GIS data with MySQL. But, the ‘G’ in geographical includes GRAPHIC and a little eye candy never hearts your understanding of the curves.

Everyone knows of  map.google.com.  Mark McClure has a great site explaining the use of Google’s APIs.  He explains how to generate interactive maps with labels for places and routes indicating driving directions. Many of Mark’s example explain exactly the sorts of things I’ve wanted to do with maps for years.  Like Cycling routes on Google Maps.

On DBA Survivor, Thomas LaRock blogged about “Finding a Good DBA“.  This is a reply.

I mostly agree what what LaRock said. Managers believe databases fall into the same category as email servers and domain name servers. System administrators are expected to take care of them all. And, having been a SysAdmin myself, I know as long as the thing is working they’re good.

I was also an application developer for a number of years as well. From that point of view, you are working as fast as you can while still trying to make your code run as fast as it can. You don’t have full access to the SQL servers. You might test a few different SQL queries to see witch is faster, but you can’t get access to the configuration files.  Asking the SysAdmin to

The “DATA”

“The wonderful thing about standards is that there are so many of them to choose from.” – Grace Hopper

A Shape file is the most common format for GIS vector data and just about every GIS program can use them.  Unfortunately not all GIS data come in a shape file format. An E00 (E-zero-zero) file is the file format used by Environment Systems Research Institute’s (ESRI). ESRI is the Photoshop of the GIS workplace. E00 files are used by ArcInfo and ArcGIS Explorer.  These are the two most common file formats.

MyTOP is a console-based (non-gui) tool for monitoring the threads and overall performance of a MySQL created by Jeremy Zawodny.

For months now I have been adding updates to mytop. When I started using version 1.6 it worked but didn’t return some data fields. After fixing these bugs I began to ideas for improvements. Here is a quick list of what I have done.

• New ‘!’ command to force fix replication errors.
• Display rows sorted per second.
• Added new ‘Cmd’ column to display the state of the query along with the statement.
• New ‘M’ command to change the [Mode] to

There’s lots of buzz lately about the so-called “open-core” business model of Marten Mickos’s new employer. But this is nothing new. Depending on how you define it, InnoDB is “open-core,” and has been for a long time. The InnoDB Hot Backup (ibbackup) tool was always closed-source. Did anyone ever cry foul and claim that this made InnoDB itself not open-source, or accuse Innobase / Oracle of masquerading as open-source? I don’t recall that happening, although sometimes people got suspicious about the interplay between the backup tool and the storage engine. Generally, though, the people I know who use InnoDB Hot Backup have no gripes about paying for it.

What is the

I have old applications that need to read (and write) MyISAM tables that themselves receive lots of bulk updates. Time to try MySQL-Proxy.

MySQL Proxy is light on documentation and very few people written anything about working.   Most of what I have read says MySQL-Proxy is not ready for prim time.  I have hope so I had to give it a try.

I started with thee VMware servers. I setup one master and two read only slaves.   I tested the replication with mysqlslap from another independent server and it worked fine.  The slave never ran more then a second behind.

I downloaded

Dr. Richard Hipp gave a talk at Southeast Linux Fest today on choosing an open-source database. He thinks that NoSQL is not a very good name for the new databases we’re seeing these days, so he proposed a new name: postmodern databases. Why postmodern?

• The absence of objective truth
• Queries return opinions, not facts

I thought this was the best proposal I’ve heard for an alternative to the NoSQL moniker. And this is not bashing — the absence of objective truth can actually be an enabling quality, not necessarily a drawback. There’s a lot to compliment about the new databases, and calling them NoSQL is really a disservice — like calling a car a horseless carriage.

Related posts:

A common problem I see people running into when using a cloud computing service is the trap of under-provisioning. There’s a chain effect that leads to this result: 1) people don’t understand how virtualization works, and therefore 2) they don’t realize how much of a computing resource they’re really buying, so 3) they assume they are entitled to more than they really are, and 4) they under-provision. A few other causes and effects come into play here, too. For example, the choice to use the cloud is sometimes founded on economic assumptions that frequently turn out to be wrong. The cloud service looks more economically attractive than it really is, due to under-provisioning.

Let’s get back to this idea that people under-provision. How do I know that’s happening? I’ll use anecdotal evidence to illustrate.

A lot of conversations with a few different people I respect (no links, sorry) have coalesced some thoughts about these newly popular “non-relational” datastores. I wanted to point out an aspect I’m not sure is very clear in the hot-topic department. This is about what happens when managers learn that their developers or operations team have installed some new technology in their systems without them knowing it.

Lest anyone think that this happens only in a poorly-managed company, I can attest that it happens everywhere, all the time. Remember Marten Mickos’s favorite story about salespeople asking prospects if they used MySQL, the managers saying absolutely not, and the developers contradicting them?

The moment of discovery is unpleasant for the manager, but everything leading up to it was a joy for the programmer. He

Tweet

Is MariaDB really a drop in replacement for MySQL?   I’m running CentOS 5.4.  What happens if…

And the install start mysql up again.  Wow.  That’s “Drop in”.

If your

Do you have MyISAM tables you reload with new data?

Do your queries, using that table, get blocked because the table is locked?

Do the waiting queries create idle connections slowing down the table load?

Do you wish you could just replace the table?

Years ago I was told you can replace CSV tables by simply replacing the CSV file. I figured this would also be true of a MyISAM file and it is. I use this perl script to replace MyISAM tables forcast and current observation weather data. The processing and tables are created on another computer. Weather forecasting is CPU and database expensive. I then copy (rsync) the files to the production system and run this script.

#!/usr/bin/perl
################################################################################

A now-famous quote that I probably don’t need to attribute: “First they ignore you, then they laugh at you, then they fight you, then you win.”

Where is Drizzle in this lifecycle? I’ve been hearing and reading some comments to the tune of “those Drizzle guys think it’s easy to rip MySQL stuff out and start over, wait till they see how hard it’s going to get when the real world sinks in.” Maybe, maybe. But maybe not, too. Maybe not.

I’ve seen more than one software project that was belittled as “never gonna amount to anything, save your time” and went on to do quite well. Never underestimate the power of a handful of passionate and talented people. I personally feel that Drizzle has a bright future.

Related posts:

Tweet

I read in his blog Jonathan Schwartz is leaving Sun/Oracle.  Jonathan was the CEO, Sun Microsystems, Inc.

Jonathan has been the force at Sun driving open source.  His voice will not be heard at Oracle so I’m wondering if this will make a change in the way projects like MySQL, Open Solaris and Open Office will be managed.

Much has been made of certain limitations of MySQL’s query optimizer (“planner”). Subqueries, lack of sort-merge joins, and so on. This is not MySQL-bashing and no one should be offended. Some people have worked to make things better, and that code is in branches that were intended for future releases. But if that work were available right now, would you trust it?

This question is important because the optimizer is complex and full of compromises and black magic. Even minor changes occasionally have weird edge cases that cause a regression in some workload. Are major changes trustworthy?

I’ll give a specific example. In version 5.0, MySQL gained the ability to use more than one index for a query. This is called index_merge in EXPLAIN, and sometimes people think it’s the best thing ever. In practice, I can

(Re-posted from Florian’s blog.)

MySQL replication support for the Pacemaker cluster manager (the stuff that we explained in this webinar) has made it into the Linux-HA resource agents default branch. If you are interested in testing — and you should! — please read the extended announcement. Feedback is extremely welcome on the linux-ha-dev mailing list.

We are expecting to release this as part of resource-agents 1.0.4, in late May/early June.

I got a couple of questions and comments about OpenSQL Camp in the past week, and I thought it would be worth noting down the history, because I think there is some difference in perception and memory about this series of events. The following is only my point of view.

What is OpenSQL Camp?

I can say what I had in mind when I created the original event, but this is bigger than me, so I don’t get to dictate anything. I wanted a free technical event created entirely by and for a community of open-source databases, in an inclusive sense. Not created or heavily influenced by someone employed by a corporation whose job title includes the word “Community,” but really by a community themselves. There’s nothing wrong with Community So-And-So employed at a corporation, but they are by nature a liaison with

Brian got interviewed by O’Relly recently, and part of it quoted him as saying this:

When everything doesn’t fit onto a computer, you have to be able to migrate data to multiple nodes. You need some sort of scaling solution there… MapReduce works as a solution when your queries are operating over a lot of data; Google sizes of data. Few companies have Google-sized datasets though. The average sites you see, they’re 10-20 gigs of data.

Users shouldn’t need to put that data onto multiple machines anyway. In fact, I don’t think we need a multi-machine solution for the common case at all. We need software that can scale up with today’s hardware. 37signals likes to

It seems that a lot of people equate non-SQL databases with non-relational-ness, or malign the word relational. This is pretty much pure ignorance. If you’ve ever uttered a sentence that includes the phrase “…non-relational database…” then I have two suggestions for you.

Yes, I’ve started yet another blog. You think I would learn.

This time it’s bout MySQL. The database we …

Tweet

Cary Millsap has a concise, readable paper on performance. Anyone involved in database performance optimization should read it. Cary’s writing has heavily influenced the mk-query-digest tool for analyzing MySQL/PostgreSQL/Memcached/HTTP query performance, and I think you’ll get a lot more from mk-query-digest if you read this paper — and you should also read his book, reviewed here. It’s one of the top books on my Essential Books List.

Related posts:

InfiniDB has a sensible Enterprise/Community release process, which seems similar to what I suggested for MySQL. Its simplicity also stands in stark contrast to MySQL’s new release policy, which is hard to understand and has been confusing people.

Related posts:

One of MySQL’s notable projects was splitting the product into two editions: Enterprise Edition and Community Edition. This move alienated many in the community, and failed to create meaningful differentiation on either side, even with a team of people beating the community bushes for “contributions.” The net differentiation was ultimately Jeremy Cole’s SHOW PROFILES functionality, which made Community better than Enterprise. Sun put less effort into making this split work, and eventually they abandoned it.

But that could change under Oracle’s stewardship. Oracle’s

Ever been around a group of people discussing some technology and heard Cool-Whip phrases like this?

It’s not about MySQL versus PostgreSQL, it’s about using the right tool for the job.

Or how about this one?

You need to take the important factors into account before you decide whether [hot new fad] or [trusty old solution] is best suited for your application.

Both are signs that someone might be trying to sound important. In situations like this, I’ve noticed that the people I look up to usually don’t make weighty-sounding statements about other people’s systems. They talk about what they are qualified to talk about: either they say something about their own systems, or if it’s warranted and invited, they ask intelligent questions about other