Planet MySQL

Liveblogging at Confoo: Blending NoSQL and SQL

Sheeri K. Cabral — Thu, 11 Mar 2010 16:11:53 +0000

Persistence Smoothie: Blending NoSQL and SQL – see user feedback and comments at http://joind.in/talk/view/1332.

Michael Bleigh from Intridea, high-end Ruby and Ruby on Rails consultants, build apps from start to finish, making it scalable. He’s written a lot of stuff, available at http://github.com/intridea. @mbleigh on twitter

NoSQL is a new way to think about persistence. Most NoSQL systems are not ACID compliant (Atomicity, Consistency, Isolation, Durability).

Generally, most NoSQL systems have:

Denormalization
Eventual Consistency
Schema-Free
Horizontal Scale

NoSQL tries to scale (more) simply, it is starting to go mainstream – NY Times, BBC, SourceForge, Digg, Sony, ShopWiki, Meebo, and more. But it’s not *entirely* mainstream, it’s still hard to sell due to compliance and other reasons.

NoSQL has gotten very popular, lots of blog posts about them, but they reach this hype peak and obviously it can’t do everything.

“NoSQL is a (growing) collection of tools, not a new way of life.”

What is NoSQL? Can be several things:

Key-Value Stores
Document Databases
Column-oriented data stores
Graph Databases

Key-Value Stores

memcached is a “big hash in the sky” – it is a key value store. Similarly, NoSQL key-value stores “add to that big hash in the sky” and store to disk.

Speaker’s favorite is Redis because it’s similar to memcached.

key-value store + datatypes (list, sets, scored sets, soon hashes will be there)
cache-like functions (like expiration)
(Mostly) in-memory

Another interesting key-value store is Riak

Combination of key-value store and document database
heavy into HTTP REST
You can create links between documents, and do “link walking” that you don’t normally get out of a key-value store
built-in Map Reduce

Map Reduce:

Massively parallel way to process large datasets
First you scour data and “map” a new set of dataM
Then you “reduce” the data down to a salient result — for example, map reduce function to make a tag cloud: map function makes an array with a tag name and a count of 1 for each instance of that tag, and the reduce tag goes through that array and counts them…
http://en.wikipedia.org/wiki/MapReduce

Other key-value stores:

Document Databases

Some say that it’s the “closest” thing to real SQL.

MongoDB – Document store that speaks BSON (Binary JSON, which is compact). This is the speaker’s favorite because it has a rich query syntax that makes it close to SQL. Can’t do joins, but can embed objects in other objects, so it’s a tradeoff

Also has GridFS that can store large files efficiently, can scale to petabytes of data
does have MapReduce but it’s deliberate and you run it every so often.

CouchDB
- Pure JSON Document Store – can query directly with nearly pure javascript (there are auth issues) but it’s an interesting paradigm to be able to run your app almost entirely through javascript.
- HTTP REST interface
- MapReduce only to see items in CouchDB. Incremental MapReduce, every time you add or modify a document, it dynamically changes the functions you’ve written. You can do really powerful queries as easy as you can do simple queries. However, some things are really complex, ie, pagination is almost impossible to do.
- Intelligent Replication – CouchDB is designed to work with offline integration. Could be used instead of SQLite as the HTML5 data store, but you need CouchDB running locally to be doing offline stuff w/CouchDB

Column-oriented store

Columns are stored together (ie, names) instead of rows. Lets you be schema-less because you don’t care about a row’s consistency, you can just add a column to a table very easily.

Cassandra – Built by Facebook, also used by Twitter
BigTable
Hypertable
HBase

Graph Databases

speaker’s opinion – there aren’t enough of these.
Neo4J – can handle modeling complex relationships – “friends of friends of cousins” but it requires a license.

When should I use this stuff?

If you have:	Use
Complex, slow joins for an “activity stream”	Denormalize, use a key-value store.
Variable schema, vertical interaction	Document database or column store
Modeling multi-step relationships (linkedin, friends of friends, etc)	Graph

Don’t look for a single tool that does every job. Use more than one if it’s appropriate, weigh the tradeoffs (ie, don’t have 7 different data stores either!)

NoSQL solves real scalability and data design issues. But financial transactions HAVE to be atomic, so don’t use NoSQL for those.

A good presentation is http://www.slideshare.net/bscofield/the-state-of-nosql.

Using SQL and NoSQL together

Why? Well, your data is already in an SQL database (most likely).

You can blend by hand, but the easy way is DataMapper:
Generic, relational ORM (adapters for many SQL dbs and many NoSQL stores)
Implements Identity Map
Module-based inclusion (instead of extending from a class, you just include into a class).

You can set up multiple data targets (default is MySQL, example sets up MongoDB too).
DataMapper is:

Ultimate Polyglot ORM
simple r’ships btween persistence engines are easy
jack of all, master none
Sometimes perpetuates false assumptions –
If you’re in Ruby, your legacy stuff is in ActiveRecord, so you’re going to have to rewrite your code anyway.

Speaker’s idea to be less generic and better use of features of each data store – Gloo – “Gloo glues together different ORMs by providing relationship proxies.” this software is ALPHA ALPHA ALPHA.

The goal is to be able to define relationships on the terms of any ORM from any class, ORM or not
Right now – partially working activeRecord relationships
Is he doing it wrong? Is it a crazy/stupid idea? Maybe.

Example:

Need	Use
Assume you already have an auth system	it’s already in SQL, so leave it there.
Need users to be able to purchase items from the storefront – Can’t lose transactions, need full ACID compliance	use MySQL.
Social Graph – want to have activity streams and 1-way and 2-way relationships. Need speed, but not consistency	use Redis
Product Listings — selling moves and books, both have different properties, products are pretty much non-relational	use MongoDB

He wrote the example in about 3 hours, so integration of multiple data stores can be done quickly and work.

PlanetMySQL Voting: Vote UP / Vote DOWN

Liveblogging at Confoo: [not just] PHP Performance by Rasmus Lerdorf

Sheeri K. Cabral — Thu, 11 Mar 2010 14:29:46 +0000

Most of this stuff is not PHP specific, and Python or Ruby or Java or .NET developers can use the tools in this talk.

The session on joind.in, with user comments/feedback, is at http://joind.in/talk/view/1320.

Slides are at http://talks.php.net/show/confoo10

“My name is Rasmus, I’ve been around for a long time. I’ve been doing this web stuff since 1992/1993.”

“Generally performance is not a PHP problem.” Webservers not config’d, no expire headers on images, no favicon.

Tools: Firefox/Firebug extension called YSlow (developed by yahoo) gives you a grade on your site.

Google has developed the Firefox/Firebug pagespeed tool.

Today Rasmus will pick on wordpress. He checks out the code, then uses Siege to do a baseline benchmark — see the slide for the results.

Before you do anything else install an opcode cache like APC. Wordpress really likes this type of caching, see this slide for the results. Set the timezone, to make sure conversions aren’t being done all the time.

Make sure you are cpu-bound, NOT I/O bound. Otherwise, speed up the I/O.

Then strace your webserver processs. There are common config issues that you can spot in your strace code. grep for ENOENT which shows you “No such file or directory” errors.

AllowOverride None to turn off .htaccess for every directory, just read settings once from your config file….(unless you’re an ISP).

Make sure DirectoryIndex is set appropriately, watch your include_path. All this low-hanging fruit has examples on the common config issues slide.

Install pecl/inclued and generate a graph – here is the graph image (I have linked it because you really want to zoom in to the graph…)

In strace output check the open() calls. Conditional includes, function calls that include files, etc. need runtime context before knowing what to open. In the example, every request checks to see if we have the config file, once we have config’d we can get rid of that stuff. Get rid of all the conditionals and hard-code “include wp-config.php”. Examples are on the slide.

His tips to change:
Conditional config include in wp-load.php (as just mentioned)
Conditional did-header check in wp-blog-header.php
Don’t call require_wp_db() from wp-settings.php
Remove conditional require logic from wp_start_object_cache

Then check strace again, now all Rasmus sees is theming and translations, which he decided to keep, because that’s the good benefit of Wordpress – Performance is all about costs vs. flexibility. You don’t want to get rid of all of your flexibility, but you want to be fast.

Set error_reporting(-1) in wp-settings.php to catch all warnings — warnings slow you down, so get rid of all errors. PHP error handling is very slow, so getting rid of errors will make you faster.

The slide of warnings that wordpress throws.

Look at all C-level calls made, using callgrind, which sits under valgrind, a CPU emulator used for debugging. See the image of what callgrind shows.

Now dive into the PHP executor, by installing XDebug.

Check xhprof – Facebook open sourced this about a year ago, it’s a PECL extension. The output is pretty cool, try it on your own site, Rasmus does show you how to use it. It shows you functions sorted by the most expensive to the least expensive.

For example, use $_SERVER[REQUEST_TIME] instead of time(). Use pconnect() if MySQL can handle the amount of webserver connections that will be persistent, etc.

After you have changed a lot of the stuff above, benchmark again with siege to see how much faster you are. In this case there is not much gained so far.

So keep going….the blogroll is very slow — Rasmus gets rid of it by commenting out in the sidebar.php file. I’d like to see something to make it “semi-dynamic” — that is, make it a static file that can be re-generated, since you might want the blogroll but links are not changed every second…..

At this point we’re out of low-hanging fruit.

HipHop is a PHP to C++ converter & compiler, including a threaded, event-driven server that replaces apache. Rasmus’ slide says “Wordpress is well-suited for HipHop because it doesn’t have a lot of dynamic runtime code. This is using the standard Wordpress-svn checkout with a few tweaks.”

Then, of course, benchmark again.

The first time you compile Wordpress with HipHop, you give it a list of files to add to the binary, it will complain about php code that generate file names, so you do have to fix that kind of stuff. There’s a huge mess of errors the first time you run it (”pages and pages”), and Rasmus had to patch HipHop (and Wordpress) but the changes in HipHop have been put back into HipHop, so you should be good for the most part.

Check out the errors, lots of them show logical errors like $foo.”bar” instead of $foo.=”bar” and $foo=”bar” instead of $foo==”bar” in an if statement. Which of course is nice for your own code, to find those logical errors.

(Wordpress takes in a $user_ID argument and immediately initializes a global $user_ID variable, which overwrites the argument passed in, so you can change the name of the argument passed in….)

You can also get rid of some code, things that check for existence of the same thing more than once. So it will take a bit of tweaking, but it’s worth it.

There are limitations to HipHop, for example:

It doesn’t support any of the new PHP 5.3 language features
Private properties don’t really exist under HipHop. They are treated as if they are protected instead.
You can’t unset variables. unset will clear the variable, but it will still be in the symbol table.
eval and create_function are limited
Variable variables $$var are not supported
Dynamic defines won’t work: define($name,$value)
get_loaded_extensions(), get_extension_funcs(), phpinfo(), debug_backtrace() don’t work
Conditional and dynamically created include filenames don’t work as you might expect
Default unix-domain socket filename isn’t set for MySQL so connecting to localhost doesn’t work

and HipHop does not support all extensions — see the list Rasmus has of extensions HipHop supports.

Then Rasmus showed an example using Twit (which he wrote) including the benchmarks. He shows that you can see what’s going on, like 5 MySQL calls on the home page and what happens when you don’t have a favicon.ico (in yellow).

In summary, “performance is all about architecture”, “know your costs”.

Be careful, because some tools (like valgrind and xdebug) you don’t want to put it on production systems, you could capture production traffic and replay it on a dev/testing box, but “you just have to minimize the differences and do your best”.

PlanetMySQL Voting: Vote UP / Vote DOWN

Writing A Storage Engine for Drizzle, Part 2: CREATE TABLE

Stewart Smith — Thu, 11 Mar 2010 14:27:34 +0000

The DDL code paths for Drizzle are increasingly different from MySQL. For example, the embedded_innodb StorageEngine CREATE TABLE code path is completely different than what it would have to be for MySQL. This is because of a number of reasons, the primary one being that Drizzle uses a protobuf message to describe the table format instead of several data structures and a FRM file.

We are pretty close to having the table protobuf message format being final (there’s a few bits left to clean up, but expect them done Real Soon Now (TM)). You can see the definition (which is pretty simple to follow) in drizzled/message/table.proto. Also check out my series of blog posts on the table message (more posts coming, I promise!).

Drizzle allows either your StorageEngine or the Drizzle kernel to take care of storage of table metadata. You tell the Drizzle kernel that your engine will take care of metadata itself by specifying HTON_HAS_DATA_DICTIONARY to the StorageEngine constructor. If you don’t specify HTON_HAS_DATA_DICTIONARY, the Drizzle kernel stores the serialized Table protobuf message in a “table_name.dfe” file in a directory named after the database. If you have specified that you have a data dictionary, you’ll also have to implement some other methods in your StorageEngine. We’ll cover these in a later post.

If you ever dealt with creating a table in MySQL, you may recognize this method:

virtual int create(const char *name, TABLE *form, HA_CREATE_INFO *info)=0;

This is not how we do things in Drizzle. We now have this function in StorageEngine that you have to implement:

int doCreateTable(Session* session, const char *path,
                  Table& table_obj,
                  drizzled::message::Table& table_message)

The existence of the Table parameter is largely historic and at some point will go away. In the Embedded InnoDB engine, we don’t use the Table parameter at all. Shortly we’ll also get rid of the path parameter, instead having the table schema in the Table message and helper functions to construct path names.

Methods name “doFoo” (such as doCreateTable) mean that there is a method named foo() (such as createTable()) in the base class. It does some base work (such as making sure the table_message is filled out and handling any errors) while the “real” work is done by your StorageEngine in the doCreateTable() method.

The Embedded InnoDB engine goes through the table message and constructs a data structure for the Embedded InnoDB library to create a table. The ARCHIVE storage engine is much simpler, and it pretty much just creates the header of the ARZ file, mostly ignoring the format of the table. The best bet is to look at the code from one of these engines, depending on what type of engine you’re working on. This code, along with the table message definition should be more than enough

PlanetMySQL Voting: Vote UP / Vote DOWN

Surveying MySQL’s Popular Storage Engines

Sean Hull — Thu, 11 Mar 2010 13:00:35 +0000

In this month’s Database Journal piece we look at the spectrum of MySQL storage engines available, and examine what some of their strengths and weaknesses are.

View the article here: Survey of MySQL Storage Engines

PlanetMySQL Voting: Vote UP / Vote DOWN

Emulating a 'top' CPU summary using /proc/stat and MySQL

Justin Swanhart — Thu, 11 Mar 2010 08:54:50 +0000

In my last blog post, I showed how we can get some raw performance information from /proc into the MySQL database using a LOAD DATA INFILE (LDI) command.

I've modified that LDI call slightly to set the `other` column to equal the sum total of the CPU counters for those rows which begin with 'cpu'.

original:
other = IF(@the_key like 'cpu%', NULL , @val1);

new:
other = IF(@the_key like 'cpu%', user + nice + system + idle + iowait + irq + softirq + steal + guest, @val1);

Top provides a useful output that looks something like the following:

top - 04:59:14 up 14 days,  3:34,  1 user,  load average: 0.00, 0.00, 0.00
Tasks: 216 total,   1 running, 215 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 99.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   8172108k total,  5115388k used,  3056720k free,   315180k buffers
Swap:  2097144k total,        0k used,  2097144k free,  3630748k cached

The information I'm currently concerned with presenting is the CPU summary:
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 99.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

In order to emulate this display, we will need to sample two data points from /proc/stat.

Load the data from proc_stat
Sleep 1 second
Load the data again
Compare the values

You should end up with something similar to the following:

mysql> select * from test.proc_stat where the_key = 'cpu';
+-----+---------+--------+-------+--------+------------+--------+------+---------+-------+-------+------------+
| seq | the_key | user   | nice  | system | idle       | iowait | irq  | softirq | steal | guest | other      |
+-----+---------+--------+-------+--------+------------+--------+------+---------+-------+-------+------------+
|   1 | cpu     | 440022 | 36207 |  94583 | 1976124562 |  89082 |  858 |   27243 |     0 |     0 | 1976812557 | 
|  24 | cpu     | 440024 | 36207 |  94583 | 1976130493 |  89082 |  858 |   27243 |     0 |     0 | 1976818490 | 
+-----+---------+--------+-------+--------+------------+--------+------+---------+-------+-------+------------+
2 rows in set (0.00 sec)

To display the CPU utilization, run the following query:

select 100 * ( ( new.user - old.user )  / ( new.other - old.other ) ) user,
       100 * ( ( new.nice - old.nice ) / ( new.other - old.other ) ) nice, 
       100 * ( ( new.system - old.system ) / ( new.other - old.other ) ) system, 
       100 * ( ( new.idle - old.idle ) / ( new.other - old.other ) ) idle, 
       100 * ( ( new.iowait - old.iowait ) / ( new.other - old.other ) ) iowait, 
       100 * ( ( new.irq - old.irq ) / ( new.other - old.other ) ) irq, 
       100 * ( ( new.softirq - old.softirq ) / ( new.other - old.other ) ) softer,
       100 * ( ( new.steal - old.steal ) / ( new.other - old.other ) ) steal, 
       100 * ( ( new.guest - old.guest ) / ( new.other - old.other ) ) guest
from test.proc_stat old, 
         test.proc_stat new
where new.seq > old.seq
     and old.the_key = 'cpu'
     and new.the_key = old.the_key;

+--------+--------+--------+---------+--------+--------+--------+--------+--------+
| user   | nice   | system | idle    | iowait | irq    | softer | steal  | guest  |
+--------+--------+--------+---------+--------+--------+--------+--------+--------+
| 0.0337 | 0.0000 | 0.0000 | 99.9663 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 
+--------+--------+--------+---------+--------+--------+--------+--------+--------+
1 row in set (0.01 sec)


edit:

for completeness sake, here is the SQL script I execute to load the data from proc:

CREATE TABLE IF NOT EXISTS test.proc_stat (
  seq tinyint auto_increment primary key, 
  the_key char(25) NOT NULL, 
  user bigint,
  nice bigint, 
  system bigint,
  idle bigint, 
  iowait bigint,
  irq bigint,
  softirq bigint, 
  steal bigint, 
  guest bigint, 
  other bigint
);  

/* MySQL treats consecutive delimiters as separate fields, so some fancy footwork
   is required to load the file successfully.  The file includes a cpu field followed
   by two spaces which is the sum of all the individual CPUs in the system.  

   To account for this each row is read into some MySQL variables. Those variables 
   are examined to determine which field holds the correct value.
*/
LOAD DATA INFILE '/proc/stat' 
   IGNORE INTO TABLE test.proc_stat 
   FIELDS TERMINATED BY ' ' 
   (@the_key, @val1, @val2, @val3, @val4, @val5, @val6, @val7, @val8, @val9, @val10)
SET 
    the_key = @the_key, 
    user = IF(@the_key NOT LIKE 'cpu%', NULL, IF(@the_key != 'cpu', IFNULL(@val1, 0), IFNULL(@val2,0))),
    nice = IF(@the_key NOT LIKE 'cpu%', NULL, IF(@the_key != 'cpu', IFNULL(@val2, 0), IFNULL(@val3,0))),
  system = IF(@the_key NOT LIKE 'cpu%', NULL, IF(@the_key != 'cpu', IFNULL(@val3, 0), IFNULL(@val4,0))),
    idle = IF(@the_key NOT LIKE 'cpu%', NULL, IF(@the_key != 'cpu', IFNULL(@val4, 0), IFNULL(@val5,0))),
  iowait = IF(@the_key NOT LIKE 'cpu%', NULL, IF(@the_key != 'cpu', IFNULL(@val5, 0), IFNULL(@val6,0))),
     irq = IF(@the_key NOT LIKE 'cpu%', NULL, IF(@the_key != 'cpu', IFNULL(@val6, 0), IFNULL(@val7,0))),
 softirq = IF(@the_key NOT LIKE 'cpu%', NULL, IF(@the_key != 'cpu', IFNULL(@val7, 0), IFNULL(@val8,0))),
   steal = IF(@the_key NOT LIKE 'cpu%', NULL, IF(@the_key != 'cpu', IFNULL(@val8, 0), IFNULL(@val9,0))),
   guest = IF(@the_key NOT LIKE 'cpu%', NULL, IF(@the_key != 'cpu', IFNULL(@val9, 0), IFNULL(@val10,0))),
   other = IF(@the_key like 'cpu%', user + nice + system + idle + iowait + irq + softirq + steal + guest, @val1);

PlanetMySQL Voting: Vote UP / Vote DOWN

Continuing the journey

Stewart Smith — Thu, 11 Mar 2010 07:42:46 +0000

A couple of months ago (December 1st for those playing along at home) it marked five years to the day that I started at MySQL AB (~~now Sun~~, now Oracle). A good part of me is really surprised it was for that long and other parts surprised it wasn’t longer. Through MySQL and Sun, I met some pretty amazing people, worked with some really smart ones and formed really solid and awesome friendships. Of course, not everything was perfect (sometimes not even close), but we did have some fun.

Up until November 2008 (that’s 3 years and 11 months for those playing at home) I worked on MySQL Cluster. Still love the product and love how much better we’re making Drizzle so it’ll be the best SQL interface to NDB :)

The ideas behind Drizzle had been talked about for a while… and with my experience with internals of the MySQL server, I thought that some change and dramatic improvement was sorely needed.

Then, in 2008, Brian created a tree. I was soon sending in patches at nights, we announced to the whole world at OSCON and it captured a lot of attention.

Since November 2008 I’ve been working on Drizzle full time. It was absolutely awesome that I had the opportunity to spend all my days hacking on Drizzle – both directly with fantastic people and for fantastic people.

But… the Sun set… which was exciting and sad at the same time.

Never to fear! There were plenty of places wanting Drizzle hackers (and MySQL hackers). For me, it came down to this: “real artists ship”. While there were other places where I would no doubt be happy and work on something really cool, the only way I could end up working out where I should really be was: what is the best way to have Drizzle make a stable release that we’d see be suitable for deployment? So, Where Am I Now?

Rackspace.

Where I’ll again be spending all my time hacking Drizzle.

PlanetMySQL Voting: Vote UP / Vote DOWN

SQL syntax with /*! c-style comments in MySQLdump

Kedar — Thu, 11 Mar 2010 07:27:24 +0000

In mysql we have — , /* and /*! comments. This post is mainly about very basic c-style comments. /*! : C-Style comments in MySQL We normally see comments in MySQLdump as follows: /*!40000 ALTER TABLE `a` DISABLE KEYS */; Or /*!50013 DEFINER=`root`@`localhost` SQL SECURITY DEFINER */ These are actually C-Style comments which has embeded sql and treated specially [...] Related posts:

PlanetMySQL Voting: Vote UP / Vote DOWN

Proper SQL table alias use conventions

Shlomi Noach — Thu, 11 Mar 2010 07:10:09 +0000

After seeing quite some SQL statements over the years, something is bugging me: there is no consistent convention as for how to write an SQL query.

I’m going to leave formatting, upper/lower-case issues aside, and discuss a small part of the SQL syntax: table aliases. Looking at three different queries, I will describe what I find to be problematic table alias use.

Using the sakila database, take a look at the following queries:

Query #1

SELECT
 R.rental_date, C.customer_id, C.first_name, C.last_name
FROM
 rental R
 JOIN customer C USING (customer_id)
WHERE
 R.rental_date >= DATE('2005-10-01')
 AND C.store_id=1;

The above looks for film rentals done in a specific store (store #1), as of Oct. 1st, 2005.

Query #2

SELECT
 F.title, C.name
FROM
 film AS F
 JOIN film_category AS S ON (F.film_id = S.film_id)
 JOIN category AS C ON (S.category_id = C.category_id)
WHERE F.length > 180;

The above lists the title and category for all films longer than three hours.

Query #3

SELECT c.customer_id, c.last_name
FROM
  customer c
  INNER JOIN address a ON (c.address_id = a.address_id)
  INNER JOIN (
    SELECT
      c.city_id
    FROM
      city AS c
      JOIN country s ON (c.country_id = s.country_id)
    WHERE
      s.country LIKE 'F%'
  ) s1 USING (city_id)
WHERE
  create_date >= DATE('2005-10-01');

The above lists customers created as of Oct. 1st, 2005, and who live in countries starting with an ‘F’. The query could be solved without a subquery, but there’s a good reason why I made it so.

The problems

I used very different conventions on any one of the queries, and sometimes within each query. And it’s common that I see the same on a customer’s site, what with having many programmers do the SQL coding. Again, I will only discuss the table aliases conventions. I’ll leaver the rest to the reader.

Here’s where I see problems:

Query #1: In itself, it looks fine. Rental turns to R, Customer turns to C. I will comment on this slightly later on when I provide my full opinion.
Query #2: So film turns to F, category turns to C. What should film_category turn into? Out of letters? Let’s just go for S, shall we? But S has nothing do with film_category. Yet it’s so commonly seen.
Query #2: We’re using the AS keyword now. We didn’t use it before.
Queries #1, #2: Hold on. Wasn’t C taken for customer in Query #1? Now, in Query #2 it stands for category? I’m beginning to get confused.
Query #3: Now aliases are lower case; I was just getting used to them being upper case.
Query #3: But, hey, c is back to customer!
Query #3: Or, is it? Take a look at the subquery. Theres another c in there! This time it’s city! And it’s perfectly valid syntax. We actually have two identical aliases in the same query.
Query #3: If I could, I would name country with c as well. But I can’t. So why not throw in s again?
Query #3: and now I don’t even bother using the alias when accessing the create_date. Well, there’s no such column in any of the other tables!

Proper conventions

What I find so disturbing is that whenever I read a complex query, I need to go back and forth, back and forth between table aliases (found everywhere in the query) and their declaration point. Such irregularities make the queries difficult to read.

Any of the above issues could be justified. But I wish to make some suggestions:

Decide whether you’re going for upper or lower case.
Do not use the same alias twice in your query, even if it’s valid.
Aliases do not have to be single character. film_category may just as well be FC.
Do not alias something that is hard to interpret. s does not stand for country.
Think ahead: use same aliases throughout all your queries, as far as you can. If uniqueness is a problem, make for longer aliases. Use cust instead of c.

The above should make for more organized and readable SQL code. Remember: what one programmer finds as a very intuitive alias, is unintuitive to another!

My own convention

Simple: I only use aliases when using self joins. I am aware that queries are much longer what with long table names. I go farther than that: I prefer fully qualifying questionable columns throughout the query. Yes, it makes the query even longer.

I know this does not appeal to many. But there’s no confusion. And it’s easily searchable. And it’s consistent. And if properly formatted, as in the above queries, is well readable.

Now please join me in asking Oracle if they can add multi-line Strings for java, as there are for python.

PlanetMySQL Voting: Vote UP / Vote DOWN

[RH]acker

Monty Taylor — Thu, 11 Mar 2010 00:39:14 +0000

As I'm sure everyone has figured out by now, I've joined Rackspace where I will continue to work on Drizzle. I'm honestly thrilled with my new home, and there are a myriad of reasons for that. I think the one that I'm most excited about is that they are already the thing that all of the hype was about MySQL and RedHat and IBM wanting to become:

A Service Company

Rackspace doesn't want you to run Rackspace-Apache or RackspaceDB or EC-Rackspace. They want you to be able to run bog-standard Apache. And Linux. And MySQL. And PHP. And Drizzle. Then, Rackspace wants to be the best at providing you the service you need around those.

No ludicrous MySQL Enterprise "we'll sell you a license to a free product, and then we'll include bundled with that a subscription a piece of non-free monitoring software" upselling. Rackspace actually wants to provide you a valuable service, and they want to do such a good job at it that you will happily pay them to do it.

For developers, there is a wonderful upside to this: Rackspace doesn't want a special internal Rackspace-only version of anything. It has no value that way. They want the good software to be ubiquitous so that they can compete in the service arena. This means that they don't want assignment of copyright. This means they don't have crazy policies about what Free Software projects you can and cannot contribute to.

Rackspace goes one step further than "do no evil" ... they actually want you to try to improve the state of the art - which goes right to the core of why I'm involved with Free Software in the first place.

I truly believe that Free will always win over Restricted, that Open beats Closed and that Sharing will always improve the world before Hoarding. I've always contended that a company can be successful and make the world a better place and that the two are not mutually exclusive.

I am thrilled to now be a part of a company where I can do my best to prove it.

PlanetMySQL Voting: Vote UP / Vote DOWN

Talking at the University of Utah

Dathan Pattishall — Wed, 10 Mar 2010 23:56:00 +0000

Giving a talk at the University of Utah on everything from scaling, clustering, mysql, mysql internals, noSQL (Cassandra) to how to manage all this stuff. If you are there at University I'm bringing some Swag!

Also I will upload the slides and put them here.

PlanetMySQL Voting: Vote UP / Vote DOWN

Peter Gulutzan at the O’Reilly MySQL Conference

Peter Gulutzan — Wed, 10 Mar 2010 21:47:54 +0000

I will be doing two talks at the O’Reilly MySQL Conference & Expo in Santa Clara CA.
Performance Schema Tuesday April 13, 11:55am.
Demos Of All The Big New Features Thursday April 15, 11:55am, with Konstantin Osipov.

The other MySQL server engineers giving talks are:
Alexander Barkov (globalization)
Chuck Bell (backup)
Mattias Jonsson (partitions)
Mats Kindahl (replication)
Konstantin Osipov (runtime)
Inaam Rana (InnoDB)
Mikael Ronstrom (partitions)
Calvin Sun (InnoDB)
Lars Thalmann (replication)
Jimmy Yang (InnoDB)

Maybe we’ll have a Birds of a Feather session too.

PlanetMySQL Voting: Vote UP / Vote DOWN

mk-schema-change? Check out ideas from oak-online-alter-table

Shlomi Noach — Wed, 10 Mar 2010 18:28:29 +0000

In response to Mark Callaghan’s post mk-schema-change.

I apologize for not commenting on the post itself, I do not hold a Facebook account. Anyway this is a long write, so it may as well deserve a post of its own.

Some of the work Mark is describing already exists under openark kit’s oak-online-alter-table. Allow me to explain what I have gained there, and how the issue can be further pursued. There is relevance to Mark’s suggestion.

oak-online-alter-table uses a combination of locks, chunks and triggers to achieve an almost non-blocking ALTER TABLE effect. I had a very short opportunity to speak with Mark on last year’s conference, in between bites. Mark stated that anything involving triggers was irrelevant in his case.

The triggers are a pain, but I believe a few other insights from oak-online-alter-table can be of interest.

The first attempt

My first attempt with the script assumed:

Table has an AUTO_INCREMENT PRIMARY KEY column
New rows always gain ascending PRIMARY KEY values
PRIMARY KEY never changes for an existing row
PRIMARY KEY values are never reused
Rows may be deleted at will
No triggers exist on the table
No FOREIGN KEYs exist on the table.

So the idea was: when one wants to do an ALTER TABLE:

Create a ghost table with the new structure.
Read the minimum and maximum PK values.
Create AFTER INSERT, AFTER UPDATE, AFTER DELETE triggers on the original table. These triggers will propagate the changes onto the ghost table.
Working out slowly, and in small chunks, copy rows within recorded min-max values range into the ghost table. The interesting part is where the script makes sure there’s no contradiction between these actions and those of the triggers, (whichever came first!). This is largely solved using INSERT IGNORE and REPLACE INTO in the proper context.
Working out slowly and in chunks again, we remove rows from the ghost table, which are no longer existent in the original table.
Once all chunking is complete, RENAME original table to *_old, and ghost table in place of the original table.

Steps 4 & 5 are similar in concept to transactional recovery through redo logs and undo logs.

The next attempt

Next phase removed the AUTO_INCREMENT requirement, as well as the “no reuse of PK”. In fact, the only remaining constraints were:

There is some UNIQUE KEY on the table which is unaffected by the ALTER operation
No triggers exist on the table
No FOREIGN KEYs exist on the table.

The steps are in general very similar to those listed previously, only now a more elaborate chunking method is used with possible non-integer, possible multi-column chunking algorithm. Also, the triggers take care of changes in UNIQUE KEY values themselves.

mk-schema-change?

Have a look at the wiki pages for OnlineAlterTable*. There is some discussion on concurrency issues; on transactional behavior, which explains why oak-online-alter-table performs correctly. Some of these are very relvant, I believe, to Mark’s suggestion. In particular, making the chunks copy; retaining transactional integrity, etc.

To remove any doubt, oak-online-alter-table is not production ready or anywhere near. Use at your own risk. I’ve seen it work, and I’ve seen it crash. I got little feedback and thus little chance to fix things. I also didn’t touch the code for quite a few months now, so I’m a little rusty myself.

PlanetMySQL Voting: Vote UP / Vote DOWN

Google Summer of Code projects, Drizzle

Brian Aker — Wed, 10 Mar 2010 18:25:25 +0000

I've been doing Google Summer of Code projects with students since its creation. As far as intern programs go, it has been one of the most successful I have ever worked with.

Last year was particularly awesome in that with Drizzle we were able to have students work on projects that made it back into Drizzle. While I have always seen good work created, it has always been hit or miss on whether the student's work has made it back into the project. Last year though we got more code in then ever before and I believe this year will be the same. We have had students go on to jobs thanks to the work they did on Drizzle.

Interning gives you real experience, and it provides resume material which differentiates students who are going on to work in the software engineering field. Working on open source means that you have real experience on your resume, experience that an employer can see. There are many positions open in the Drizzle/MySQL ecosystem and students who have real world experience should have any easy time finding work with the knowledge you will gain from this program.

For Drizzle we have worked out a partial list for this year:
http://drizzle.org/wiki/Soc

Don't see anything you like? I am happy to add new projects or work with students on libmemcached or Gearman.

Are you interested in working on a different project? Apache, Linux, Postgres? Talk to those projects and ask them to either participate or suggest ideas on projects to them.
PlanetMySQL Voting: Vote UP / Vote DOWN

mk-schema-change

Wed, 10 Mar 2010 15:05:40 +0000

I want a tool to make some long-running schema changes almost non-blocking. They should block access to a table for no more than a few seconds. I also want to do some of these in place on a master rather than on a slave that has been taken offline. I think this will work for most schema changes. It doesn't have to work for all of them and there are restrictions. This will not work when statements that modify the table for which the schema change is done reference other tables and the other tables are modified during the schema change. If production SQL cannot be changed to meet this restriction, then the schema change can be done on a slave that has been taken offline. Is anyone else interested in such a tool? A hand-waving description of the process is:

Create the new table on the master. The new table might use MyISAM without indexes initially to make the insert as fast as possible and reduce the load on InnoDB.
Run set sql_log_bin=0 as what follows should not be written to the binlog
Run start transaction with consistent innodb snapshot to start an Innodb transaction and get current binlog offset of the master
Run insert into new_table select * from original_table on the master. Alas, this will get a transaction duration read lock on every row in original_table unless you use row based replication or hack InnoDB or set innodb_locks_unsafe_for_binlog.
Convert new_table to InnoDB and create indexes on it
Replay changes from the binlogs after the point in time recorded in step #3. This should extract changes to original_table and replay them against new_table.

PlanetMySQL Voting: Vote UP / Vote DOWN

Presenting on new MySQL Cluster 7.1 features at MySQL UC (and discount code!)

Andrew Morgan — Wed, 10 Mar 2010 10:12:06 +0000

Together with Berndt I’ll be presenting on the new features in MySQL Cluster 7.1 at this year’s MySQL Cluster User Conference – Santa Clara, on April 12th. If you’re interested in using MySQL Cluster but aren’t sure how to get started (or you’ve used it but would like some tips) then this is a great opportunity. Check out the presentation description.

If you register by 15 March then you get the early-bird price and if you use this ‘friend of a speaker’ code then you get an additional 25% off: mys10fsp

mys10fsp

PlanetMySQL Voting: Vote UP / Vote DOWN

MySQL Cluster on Windows – webinar replay available

Andrew Morgan — Wed, 10 Mar 2010 09:58:13 +0000

If you missed the recent webinar on running MySQL Cluster on Windows then you can watch/listen to the replay at http://www.mysql.com/news-and-events/on-demand-webinars/display-od-517.html

PlanetMySQL Voting: Vote UP / Vote DOWN

Things to monitor on MySQL, the user’s perspective

Shlomi Noach — Wed, 10 Mar 2010 09:12:24 +0000

Working on mycheckpoint, I have the intention of adding custom monitoring. That is, letting the user define things to monitor. I have my own thoughts, I would be grateful to get more input!

What would the user want to monitor?

Monitoring for the number of SELECT statements per second, InnoDB locks, slave replication lag etc. is very important, and monitoring utilities provide with this information. But what does that tell the end user? Not much.

The experienced DBA may gain a lot. The user would be more interested in completely other kind of information. In between, some information is relevant to both.

Say we were managing an on-line store. We want to monitor the health of the database. But the health of the database is inseparable from the health of the application. I mean, having little to no disk usage is fine, unless… something is wrong with the application, which leads to no new purchases.

And so a user would be interested in monitoring the number of purchases per hour, or the time passed since last successful purchase. This kind of data can only be generated by a user’s specific query. Looking at the charts, the user would then feel safer and confident in the wellness of his store app.

But let’s dig further. We want the store’s website to provide with good response. In particular, the query which returns the items in a customer’s cart must react quickly. Our user would not only want to see that purchases get along, but also that page load times (as in our example) are quick for those critical parts. And so a user should be able to monitor the time it took to execute a given query.

It can be of further interest to know how many times per second a given query is executed. This part is not easily done on the server side, and requires the user’s cooperation (or else we must analyze the general log, sniff, or set up a proxy). If the user is willing, she can log to some table each time she executes a certain query. Then we’re back to monitoring a regular table, as with the first example.

It is also possible to monitor for a query’s execution plan. Is it full scan? How many rows are expected? But given that we can monitor the time it took to execute a query, I’m not sure this is useful. If everything runs fast enough — who cares about how it executes?

Some of the above can be monitored on an altogether higher level: if we’re talking about some web application, then we can use our Apache logs to determine load time for pages, or number of requests to our “cart items” page. But not always do we work with web servers, and we may be interested in checking the specific queries behind the scenes.

Summary

Custom monitoring can include:

User defined queries (number of concurrent visitors; count of successful operations per second; number of rows per given table or condition; …)
Execution time for user defined queries (time it takes to return cart items; find rows matching condition; sort a table; …)
Number of executions for a given query, per second.

I intend to incorporate the above into mycheckpoint as part of its standard monitoring scheme.

Please share your thought below.

PlanetMySQL Voting: Vote UP / Vote DOWN

Its a cheat! Get Linux performance information from your MySQL database without shell access.

Justin Swanhart — Wed, 10 Mar 2010 09:01:59 +0000

System administrators familiar with the Linux operating system use the tools in the 'procps' toolset all the time. Tools which read from /proc include top, iostat, vmstat, sar and others. The files in /proc contain useful information about the performance of the system. Most of the files are documented in the Linux kernel documentation. You can also check man 5 proc.

Most performance monitoring tools invoke other tools like iostat to collect performance information instead of reading from the /proc filesytem itself. This begs the question, what can you do if you don't have access to those tools? Perhaps you are using a hosted Linux database and have no access to the underlying shell to execute tools like iostat or top? How could you gather information about the performance of the actual system without being allowed to run the tools?

MySQL includes a command called LOAD DATA INFILE which can read the contents of a delimited text file and store the contents into a database table. The contents of /proc are world readable, so your MySQL database should have access to this information as long as it is running on a Linux server.

Lets start by collecting and reporting on some CPU performance information.

CREATE TEMPORARY TABLE test.proc_stat (
  seq tinyint auto_increment primary key, 
  the_key char(25) NOT NULL, 
  user bigint,
  nice bigint, 
  system bigint,
  idle bigint, 
  iowait bigint,
  irq bigint,
  softirq bigint, 
  steal bigint, 
  guest bigint, 
  other bigint
);  

/* MySQL treats consecutive delimiters as separate fields, so some fancy footwork
   is required to load the file successfully.  The file includes a cpu field followed
   by two spaces which is the sum of all the individual CPUs in the system.  

   To account for this each row is read into some MySQL variables. Those variables 
   are examined to determine which field holds the correct value.
*/
LOAD DATA INFILE '/proc/stat' 
   IGNORE INTO TABLE test.proc_stat 
   FIELDS TERMINATED BY ' ' 
   (@the_key, @val1, @val2, @val3, @val4, @val5, @val6, @val7, @val8, @val9, @val10)
SET other = IF(@the_key like 'cpu%', NULL, @val1),  
    the_key = @the_key, 
    user = IF(@the_key NOT LIKE 'cpu%', NULL, IF(@the_key != 'cpu', IFNULL(@val1, 0), IFNULL(@val2,0))),
    nice = IF(@the_key NOT LIKE 'cpu%', NULL, IF(@the_key != 'cpu', IFNULL(@val2, 0), IFNULL(@val3,0))),
  system = IF(@the_key NOT LIKE 'cpu%', NULL, IF(@the_key != 'cpu', IFNULL(@val3, 0), IFNULL(@val4,0))),
    idle = IF(@the_key NOT LIKE 'cpu%', NULL, IF(@the_key != 'cpu', IFNULL(@val4, 0), IFNULL(@val5,0))),
  iowait = IF(@the_key NOT LIKE 'cpu%', NULL, IF(@the_key != 'cpu', IFNULL(@val5, 0), IFNULL(@val6,0))),
     irq = IF(@the_key NOT LIKE 'cpu%', NULL, IF(@the_key != 'cpu', IFNULL(@val6, 0), IFNULL(@val7,0))),
 softirq = IF(@the_key NOT LIKE 'cpu%', NULL, IF(@the_key != 'cpu', IFNULL(@val7, 0), IFNULL(@val8,0))),
   steal = IF(@the_key NOT LIKE 'cpu%', NULL, IF(@the_key != 'cpu', IFNULL(@val8, 0), IFNULL(@val9,0))),
   guest = IF(@the_key NOT LIKE 'cpu%', NULL, IF(@the_key != 'cpu', IFNULL(@val9, 0), IFNULL(@val10,0)));

Depending on your kernel version you may get 1 or more warnings about unexpected numbers of columns. You can safely ignore these.

mysql> select * from test.proc_stat;
+-----+---------------+--------+-------+--------+------------+--------+------+---------+-------+-------+------------+
| seq | the_key       | user   | nice  | system | idle       | iowait | irq  | softirq | steal | guest | other      |
+-----+---------------+--------+-------+--------+------------+--------+------+---------+-------+-------+------------+
|   1 | cpu           | 378340 | 33588 |  82489 | 1838257830 |  75444 |  750 |   23065 |     0 |     0 |       NULL |
|   2 | cpu0          |   4152 |   125 |   1613 |  114920899 |    624 |    0 |     869 |     0 |     0 |       NULL | 
|   3 | cpu1          |   2182 |    78 |   1474 |  114924477 |     50 |    2 |       3 |     0 |     0 |       NULL | 
|   4 | cpu2          |   6037 |  5418 |   2289 |  114914024 |     55 |   34 |     401 |     0 |     0 |       NULL | 
|   5 | cpu3          |   3519 |    55 |    842 |  114923794 |     37 |    1 |       1 |     0 |     0 |       NULL | 
|   6 | cpu4          |  71851 |  5443 |   6656 |  114840363 |   3197 |   11 |     720 |     0 |     0 |       NULL | 
|   7 | cpu5          |   2435 |     5 |    801 |  114924963 |     29 |    2 |       0 |     0 |     0 |       NULL | 
|   8 | cpu6          | 136246 |  4711 |  36628 |  114690032 |  46119 |   20 |   14471 |     0 |     0 |       NULL | 
|   9 | cpu7          |   1119 |     2 |    366 |  114926691 |     40 |    1 |       0 |     0 |     0 |       NULL | 
|  10 | cpu8          |   4126 |    34 |   2772 |  114920032 |     92 |    1 |    1153 |     0 |     0 |       NULL | 
|  11 | cpu9          |   1618 |     2 |    694 |  114925811 |     77 |    1 |       0 |     0 |     0 |       NULL | 
|  12 | cpu10         |  18096 |  8735 |   6823 |  114891588 |    396 |  179 |    2379 |     0 |     0 |       NULL | 
|  13 | cpu11         |   7243 |  2583 |   3559 |  114914559 |    241 |    1 |       2 |     0 |     0 |       NULL | 
|  14 | cpu12         |   5215 |  2380 |   2776 |  114915814 |    417 |  342 |    1237 |     0 |     0 |       NULL | 
|  15 | cpu13         |   3224 |    28 |   1507 |  114923336 |     77 |    2 |       0 |     0 |     0 |       NULL | 
|  16 | cpu14         | 109818 |  3979 |  13071 |  114775431 |  23901 |  143 |    1823 |     0 |     0 |       NULL | 
|  17 | cpu15         |   1450 |     1 |    612 |  114926010 |     83 |    1 |       0 |     0 |     0 |       NULL | 
|  18 | intr          |   NULL |  NULL |   NULL |       NULL |   NULL | NULL |    NULL |  NULL |  NULL | 1176485951 | 
|  19 | ctxt          |   NULL |  NULL |   NULL |       NULL |   NULL | NULL |    NULL |  NULL |  NULL |  171220339 | 
|  20 | btime         |   NULL |  NULL |   NULL |       NULL |   NULL | NULL |    NULL |  NULL |  NULL | 1267061074 | 
|  21 | processes     |   NULL |  NULL |   NULL |       NULL |   NULL | NULL |    NULL |  NULL |  NULL |     168510 | 
|  22 | procs_running |   NULL |  NULL |   NULL |       NULL |   NULL | NULL |    NULL |  NULL |  NULL |          1 | 
|  23 | procs_blocked |   NULL |  NULL |   NULL |       NULL |   NULL | NULL |    NULL |  NULL |  NULL |          0 | 
+-----+---------------+--------+-------+--------+------------+--------+------+---------+-------+-------+------------+
23 rows in set (0.00 sec)

Now that you know you can collect that information, then you can emulate top to calculate the current total CPU usage. I'll show you how to do that in my next blog post.
PlanetMySQL Voting: Vote UP / Vote DOWN

Do you need more data in the slow query log?

Wed, 10 Mar 2010 03:04:06 +0000

Imagine you tried to use the slow query log to debug a performance problem. Does the current format have enough details?

# Time: 100309 18:48:23
# User@Host: root[root] @ localhost []
# Query_time: 0  Lock_time: 0  Rows_sent: 1  Rows_examined: 1

I have added Thread_id, Errno, Start and End. Thread_id can be used to find similar data from SHOW PROCESSLIST and the binlog. Errno is useful in many cases. Start and End are there for convenience. Can you suggest anything else that would be easy to add? Note that Rows_sent and Rows_examined are always zero for insert, update and delete statements. Feature request 49756 is open to change that. Maybe that is easy to fix.

# Query_time: 0  Lock_time: 0  Rows_sent: 1  Rows_examined: 1\
           Thread_id: 3 Errno: 0 Start: 18:48:23 End: 18:48:23

Update, I found more data that is easy to add and the proposed output is:

# Time: 100310  7:51:28
# User@Host: root[root] @ localhost []
# Query_time: 0  Lock_time: 0  Rows_sent: 1  Rows_examined: 0 Thread_id: 1 Errno: 0 \
Killed: 0 Bytes_received: 104 Bytes_sent: 161 Read_first: 0 Read_last: 0 Read_key: 0 \
Read_next: 0 Read_prev: 0 Read_rnd: 0 Read_rnd_next: 0 Sort_merge_passes: 0 \
Sort_range_count: 0 Sort_rows: 0 Sort_scan_count: 0 Created_tmp_disk_tables: 0 \
Created_tmp_tables: 0 Start:  7:51:28 End:  7:51:28

PlanetMySQL Voting: Vote UP / Vote DOWN

Speaking at MySQL Conference: The Thinking Person's Guide to Data Warehouse Design

Robin Schumacher — Wed, 10 Mar 2010 02:37:42 +0000

I'll be presenting "The Thinking Person's Guide to Data Warehouse Design" at the upcoming MySQL User conference. While a lot of people think that bad SQL code is the #1 wrecking ball of data warehouses and marts, the fact is that poor database design is the first cause of both downtime and bad performance. In my presentation, I'll do my best to show how up-front worRead More...
PlanetMySQL Voting: Vote UP / Vote DOWN

Hotsos Symposium 2010 — Battle Against Any Guess Is Won

Alex Gorbachev — Tue, 09 Mar 2010 20:16:51 +0000

Video fragments of my session posted at the end — read on.

I arrived at Omni Mandalay Hotel on Sunday evening with Dan Norris. I was flying through Chicago and it turned out that Dan was on the same flight and only few rows behind me. Small world.

Preparations for the conference were very chaotic on my part and, of course, I didn’t have either of my presentations ready. I was very stressed and getting sick as well — it looked like a complete disaster waiting to happen. I’d like to say that I was feeling like Doug Burns as he often managed to get sick just before a conference. Of course, I worked on my slides for the last few days as well as on the flight and presentation was slowly getting there but boy was I tired!

I quickly said hello to the crowd in the bar on the way to my room and rushed away to do some more damage to my slides. And then I had a brilliant idea — I could still see one of my best mates and do something good about my presentation! I asked Doug if he was interested in the preview (he probably wasn’t interested but he couldn’t say it to me) especially that my session wasn’t on his original agenda. Of course, that would mean that he had to leave a bunch of other good friends and spend some time tete-a-tete. Knowing Doug, this is some of the hardest thing to ask from him but it shows how good of a friend he is! (Plus, everyone thinks that he is anti-social anyway. Shhhh!)

Doug has made my day — while he provided lots of ideas and feedback on few things that I was lucking, he generally approved the idea and confirmed that it wasn’t totally crazy. I guess that was all I needed back then and Doug knew how nervous I was about it. (Thanks mate!)

So I called Sunday a day very early and went to bed before midnight. I really needed some sleep. Woken up by the alarm at 5AM (I woke up few times during the night looking at the clock — making sure I didn’t sleep through) and slides were ready just before lunch. I even managed to do a test run and it took 65 minutes — a wee bit too long for one hour session. But it was good test and I knew I had to be just a bit more concise in few parts.

Mi morning was very productive. Unfortunately, I missed the opening keynote from Tom Kyte. Such a pity! If what Doug wrote is true, Tom was talking about the mistakes we make *because* of our experience and our assumptions. This was exactly one of the points I was making in my Battle Against Any Guess — experience is danger. I wish I could see Tom’s example. Oh well, maybe another time.

I managed to attend half of the Richard Foote’s session on indexes but my mind was far away — with my own slides. Though, I did manage to focus on bitmap indexes part and the myth of bitmap indexes not working well for columns with high cardinality. Very interesting conclusions. I’m still wondering how much overhead updates will do to such bitmap index.

After lunch, it was my turn. I ordered few copies of the latest OakTable book — Expert Oracle Practices: Oracle Database Administration from the Oak Table — that I co-authored with the bunch of other Oakies. I contributed chapter 1 in the book titled just like my presentation — Battle Against Any Guess. The plan was to give a copy away during the presentation and do a draw for another one at the end of the session. I was so nervous that I forgot about it until the end of the session so I just did a draw for two copies. The lucky winners were Lynn-Georgia Tesch and Surendra Anchula. Congratulations! For the rest of you who left the contact details — please stay tuned and we’ll organize few things online.

Now the main topic of this post — my presentation. What’s unusual about this session is that it’s not some technical stuff that I usually do but a more conceptual and motivational talk. Could I pull it off? Well, I think it went fairly well in general even though I did identify few rough places and my lack of English language mastering. Might need to work a little bit more on the flow of the presentation.

We had quite a few good laughs. Later, people in the next hall were asking about it and Dan was making the jokes on the stage so it must have been loud. Anyway, I think nobody fell asleep and I managed to get people thinking about the topic. I received many “thank you” notes yesterday and compliments on a good session so by the end of the day I was more and more pleased. Thanks everyone for attending and especially big thanks to those of you who brought to my attention examples from their own battles. If you have more to discuss — contact me by email (my last name) {at} pythian.com.

Thanks to Marco Gralike for recording some fragments and sharing them. I think he has more to come.

This is the introductory couple minutes. You can definitely notice how nervous I am starting on the stage:

Solving the wrong problem example:

That’s all for now. Stay tuned — more to come.

PlanetMySQL Voting: Vote UP / Vote DOWN

How do I identify the MySQL my.cnf file?

Ronald Bradford — Tue, 09 Mar 2010 19:14:19 +0000

As part of my upcoming FREE my.cnf check advice I first need to ask people to provide the current MySQL configuration file commonly found as a file named my.cnf

If only that question was easy to answer!

Use of configuration files

MySQL will by default use at least one configuration file from the following defaults. MySQL also uses a cascade approach for configuration files. When you have multiple files in the appropriate paths you can see unexpected behavior when you override certain values in different files.

You can however for example specify –no-defaults to use no configuration file, or add options to your command line execution, so even looking at all configuration files is no guarantee of your operating configuration.

However for most environments, these complexities do not exist.

Default Location

By default and on single instance MySQL servers you are most likely to find this file called my.cnf and found at:

/etc/my.cnf
/etc/mysql/my.cnf

These are known as the global options files.

Alternative Locations

MySQL has both instance specific and user specific locations. For the inclusion of an instance specific file, the location is:

$MYSQL_HOME/my.cnf

where MYSQL_HOME is a defined environment variable. Historical MySQL versions also looked at [datadir]/my.cnf however I am unaware if this is applicable in 5.x versions.

You can also specific options on a per user basis for default inclusion. These are found at:

$HOME/.my.cnf

Distro specific locations

Ubuntu for example also provides an ability to add options via an include directory.

Specifying a configuration at runtime

While you may have these default files, you may elect to start mysql with a specific configuration file as specified by –defaults-file. This option will override all global/instance/user locations and use just this configuration file. You can also specify additional configuration that supplements and not overrides the default with –defaults-extra-file.

What files are on my system?

Again, assuming the default names you can perform a brute force check with:

$ sudo find / -name "*my*cnf"

This is actually worthwhile, especially if you find a /root/.my.cnf file which is default MySQL settings for the Operating System ‘root’ user.

MySQL recommendations

MySQL by default provides a number of recommended files however these are generally outdated especially for newer hardware. These files include my-huge.cnf, my-large.cnf, my-medium.cnf, my-small.cnf and my-innodb-heavy-4G.cnf. Don’t assume replacing your configuration with one of these files will make your system perform better.

MySQL made some attempt to correct these and at least some very poor defaults with MySQL 5.4 however I am unsure what’s in MySQL 5.5

MySQL Configuration at runtime

While several commands can help with identifying your configuration files and print defaults etc, it’s also possible to change your configuration at runtime. It’s possible that these changes are not reflected in your configuration files and pose an additional mismatch.

References

PlanetMySQL Voting: Vote UP / Vote DOWN

Data Comparison Methods Overview

Julia Samarska — Tue, 09 Mar 2010 15:38:28 +0000

Data comparison is a difficult and resource-intensive process. For convenience, this process can be divided into several steps.
First, you should compare tables from one database on one server with the database on the other server. You should choose columns for data comparison, and also choose a column that will be a comparison key.
The next step is to choose all data from these tables or some specified part of the data.
The third and the most important step is comparison of the two tables by the selected comparison key itself. During this process the status of each record is set to “only in source”, “only in target”, “different”, or “equal”.
The final steps of the data comparison process are including records to the synchronization and synchronization itself. During these steps records needed for synchronization are chosen, update script is created, and after that the script is executed.
You can read a detailed description of the comparison process here.

Now let’s look at the third step (data comparison) thoroughly.

There are several ways of data comparison that differ only by the side where data comparison is going to be performed – on the server side or on the client PC.

Data comparison on the server side is performed using the resources of the server.
The algorithm of comparison is the following:
1. For each record of each of the two tables its checksum is calculated;
2. Then the checksum of every record from one table is compared to the checksum of the corresponding record from another table and conclusion if the records are equal or different is made;
3. The comparison result is stored in a temporary table on the server.

Performance indicators:
1. The speed of data comparison directly depends on the server capacity and occupancy;
2. The maximal size of database for comparison is limited by the resources of the server itself.

Advantages:
1. There is no need to transfer large amounts of data for comparison to the client PC through network. This way we save network traffic;
2. The speed of comparison does not depend on the client PC resources;
3. Ability to compare blob data of any size.

Disadvantages:
1. Because of the record checksum calculation algorithm in some cases different data can result in equal checksum, and instead of the expected “different” status the “equal” status will be received;
2. There is no flexibility in the synchronization and comparison options usage;
3. There is no possibility to view records differences and exclude a part of the records from the synchronization manually;
4. During the synchronization script creation you should perform data transfer from the server to the client side;
5. The control checksum calculation of a large amount of records consumes all server resources;
6. One should provide extra space on the server for the comparison results storage in the temporary table.

As we can see, this way of comparison has more disadvantages than advantages, that’s why this way is rarely used.

Data comparison on the client PC is performed using the client machine resources, and the server only provides data for comparison. In turn, this way of comparison can be divided into several more ways depending on the way how comparison information will be stored.

Comparing Data on local PC when comparison result is stored in RAM.
The comparison algorithm is the following:
1. Server passes all data from both tables to the local PC;
2. Every record of every table is placed to RAM and is compared without checksum calculation;
3. If a record gets “only in source”, “only in target” or “equal” status, only comparison key is stored in RAM. If records get “different” status, they are placed to RAM for storage completely.

Performance indicators:
1. The speed of data comparison directly depends on the client PC resources and on the speed of data transfer through the network;
2. Maximum size of the database for comparison is limited by the size of RAM on the client PC, and this maximum size also depends on the degree to which the databases that should be compared are different – the smaller is the amount of different records, the larger databases can be compared.

Advantages:
1. Minimal server occupancy – server performs only simple data selection;
2. The simplest algorithm of data comparison because records are sorted on the client side;
3. Flexibility in the comparison options usage;
4. Minimal size of the comparison data store;
5. Status of every record for any data is always correct.

Disadvantages:
1. To view records with “only in source”, “only in target”, or “equal” status an extra data selection is needed;
2. An extra data selection is needed to create a synchronization script;
3. OutOfMemory Exception may be arisen when there are a lot of differences in data in databases;
4. Possibility to compare blob data only of the size that equals to the size of free RAM.

This way of comparison is implemented in dbForge Data Compare for SQL Server v1.10, dbForge Data Compare for MySQL v2.00 and allows to compare databases of any size if data in these databases does not differ a lot.

Comparing Data on local PC when comparison result is stored as a cashed file on the disk.
The algorithm of comparison is the following:
The server passes all data from both tables sorted by comparison key to a local PC. Data is read by bytes, compared without checksum calculation and written to a file on the disk.

Performance indicators:
1. The speed of data comparison directly depends on the client PC resources and on the speed of data transferring through the network;
2. The maximum size of a database to compare is limited by free disk space and does not depend on the degree of data difference in databases.

Advantages:
1. Medium server occupancy – server performs data sorting and selection;
2. To view records and synchronization script creation extra requests to the server are not necessary;
3. The status for every record is always correct for any data;
4. Possibility to compare blob data of the size equal to the size of free space available on the disk.

Disadvantages:
1. Difficult algorithm of data comparison for the records comparison key of which is of the string data type;
2. Difficult algorithm of disk cash for temporary information storage creation.

We can see that in this case the only disadvantage of this way of comparison is the difficulty of implementation. There are more advantages than in the ways of comparison listed above. That’s why this way of comparison will be used in the new version of dbForge Data Compare for SQL Server v2.00 and dbForge Data Compare for MySQL v3.00 for data comparison.

PlanetMySQL Voting: Vote UP / Vote DOWN

MySQL Version Updates

Michal Hrušecký — Tue, 09 Mar 2010 15:19:25 +0000

Few weeks ago I was at FOSDEM. It was really amazing experience. I meet many interesting people, learned quite some thing and I returned full of enthusiasm. Open Source events are really great.

But all the fun wasn't over even after the FOSDEM. I spent few more days in Bruxelles attending MySQL packagers meeting organized by SUN/Oracle. We spent quite some time talking to each other. We learned what MySQL people are doing and how. And they learned how do we deal with MySQL and what is troubling us. And many good things will come from this.

First but certainly not last of them is about to appear now. One very interesting thing we learned at meeting was MySQL release policy. What openSUSE and Ubuntu and maybe some others are doing is that after release date there is generaly no version updates allowed. We are only fixing serious bugs and security related issues. It takes quite some work. What we learned is that new releases in stable branch of MySQL are in fact maintanance updates. If you update from 5.1.43 to 5.1.44 you wouldn't get any new features. All you will get are bugfixes. And only bugfixes of serious or security related issues. Does it sound familiar? Yes it is the same thing we are doing! So I discussed it with our maintanance team. And we came to the conclusion that we want to give our users all serious fixes. Not only these few selected. And the best way to do it is to use maintanance updates provided by MySQL people themself. I'm not saying that I don't have enough confidence to play with MySQL sources, but I think that MySQL people can do it better

Yes, you are guessing right. What I'm trying to say is that we are going to update MySQL to the latest available version. This means 5.1.44 for openSUSE 11.2 and 5.0.90 for older openSUSE. We will start with 11.2 as version gap is smaller there and if everything will proceed smoothly, we will continue with 11.1 and 11.0. For 11.2 you can help by testing update. Currently 5.1.44 update is prepared for 11.2 in server: database: STABLE and I'm running some final tests. If you want, you can try it too (not recomended on production servers yet) and if you'll find any problems, please report them before it will hit official updates.

Remember, this is just the beginning. I've got some bigger plans regarding MySQL in 11.3

PlanetMySQL Voting: Vote UP / Vote DOWN

SQLyog MySQL GUI 8.3 Has Been Released

Webyog — Tue, 09 Mar 2010 14:04:16 +0000

Changes (as compared to 8.22) include:

Features:
* Added an option to define a ‘color code’ for a connection. The color will be used as background color in the Object Browser.
* A Query Builder session can now be saved and resumed.
* In Query Builder a table alias can be defined for any table by double-clicking the title bar of the table symbol.
* In RESULT tab results can now be retrieved page-wise. This is ON as default with this build with a defined LIMIT of 1000 rows. For a specific query user can change and for this specific query the setting is persistent across sessions. Also read ‘miscellaneous’ paragraph below.
* Added a context menu to Query Builder canvas.

Bug Fixes:
* Deleting a user would leave non-global privileges orphaned in the ‘mysql’ database. Now we use DELETE USER syntax if server supports.
* Also using EDIT USER dialogue to change host or user specifier for a user would not move non-global privileges. We have split the old ALTER USER dialogue into two: a EDIT USER and RENAME USER dialogue. The latter will use RENAME USER syntax if server supports.
* On Wine Data Sync could generate a malformed XML-string what would case Data Sync to abort.
* Fixed an issue where SSH-tunneling failed with public/private key authentication. Technically the fix is in the PLINK binary shipped with SQLyog.
* SJA failed to send notification mails if Yahoo SMTP servers were used. Note that the fix disables encryption option with Yahoo SMTP servers – but it won’t work anyway due to a non-standard SMTP implementation server-side.
* When importing data from a Universe ODBC-source string data could be truncated.
* The fix in 8.22 for the issue that horizontal scrollbar in GRID would sometime not appear was not complete. It could still happen.
* SQLyog will now trim trailing whitespaces in Connection Manager and Create object dialogs to avoid MySQL Errors..
* Opening a file from ‘recent files’ list could crash SQLyog if a Query Builder or Schema Designer tab was selected and the file specified was not a valid XML file for that tab. This bug was introduced in beta 1.
* When calling a Stored Procedure with more than one SELECT statement from ‘Notifications Services’ only one result set was sent by mail.
* The sja.log file had no line-breaks between what was recorded for two jobs.
* On multi-monitor system resizable dialogues could open on the wrong monitor. New implementation is like this: on multi-monitor systems main program dialogue and ‘first child dialogue’ (example: ALTER TABLE) will open where they were closed (if possible), second and higher child dialogues (example: table advanced properties) will always open on top of its ‘parent’ dialogue. Non-resizable dialogues (such as confirmation boxes) will always open on top of their ‘parent’.
* With multiple SSH-tunnelled connections open stopping and re-executing queries in multiple connections in a fast manner could crash SQLyog.
* If more than one comment occurred before a SELECT statement in the editor, the statement was not identified as a SELECT statement by the Query Profiler and the Query Profiler TAB would not display.
* We did not validate client-side if user checked atoincrement option for a bit column with Create/Alter table dialog.
* If an error occurred while renaming a trigger then trigger was lost as SQLyog was not recreating it back.
* Small GUI fixes.

Miscellaneous:
* The default LIMIT setting for DATA tab has been removed. The setting is not required since we introduced table-level persistence for number of rows displayed. The default for new tables that have not been opened before is 50 – but when user changes the value and next ‘refresh’es SQLyog will save the LIMIT for that particular table persistently across sessions. This in combination with page-wise display in RESULT tab results in a more uniform User Interface for DATA and RESULT tabs.

Downloads: http://webyog.com/en/downloads.php
Purchase: http://webyog.com/en/buy.php

PlanetMySQL Voting: Vote UP / Vote DOWN

Maatkit BoF session at the MySQL conference

Baron Schwartz (xaprb) — Tue, 09 Mar 2010 13:15:06 +0000

I’ve submitted a Birds of a Feather session for Maatkit at the upcoming MySQL conference. It’s not on the public schedule yet, but it has been accepted and scheduled for 19:00 on 13 Apr 2010. See you there!

Related posts brought to you by Yet Another Related Posts Plugin.

PlanetMySQL Voting: Vote UP / Vote DOWN

Tip: faster than TRUNCATE

Shlomi Noach — Tue, 09 Mar 2010 11:37:01 +0000

TRUNCATE is usually a fast operation (much faster than DELETE FROM). But sometimes it just hangs; I’ve has several such uncheerful events with InnoDB (Plugin) tables which were extensively written to. The TRUNCATE hanged; nothing else would work; minutes pass.

TRUNCATE on tables with no FOREIGN KEYs should act fast: it translate to dropping the table and creating a new one (and it all depends on the MySQL version, see the manual).

What’s faster than TRUNCATE, then? If you don’t have triggers nor FOREIGN KEYs, a RENAME TABLE can come to the rescue. Instead of:

TRUNCATE log_table

Do:

CREATE TABLE log_table_new LIKE log_table;
RENAME TABLE log_table TO log_table_old, log_table_new TO log_table;
DROP TABLE log_table_old;

I found this to work well for me. Do note that AUTO_INCREMENT values can be tricky here: the “new” table is created with an AUTO_INCREMENT value which is immediately taken in the “working” table. If you care about not using same AUTO_INCREMENT values, you can:

ALTER TABLE log_table_new AUTO_INCREMENT=some high enough value;

Just before renaming.

I do not have a good explanation as for why the RENAME TABLE succeeds to respond faster than TRUNCATE.

PlanetMySQL Voting: Vote UP / Vote DOWN

Speaking at the O'Reilly MySQL Conference & Expo: "A look into a MySQL DBA's toolchest"

Lenz Grimmer — Tue, 09 Mar 2010 10:38:27 +0000

I'm happy to announce that my talk "Making MySQL administration a breeze - a look into a MySQL DBA's toolchest" has been accepted for this year's edition of the MySQL Conference & Expo in Santa Clara, which will take place on April 12-15, 2010. The session is currently scheduled for Wednesday 14th, 10:50 in Ballroom E.

My plan is to provide an overview over the most popular utilities and applications that a MySQL DBA should be aware of to make his life easier. The focus will be on Linux/Unix applications available under opensource licenses that ease tasks related to user administration, setting up and administering replication setups, performing backups and security audits.

Of course I will cover the usual suspects (e.g. Maatkit), some of these are actually collections of different utilities by themselves. As it's impossible to go over each individual component in the given time frame, I will try to pick out the most popular/useful parts related to the scopes mentioned above. But I will also cover some lesser known gems that migh be worth taking a look at. What's your the most valued tool in your toolchest? I am still looking for more inspiration.

I look forward to being at the conference again and meeting with colleagues and friends in the MySQL community. Judging from the current schedule, it will be a very interesting mix of talks.

If you're interested in attending, you should consider registering soon! The early registration ends on March 15th. Until then, I encourage you to make use of this "Friend of Speaker" discount code (25% off): mys10fsp

PlanetMySQL Voting: Vote UP / Vote DOWN

Drizzle BoF at the MySQL Conference and Expo

Stewart Smith — Tue, 09 Mar 2010 04:46:39 +0000

At the 2010 O’Reilly MySQL Conference and Expo there will be a Drizzle BoF!

It’s currently scheduled for 7pm on April 13th.

Come along, it will be awesome.

PlanetMySQL Voting: Vote UP / Vote DOWN

Speaking At The MySQL Users Conference

Gerardo Narvaja — Tue, 09 Mar 2010 00:10:00 +0000

My proposal has been accepted, yay!

I'll be speaking on a topic that I feel passionate about: MySQL Server Diagnostics Beyond Monitoring. MySQL has limitations when it comes to monitoring and diagnosing as it has been widely documented in several blogs.

My goal is to share my experience from the last few years and, hopefully, learn from what others have done. If you have a pressing issue, feel free to comment on this blog and I'll do my best to include the case in my talk and/or post a reply if the time allows.

I will also be discussing my future plans on sarsql. I've been silent about this utility mostly because I've been implementing it actively at work. I'll post a road map shortly based on my latest experience.

I'm excited about meeting many old friends (and most now fellow MySQL alumni) and making new ones. I hope to see you there!

PlanetMySQL Voting: Vote UP / Vote DOWN

MySQL Drizzle team joins Rackspace

David Van Couvering — Mon, 08 Mar 2010 23:38:00 +0000

Well, more defections from Oracle, it's clear where the wind is blowing. It's as if all the cool and interesting stuff is quickly shedding itself from Oracle.

Jay Pipes has a good blog post about the announcement and the history behind them ending up at Rackspace.

Interesting quote:

"Rackspace is also heavily invested in Cassandra, and sees integration of Drizzle and Cassandra as being a key way to add value to its platforms and therefore for its customers".

I look forward to seeing what that's about.

I also liked this from Jay:

"I don't know whether Larry understands that cloud computing and infrastructure-as-a-service, platform-as-a-service, and database-as-a-service will eventually put his beloved Oracle cash cow in its place or not. I don't know whether Oracle is planning on embracing the cloud environments which will continue to eat up the market share of more traditional in-house environments upon which their revenue streams depend. I really don't."

PlanetMySQL Voting: Vote UP / Vote DOWN

An SQL Puzzle?

Jay Pipes — Mon, 08 Mar 2010 22:04:42 +0000

Dear Lazy Web,

What should the result of the SELECT be below? Assume InnoDB for all storage engines.

CREATE TABLE t1 (a int, b int);
insert into t1 values (1,1),(1,2);
CREATE TEMPORARY TABLE t2 (a int, b int, primary key (a));
BEGIN;
INSERT INTO t2 values(100,100);
CREATE TEMPORARY TABLE IF NOT EXISTS t2 (primary key (a)) select * from t1;

# The above statement will correctly produce an ERROR 23000: Duplicate entry '1' for key 'PRIMARY'
# What should the below result be?

SELECT * from t2;
COMMIT;

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN

Chris is speaking at the MySQL 2010 Conference

Chris Schneider — Mon, 08 Mar 2010 21:21:28 +0000

I'll be presenting two talks this year:

Faster Than Alter - Less Downtime

"This will be a informative talk about real world problem solving and the powerful yet sometimes overlooked LOAD DATA INFILE command. This talk is for MySQL DBAs who want to expand their
knowledge, improve performance and decrease customer facing downtime"

Get Your Replication On: Advanced Techniques, Tips and Tricks

Co-speaking with Sarah Sproehnle for this one! We have lots of interesting uses for replication and some best practices up our
sleeves. Warning: we won't be covering how to set up basic replication
- that's a prerequisite for this talk!

Hope to see you there!

PlanetMySQL Voting: Vote UP / Vote DOWN

Qsh.pl: distributed query tool

Gavin Towey — Mon, 08 Mar 2010 20:48:00 +0000

I've written quite a few tools over time to connect to many mysql servers and run queries. Most of these have been pretty specific to a small set of tasks such as running an alter across many servers. Any sysadmin that is in charge of many servers is probably familiar with dsh, and as I was using recently I realized how all those specific tools I've written for mysql could be generalized into a dsh like tool. Thus, Qsh.pl was born! (download at launchpad)

Usage should be familiar to anyone who has used dsh before, it even will read group files made for dsh in /etc/dsh/group/or /usr/local/etc/group/.

Here's an example where this tool was quite useful. I was getting a query error for SHOW GLOBAL STATUS. This was a curious result since we're running mysql 5.0 everywhere. So what better way to find out which machines are complaining than just run it everywhere:


# qsh.pl -Mcg all_servers --user root --ask-pass --db=test -e 'SHOW GLOBAL STATUS'  2>error.log
{snip ... lots of output}
Done. Total time 2.919

My group file for all_servers includes 120 mysql servers, executing that query on all of them took a total of 2.9 seconds, not bad. I also redirected stderr to a file, so any query errors are easy to find:

cat error.log
myserver1: Query Error (1064) You have an error in your SQL syntax ...
myserver2: Query Error (1064) You have an error in your SQL syntax ...

Ok, we found all the servers that return an error, why do they complain?

# qsh.pl -Mcm myserver1,myserver2,myserver3 --user root --ask-pass --db=test -e 'SELECT VERSION()'
Password:
myserver1: +-----------------+
myserver1: |       VERSION() |
myserver1: +-----------------+
myserver1: | 4.1.21-standard |
myserver1: +-----------------+
myserver3: +----------------------------+
myserver3: |                  VERSION() |
myserver3: +----------------------------+
myserver3: | 5.0.66a-enterprise-gpl-log |
myserver3: +----------------------------+
myserver2: +-----------------+
myserver2: |       VERSION() |
myserver2: +-----------------+
myserver2: | 4.1.21-standard |
myserver2: +-----------------+
Done. Total time 0.063

Ooops! That's right, still a few old versions for legacy reasons.

That's just one example of how I used it. There are probably lots of use cases out there, but since it's new I'm still learning to rely on it. It certainly makes things faster when I can think about querying many servers at once, and is a more efficient way to work when dealing with many machines. It might be useful for:

+ comparing explain plan between many machines
+ altering large tables across many slaves, before promoting one to master.
+ grabbing status output from many machines to feed into awk or sed

PlanetMySQL Voting: Vote UP / Vote DOWN

International Women’s Day

Sheeri K. Cabral — Mon, 08 Mar 2010 19:52:16 +0000

If you do not know what International Women’s Day is: http://www.internationalwomensday.com/

Start planning your blog posts for Ada Lovelace day now (March 24th, http://findingada.com/ Ada Lovelace Day is an international day of blogging (videologging, podcasting, comic drawing etc.!) to draw attention to the achievements of women in technology and science.)

To that end, I would like to point out all the women currently in science and tech fields that I admire and think are doing great things. I think it would be great if everyone, male or female, made a list like this:

The women that have taught me science/tech along the way:

High School:
Mary Lou Ciavarra (Physics)
Maria Petretti (Pre-Algebra, and Academic Decathlon)
Reneé Fishman (Biology)
Lisa Acquaire (Economics during Academic Decathlon)

College:
Professor Kalpana White (Biology), and in whose fruit fly lab I worked for 2 semesters.
Professor Eve Marder (Introductory Neuroscience)

Though Brandeis does have female faculty in the Computer Science department, I did not manage to have any classes with female Computer Science faculty members.

My current female DBA co-workers at Pythian: Isabel Pinarci (Oracle), Michelle Gutzait (SQL Server), Catherine Chow (Oracle) and Jasmine Wen (Oracle).

And to folks in the greater MySQL/tech community and tech co-workers past and present, especially those I have been inspired and helped by: Tracy Gangwer, Selena Deckelmann (Postgres), Amy Rich, Anne Cross, and more (If I have forgotten you, I apologize!).

PlanetMySQL Voting: Vote UP / Vote DOWN

InnoDB plugin training

Dave Stokes — Mon, 08 Mar 2010 18:27:00 +0000

MySQL is a constantly moving target and keeping up on the latest changes can be a difficult chore. A lot of folks seeking certification are often swamped by evolutions in the MySQL product. On Tuesday there is a FREE web seminar where MySQL Professional Services experts will walk you through best practices for achieving performance and scalability improvements using MySQL 5.1 and the new InnoDB Plugin. This is part two of the series and you do not have to have seen part one to join.

List of Web Seminars

PlanetMySQL Voting: Vote UP / Vote DOWN

Don’t Assume – Per Session Buffers

Ronald Bradford — Mon, 08 Mar 2010 18:24:25 +0000

MySQL has a number of global buffers, i.e. your SGA. There are also a number of per session/thread buffers that combined with other memory usage constitutes an unbounded PGA. One of the most common errors in mis-configured MySQL environments is the setting of the 4 primary per session buffers thinking they are global buffers.

Global buffers include:

key_buffer_size – For MyISAM Indexes (note you can define multiple key_buffer’s The MyISAM Key Cache)
innodb_buffer_pool_size – For Innodb Table/Indexs
innodb_additional_mem_pool_size – Innodb additional data dictionary data
query_cache_size – The MySQL Query Cache

The four important per session buffers are:

I have seen people see these values > 5M. The defaults range from 128K to 256K. My advice for any values above 256K is simple. What proof do you have this works better? When nothing is forthcoming, the first move is to revert to defaults or a maximum of 256K for some benchmarkable results. The primary reason for this is MySQL internally as quoted by Monty Taylor – for values > 256K, it uses mmap() instead of malloc() for memory allocation.

These are not all the per session buffers you need to be aware of. Others include thread_stack, max_allowed_packet,binlog_cache_size and most importantly max_connections.

MySQL also uses memory in other areas most noticeably in internal temporary tables and MEMORY based tables.

As I mentioned, there is no bound for the total process memory allocation for MySQL, so some incorrectly configured variables can easily blow your memory usage.

References

Read Buffer performance hit by Monty Taylor

About “Don’t Assume”

“Don’t Assume” is a series of posts to help the Oracle DBA understand, use and appreciate the subtle differences and unique characteristics of the MySQL RDBMS in comparison to Oracle. These points as essential to operate MySQL effectively in a production environment and avoid any loss of data or availability.

For more posts in this series be sure to follow the mysql4oracledba tag and also watch out for MySQL for Oracle DBA presentations.

The MySQLCamp for the Oracle DBA is a series of educational talks all Oracle DBA resources should attend. Two presentations from this series IGNITION and LIFTOFF will be presented at the MySQL Users Conference 2010 in Santa Clara, April 2010 This series also includes JUMPSTART and VELOCITY. If you would like to here these presentations in your area, please contact me.

Vote UP

Vote DOWN

[Finally] Slides posted for the DW Breakfast in London

Ivan Zoratti — Mon, 08 Mar 2010 17:49:00 +0000

It took a while, but here they are. We have posted only the slides from Sun/MySQL, since the other material is copyright by Infobright and Talend.

PlanetMySQL Voting: Vote UP / Vote DOWN

Free advice on your my.cnf

Ronald Bradford — Mon, 08 Mar 2010 17:36:51 +0000

Today, while on IRC in #pentaho I came across a discussion and a published my.cnf. In this configuration I found some grossly incorrect values for per session buffers (see below).

It doesn’t take a MySQL expert to spot the issues, however there is plenty of bad information available on the Internet and developers not knowing MySQL well can easily be mislead. This has spurred me to create a program to rid the world of bad MySQL configuration. While my task is potential infinite, it will enable me to give back and hopefully do a small amount of good. You never know, saving those CPU cycles may save energy and help the planet.

Stay tuned for more details of my program.

[mysqld]
...
sort_buffer_size = 6144K
myisam_sort_buffer_size = 1G
join_buffer_size = 1G
bulk_insert_buffer_size = 1G
read_buffer_size     = 6144K
read_rnd_buffer_size = 6144K
key_buffer_size		= 1024M
max_allowed_packet	= 32M
thread_stack		= 192K
thread_cache_size       = 256

query_cache_limit	= 512M
query_cache_size        = 512M
...

PlanetMySQL Voting: Vote UP / Vote DOWN

Drizzling from the Rackspace Cloud

Eric Day — Mon, 08 Mar 2010 16:57:20 +0000

Since I left Sun back in January, folks have been asking what was next. I’m happy to say that I’m going to continue hacking on open source projects like Drizzle and Gearman, but now at the Rackspace Cloud. Not only will I be there, but I get to continue working closely with a few of the amazing Drizzle hackers who have also joined, including Monty Taylor, Jay Pipes, Stewart Smith, and Lee Bieber.

Why Rackspace Cloud? Late last year I was considering what I wanted to do next with the Oracle acquisition looming near, and this was one of the options that presented itself. Rackspace had been a supporter of Drizzle from early on by offering virtual machines to develop and test on, and when talking to some folks more closely, something really hit home. Rackspace provides first-class service and “fanatical” support – they are not a software company. One might ask why an open source software developer would be interested in a company that doesn’t create software or vice-versa, and the answer is that Rackspace wants to find ways to offer the best possible service now and into the future. What better way than to help develop the next generation of service software and get a jump start into integrating this into their architecture? Both the open source community and Rackspace win.

Another thing I learned while talking with Rackspace is that one of their core principles is transparency. This applies to both customer and employees, and anyone within an open source community can appreciate this. The more I learned about the company and the folks within it, the more impressed I was at the lack of internal barriers or “need-to-know” information. One of Drizzle’s core goals is also transparency, from discussing design decisions on public mailing lists and IRC, to having the entire project management infrastructure hosted out in the open at Launchpad.

What does this mean for the Drizzle project? It means continued support for a number of core developers, more infrastructure for development, and most importantly in my eyes, more context. One of the Drizzle tag-lines is “A Lightweight SQL Database for Cloud and Web,” so what better place to develop a database designed for the cloud than on one of the fastest growing cloud platforms. We’ll get a detailed look at the demands, get feedback from cloud customers, and have the perfect test bed for offering new services. We’ll also be able to work closely with a top-notch group of DBAs, developers, and sysadmins in one of the most demanding service architectures out there. This invaluable context will help the Drizzle developers make more informed decisions moving forward, which also means better software for the community.

Personally, this also means getting back to my hosting roots. Before Sun, I worked at Concentric for almost 10 years in a clustered hosting environment. I’m very familiar with many of the multi-tenant scalability concerns Rackspace has, and I’m excited to be working in this type of environment again. We’ve already been working closely with the MySQL DBAs at Rackspace to learn what the biggest pain points are for a multi-tenant architecture, and we’ll be taking steps to address these as it will help anyone wanting to run Drizzle in a cloud-like environment. Drizzle’s modular architecture has already proved useful, as some of these concerns are easily answered with “oh, we have a plugin point for that.”

I’m excited, this is going to be a fun ride.

PlanetMySQL Voting: Vote UP / Vote DOWN

MySQL is crashing, what do I do?

Ronald Bradford — Mon, 08 Mar 2010 16:34:31 +0000

Let me start by saying the majority of environments never experience problems of MySQL crashing. I have seen production environments up for years. On my own server I have seen 575 days of MySQL uptime and the problem was hardware, not MySQL.

However it does occur, and the reasons may be obscure.

Confirming mysqld has crashed

To the unsuspecting, MySQL may indeed be crashing and you never know about it. The reason is because most MySQL installations have two running processes, these are mysqld and mysqld_safe.

ps -ef | grep mysqld
root     28822     1  0 Feb22 ?        00:00:00 /bin/sh bin/mysqld_safe
mysql    28910 28822  0 Feb22 ?        00:30:08 /opt/mysql51/bin/mysqld --basedir=/opt/mysql51 --datadir=/opt/mysql51/data --user=mysql --log-error=/opt/mysql51/log/error.log --pid-file=/opt/mysql51/data/dc1.onegreendog.com.pid

One of the functions of mysqld_safe is to restart mysqld if it fails. Unless you review your mysql error log and for low volume systems you will never know. Hint Have you checked your MySQL error log today?

You can determine quickly via SQL your instance uptime.

mysql> SHOW GLOBAL STATUS LIKE '%uptime%';
+---------------+---------+
| Variable_name | Value   |
+---------------+---------+
| Uptime        | 1033722 |
+---------------+---------+
1 row in set (0.00 sec)

This is the number of seconds since start time. While not easily readable for humans, this is more user friendly display. (NOTE: Works for 5.1+ only)

mysql> SELECT FROM_UNIXTIME(UNIX_TIMESTAMP() - variable_value) AS server_start
FROM INFORMATION_SCHEMA.GLOBAL_STATUS
WHERE variable_name='Uptime';
+---------------------+
| server_start        |
+---------------------+
| 2010-02-22 15:22:13 |
+---------------------+
1 row in set (0.07 sec)

Debugging a mysqld core file

When correctly configured, mysqld will generate a core file (See How to crash mysqld intentionally for background information on required settings).

Your first check is to determine if the mysqld binary used has debugging information and symbols stripped. You need this information not stripped for identifying symbol names.

$ file bin/mysqld
bin/mysqld: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.4.0,
dynamically linked (uses shared libs), for GNU/Linux 2.4.0, not stripped

You can use gdb and with a backtrace command (bt) you can see a stack trace of calls. This won’t help the average DBA without C or MySQL internal knowledge greatly, however it’s essential information to get to the bottom of the problem.

In the following example I’m going to use Bug #38508 to intentionally crash my test instance.

mysql> drop table if exists t1,t2;
mysql> create table t1(a bigint);
mysql> create table t2(b tinyint);
mysql> insert into t2 values (null);
mysql> prepare stmt from "select 1 from t1 join  t2 on a xor b where b > 1  and a =1";
mysql> execute stmt;
mysql> execute stmt;
ERROR 2013 (HY000): Lost connection to MySQL server during query

Lost connection is the first sign of a problem. We check the error log to confirm.

$ tail data/`hostname`.err
100306 14:51:49 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8384512
read_buffer_size=131072
max_used_connections=1
max_threads=151
threads_connected=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 338301 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x521f160
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x401b6100 thread_stack 0x40000
/home/rbradfor/mysql/mysql-5.1.38-linux-x86_64-glibc23/bin/mysqld(my_print_stacktrace+0x2e)[0x8abfbe]
/home/rbradfor/mysql/mysql-5.1.38-linux-x86_64-glibc23/bin/mysqld(handle_segfault+0x322)[0x5df252]
/lib64/libpthread.so.0[0x35fb00de80]
/home/rbradfor/mysql/mysql-5.1.38-linux-x86_64-glibc23/bin/mysqld(_ZN9Item_cond10fix_fieldsEP3THDPP4Item+0x7f)[0x5654ff]
/home/rbradfor/mysql/mysql-5.1.38-linux-x86_64-glibc23/bin/mysqld(_ZN9Item_cond10fix_fieldsEP3THDPP4Item+0xb8)[0x565538]
/home/rbradfor/mysql/mysql-5.1.38-linux-x86_64-glibc23/bin/mysqld(_Z11setup_condsP3THDP10TABLE_LISTS2_PP4Item+0xf6)[0x621f96]
/home/rbradfor/mysql/mysql-5.1.38-linux-x86_64-glibc23/bin/mysqld(_ZN4JOIN7prepareEPPP4ItemP10TABLE_LISTjS1_jP8st_orderS7_S1_S7_P13st_select_lexP18st_select_lex_unit+0x2db)[0x645f3b]
/home/rbradfor/mysql/mysql-5.1.38-linux-x86_64-glibc23/bin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x7a4)[0x654d24]
/home/rbradfor/mysql/mysql-5.1.38-linux-x86_64-glibc23/bin/mysqld(_Z13handle_selectP3THDP6st_lexP13select_resultm+0x16c)[0x659f9c]
/home/rbradfor/mysql/mysql-5.1.38-linux-x86_64-glibc23/bin/mysqld[0x5ec92a]
/home/rbradfor/mysql/mysql-5.1.38-linux-x86_64-glibc23/bin/mysqld(_Z21mysql_execute_commandP3THD+0x602)[0x5efb22]
/home/rbradfor/mysql/mysql-5.1.38-linux-x86_64-glibc23/bin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x3bd)[0x66587d]
/home/rbradfor/mysql/mysql-5.1.38-linux-x86_64-glibc23/bin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x7c)[0x66874c]
/home/rbradfor/mysql/mysql-5.1.38-linux-x86_64-glibc23/bin/mysqld(_Z22mysql_sql_stmt_executeP3THD+0xa7)[0x668c27]
/home/rbradfor/mysql/mysql-5.1.38-linux-x86_64-glibc23/bin/mysqld(_Z21mysql_execute_commandP3THD+0x1123)[0x5f0643]
/home/rbradfor/mysql/mysql-5.1.38-linux-x86_64-glibc23/bin/mysqld(_Z11mysql_parseP3THDPKcjPS2_+0x357)[0x5f5047]
/home/rbradfor/mysql/mysql-5.1.38-linux-x86_64-glibc23/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0xe93)[0x5f5ee3]
/home/rbradfor/mysql/mysql-5.1.38-linux-x86_64-glibc23/bin/mysqld(_Z10do_commandP3THD+0xe6)[0x5f67a6]
/home/rbradfor/mysql/mysql-5.1.38-linux-x86_64-glibc23/bin/mysqld(handle_one_connection+0x246)[0x5e9146]
/lib64/libpthread.so.0[0x35fb006307]
/lib64/libc.so.6(clone+0x6d)[0x35fa4d1ded]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x5249320 = select 1 from t1 join  t2 on a xor b where b > 1  and a =1
thd->thread_id=1
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file
100306 14:51:49 mysqld_safe Number of processes running now: 0
100306 14:51:49 mysqld_safe mysqld restarted
100306 14:51:49 [Note] Plugin 'FEDERATED' is disabled.
100306 14:51:50  InnoDB: Started; log sequence number 0 44233
100306 14:51:50 [Note] Event Scheduler: Loaded 0 events
100306 14:51:50 [Note] /home/rbradfor/mysql/mysql-5.1.38-linux-x86_64-glibc23/bin/mysqld: ready for connections.
Version: '5.1.38'  socket: '/tmp/mysql.sock.3999'  port: 3999  MySQL Community Server (GPL)

Confirming we got the “Writing a core file” line, we can find and use this.

$ find . -name "core*"
./data/core.23290

$ gdb bin/mysqld data/core.23290
GNU gdb Red Hat Linux (6.5-37.el5_2.2rh)
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/libthread_db.so.1".

Reading symbols from /lib64/libpthread.so.0...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libcrypt.so.1...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libnsl.so.1...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libm.so.6...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libgcc_s.so.1...done.
Loaded symbols for /lib64/libgcc_s.so.1
Core was generated by `/home/rbradfor/mysql/mysql-5.1.38-linux-x86_64-glibc23/bin/mysqld --defaults-fi'.
Program terminated with signal 11, Segmentation fault.
#0  0x00000035fb00b142 in pthread_kill () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00000035fb00b142 in pthread_kill () from /lib64/libpthread.so.0
#1  0x00000000005df285 in handle_segfault (sig=11) at mysqld.cc:2552
#2  
#3  0x00000000005654ff in Item_cond::fix_fields (this=0x5249dd0, thd=0x521f160, ref=) at item_cmpfunc.cc:3900
#4  0x0000000000565538 in Item_cond::fix_fields (this=0x52435b8, thd=0x521f160, ref=) at item_cmpfunc.cc:3912
#5  0x0000000000621f96 in setup_conds (thd=0x521f160, tables=, leaves=0x52494d0, conds=0x5244e38) at sql_base.cc:7988
#6  0x0000000000645f3b in JOIN::prepare (this=0x5243770, rref_pointer_array=0x5248a90, tables_init=, wild_num=, conds_init=,
    og_num=, order_init=0x0, group_init=0x0, having_init=0x0, proc_param_init=0x0, select_lex_arg=0x52488c0, unit_arg=0x5248498) at sql_select.cc:412
#7  0x0000000000654d24 in mysql_select (thd=0x521f160, rref_pointer_array=0x5c3fd0, tables=0x4, wild_num=0, fields=@0x52489c8, conds=0x52435b8, og_num=0, order=0x0, group=0x0, having=0x0,
    proc_param=0x0, select_options=0, result=0x52688a0, unit=0x5248498, select_lex=0x52488c0) at sql_select.cc:2377
#8  0x0000000000659f9c in handle_select (thd=0x521f160, lex=0x52483f8, result=0x52688a0, setup_tables_done_option=0) at sql_select.cc:268
#9  0x00000000005ec92a in execute_sqlcom_select (thd=0x521f160, all_tables=0x52494d0) at sql_parse.cc:5011
#10 0x00000000005efb22 in mysql_execute_command (thd=0x521f160) at sql_parse.cc:2206
#11 0x000000000066587d in Prepared_statement::execute (this=0x5245d60, expanded_query=, open_cursor=false) at sql_prepare.cc:3579
#12 0x000000000066874c in Prepared_statement::execute_loop (this=0x5245d60, expanded_query=0x401b43c0, open_cursor=false, packet=, packet_end=)
    at sql_prepare.cc:3253
#13 0x0000000000668c27 in mysql_sql_stmt_execute (thd=) at sql_prepare.cc:2524
#14 0x00000000005f0643 in mysql_execute_command (thd=0x521f160) at sql_parse.cc:2215
#15 0x00000000005f5047 in mysql_parse (thd=0x521f160, inBuf=0x5243520 "execute stmt", length=12, found_semicolon=0x401b6060) at sql_parse.cc:5931
#16 0x00000000005f5ee3 in dispatch_command (command=COM_QUERY, thd=0x521f160, packet=0x525fde1 "execute stmt", packet_length=) at sql_parse.cc:1213
#17 0x00000000005f67a6 in do_command (thd=0x521f160) at sql_parse.cc:854
#18 0x00000000005e9146 in handle_one_connection (arg=dwarf2_read_address: Corrupted DWARF expression.
) at sql_connect.cc:1127
#19 0x00000035fb006307 in start_thread () from /lib64/libpthread.so.0
#20 0x00000035fa4d1ded in clone () from /lib64/libc.so.6
(gdb) quit

You can use gdb to obtain additional information based on the type of information available.

Now what?

Is the problem a bug? Is it data corruption? Is it hardware related?

Gathering the information is the first step in informing you of more detail that will enable you to search, discuss and seek professional advice to address your problem.

References

MySQL Internals – Debugging a MySQL Server
What to Do If MySQL Keeps Crashing
Debugging mysqld under gdb
Hunting the core – An old by good intro article to cores and MySQL

PlanetMySQL Voting: Vote UP / Vote DOWN

Happiness is a Warm Cloud

Jay Pipes — Mon, 08 Mar 2010 16:31:34 +0000

Although a few folks knew about where I and many of the Sun Drizzle team had ended up, we've waited until today to "officially" tell folks what's up. We — Monty Taylor, Eric Day, Stewart Smith, Lee Bieber, and myself — are all now "Rackers", working at Rackspace Cloud. And yep, we're still workin' on Drizzle. That's the short story. Read on for the longer one

An Interesting Almost 3 Years at MySQL

I left my previous position of Community Relations Manager at MySQL to begin working on Brian Aker's newfangled Drizzle project in October 2008.

Many people at MySQL still think that I abandoned MySQL when I did so. I did not. I merely had gotten frustrated with the slow pace of change in the MySQL engineering department and its resistance to transparency. Sure, over the 3 years I was at MySQL, the engineering department opened up a bit, but it was far from the ideal level of transparency I had hoped to inspire when I joined MySQL.

For almost 3 years, I had sent numerous emails to the MySQL internal email discussion lists asking the engineering and marketing departments, both headed by Zack Urlocker, to recognize the importance and necessity of major refactoring of the MySQL kernel, and the need to modularize the kernel or risk having more modular databases overtake MySQL as the key web infrastructure database. The focus was always on the short term; on keeping up with the Jones' as far as features went, and I railed against this kind of roadmap, instead pushing the idea of breaking up the server into modules that could be blackboxed and developed independently of the kernel. My ideas were met with mostly kind responses, but nothing ever materialized as far as major refactoring efforts were concerned.

I remember Jim Winstead casually responding to one of my emails, "Congratulations, you've just reinvented Apache 2.0". And, yes, Jim, that was kind of the point...

The MySQL source code base had gotten increasingly unmaintainable over the years, and key engineers were extremely resistant to changing the internals of MySQL and modernizing it. There were some good reasons for being resistant, and some poor reasons (such as "this is the way we've always done it"). Overall, it's tough to question the strategy that Zack, Marten Mickos, and others had regarding the short term gains. After all, they managed to maneuver MySQL into a winning position that Sun Microsystems thought was worth one billion dollars. Because of this, it's tough to argue with them.

Working on Drizzle since October 2008 (officially)

I'm not the kind of person which likes to wait for years to see change, and so the Drizzle project interested me because it was not concerned with backwards compatibility with MySQL, it wasn't concerned with having a roadmap that was dependent on the whims of a few big customers, and it was very much interested in challenging the assumptions built into a 20 year-old code base. This is a project I could sink my teeth into. And I did.

Many folks have said that the only reason Drizzle is still around is because Sun continued to pay for a number of engineers to work on Drizzle as "an experiment of sorts" and that Drizzle has no customers and therefore nothing to lose and everything to gain. This was true, no doubt about it. At Sun CTO Labs, the few of us did have the ability to code on Drizzle without the pressure-cooker of product marketing and sales demands. We were lucky.

4 6 9 10 Months in Purgatory

So, around rolls April 2009. The stock market and worldwide economy had collapsed and recession was in the air. There's one thing that is absolutely certain in recession economies: companies that have poor leadership and direction and are beholden to the interests of a large stockholder will seek an end to their misery through acquisition by a larger, stronger firm.

And Sun Microsystems was no different. JAVA stock plummeted to two dollars a share, and Jonathan Schwartz and the Sun board began shopping Sun around to the highest bidder. IBM was courted along with other tech giants. So was Oracle.

And it was with a bit of a hangover that I awoke at the MySQL conference in April 2009 to the news that Oracle had purchased Sun Microsystems. Joy. We'd just gone through 14 months of ongoing integration with Sun Microsystems and now it was going to start all over again.

Anyone who follows PlanetMySQL knows about the ensuing battle in the European Commission's court regarding monopoly of Oracle in the database market with its acquisition of MySQL. Monty Widenius, Eben Moglen, even Richard Stallman, weighed in on the pros and cons of Oracle's impending control over MySQL.

All the while, us Sun Microsystems employees had to hold our tongues and try to keep our jobs as Sun laid off thousands more workers while the EC battle ensued. Not fun. It was the employment equivalent of purgatory. And the time just dragged on, with many employees, including myself and the Sun Drizzle team, not having a clue as to what would happen to us. Management was completely silent about future plans. Oracle made zero attempts to outline its future strategy regarding software, and thus most software employees simply kept on doing their work not knowing if the pink slip was arriving tomorrow or not. Lots of fun that was.

Oracle Doesn't Need Our Services — Larry Don't Need No Stinkin' Cloud

The acquisition finally closed and very shortly afterwards, I got a call from my boss, Lee Bieber, that Oracle wouldn't be needing our services. Monty, Eric, and Stewart had already resigned; none of them had any desire to work for Oracle. Lee and I had decided to see what they had in mind for us. Apparently, not much.

Larry Ellison has gone on record that the whole "cloud thing" is faddish. I don't know whether Larry understands that cloud computing and infrastructure-as-a-service, platform-as-a-service, and database-as-a-service will eventually put his beloved Oracle cash cow in its place or not. I don't know whether Oracle is planning on embracing the cloud environments which will continue to eat up the market share of more traditional in-house environments upon which their revenue streams depend. I really don't.

But what I do know is that Rackspace is betting that providing these services is what the future of technology will be about.

Happiness is a Warm Cloud

Our team has landed at Rackspace Cloud. I've now been down to San Antonio twice to meet with key individuals with whom we'll be working closely. Rackspace is not shy about why the wanted to acquire our team. They see Drizzle as a database that will provide them an infrastructure piece that will be modular and scalable enough to meet the needs of their very diverse Cloud customers, of which there are many tens of thousands.

Rackspace recognizes that the pain points they feel with traditional MySQL cannot be solved with simple hacks and workarounds, and that to service the needs of so many customers, they will need a database server that thinks of itself as a friendly piece of their infrastructure and not the driver of its applications. Drizzle's core principles of flexibility and focus on scalability align with the goals Rackspace Cloud has for its platform's future.

Rackspace is also heavily invested in Cassandra, and sees integration of Drizzle and Cassandra as being a key way to add value to its platforms and therefore for its customers.

Rackspace is all about the customers, and this is a really cool thing to experience. It's typical for companies to claim they are all about the customer — in fact, every company I've ever worked for has claimed this. Rackspace is the first company I've worked for where you actually feel this spirit, though. You can see the fanaticism of Rackers and how they view what they do always in terms of service to the customer. It's infectious, and I'm pretty psyched to be on their team.

Anyway, that's my story and I'm stickin' to it. See y'all on the nets.

PlanetMySQL Voting: Vote UP / Vote DOWN

Actually, the Relational Model doesn't scale

Robert Treat — Mon, 08 Mar 2010 16:20:56 +0000

Before all my fellow DBAs' heads explode, let me just say that I am a relational guy. I like the relational model, think it's the best tool for the job, and think every programmer (not just DBA's) should aspire to be as familiar with it as they are with AJAX, MVC, or whatever other technology pattern you think is important. I'll even take that a step further; I think the NoSQL movement is mostly a re-hash of failed technologies from the last century. Object and document databases had their run in the market (some might say "they had their time"), and they were pretty thoroughly beaten by the RDBMS; that some people have reinvented that wheel doesn't change the game.

That said, I find the recent comments from Jeff Davis on the relational model and scalability to be overlooking some things. The state of computing tasks has changed over the past two decades, and what we know about computer engineering has also changed. Working on highly scalable systems like we do at OmniTI, you can't escape some of the inherent problems that you face when working in these types of environments. As much as I'd like the answer to every problem to be "just use an RDBMS", Brewer's CAP theorem just isn't something you can ignore.

When most people think about the relational model, they think of it in terms of parent-child relationships between tables. Without getting too deep in the details of it, I think it's pretty fair to say that Primary Keys and Foreign Keys are very large part of any relational implementation, and that pretty much all RDBMS strive to allow you to add these constraints to your model; it's what helps keep the data consistent. But there's the rub. CAP theorem points out that as we strive for tighter and tighter consistency, we are pulling away from availability, and sacrificing partition tolerance. Two theoretical systems that run smack dab into each other in the real world. This isn't really something new; if you have ever de-normalized, dropped a foreign key, or split data across multiple nodes, you've run into this before.

Now, where CAP theorem falls on it's face (imho) is that it also ignores another holy trinity of software development; Cheap, Fast, and Good. The size of your problem is dictated by the resources you have available; if you can afford decent tools (and let's be clear, decent is not your web dev throwing up MySQL on an EC2 instance) it is quite likely that the stressors of the relational model will never impact you in a way that most CAP folks are worried about. This is also one of the places the NoSQL movement fails; by throwing the baby out with the bath water. Giving up your data integrity before you have scalability issues is a form of premature optimization. The trick, as Theo would say, is having the experience to know when such optimizations are and aren't premature.

So what's the take away? I like to say that you use the relational model because it is best, and you use something else because it is necessary. Most SQL implementations can scale very well, and they should be your first choice when starting a new project. But we also can't pretend that there aren't inherent problems as these systems grow larger; let's understand the trade-offs and engineer appropriately.

PlanetMySQL Voting: Vote UP / Vote DOWN

MySQL Performance Schema is in the trunk

Marc Alff — Mon, 08 Mar 2010 15:41:27 +0000

It is in

As of 2010-03-06, the performance schema is merged into MySQL version 5.5.3-m3, in mysql-trunk.

The documentation for 5.5 has also been updated, and contains a new MySQL Performance Schema chapter.

Disclaimer

Please note that MySQL 5.5 is not a GA product. The performance schema feature may still change at any time, for any reason, and without notice. Customers should not make purchasing decisions based on the availability of the performance schema.

Marc Alff, Oracle.

PlanetMySQL Voting: Vote UP / Vote DOWN

NoSQL doesn’t mean non-relational

Baron Schwartz (xaprb) — Mon, 08 Mar 2010 12:13:45 +0000

It seems that a lot of people equate non-SQL databases with non-relational-ness, or malign the word relational. This is pretty much pure ignorance. If you’ve ever uttered a sentence that includes the phrase “…non-relational database…” then I have two suggestions for you.

Study relational algebra. At a bare minimum, read the Wikipedia article on relational algebra. There is much more you could do — take a class on the topic, or read C.J. Date’s SQL and Relational Theory (my review). Ask yourself how similar SQL is to the relational algebra. How is relational algebra different from SELECT and GROUP BY? Is relational theory about relationships between data? What part do transactions play in relational algebra? Is MySQL a relational database? What about PostgreSQL, Oracle, or DB2?
Now that you understand relational theory more, choose a database that you think is non-relational and write a formal proof that it is not relationally complete. Please do post a link to the proof in the comments.

The truth is, a non-relational database would be of very little use. In layman’s terms, it would mean you have some data that represents true statements, and a piece of software designed to answer questions using those facts, and you can’t answer simple first-order logic questions with the software. How is this an improvement? How is this useful?

A review of SQL and Relational Theory by C. J. Date SQL and Re
InnoDB is a NoSQL database As long as
On the unhelpfulness of NoSQL My favorit

Related posts brought to you by Yet Another Related Posts Plugin.

PlanetMySQL Voting: Vote UP / Vote DOWN

The history of MySQL AB

Dries Buytaert — Mon, 08 Mar 2010 12:00:54 +0000

MySQL, the open source database product that puts the "M" in LAMP, was created by MySQL AB, a company founded in 1995 in Sweden. In 2008, MySQL AB announced that it had agreed to be acquired by Sun Microsystems for approximately $1 billion.

The story of MySQL AB is pretty amazing, so I unleashed my "inner academic", did some research and compiled a timeline of MySQL AB's history. This timeline is assembled based on different resources online, such as MySQL press releases (example 1) and interviews with MySQL AB executives (example 2, example 3), etc.

Things to add? Let me know in the comments and I'll update the post.

1995

MySQL AB founded by Michael Widenius (Monty), David Axmark and Allan Larsson in Sweden.

2000

MySQL goes Open Source and releases software under the terms of the GPL. Revenues dropped 80% as a result, and it took a year to make up for it.

2001

Mårten Mickos elected CEO at age 38. Mårten was the CEO of a number of Nordic companies before joining MySQL, and comes with a sales and marketing background.
2 million active installations.
Raised series A with undisclosed amount from Scandinavian venture capitalists. Estimated to be around $1 to $2 million.

2002

MySQL launched US headquarters in addition to Swedish headquarters.
3 million active users.
Ended the year with $6.5 million in revenue with 1,000 paying customers.

2003

Raised a $19.5 million series B from Benchmark Capital and Index Ventures.
4 million active installations and over 30,000 downloads per day.
Ended the year with $12 million in revenue.

2004

With the main revenue coming from the OEM dual-licensing model, MySQL decides to move more into the enterprise market and to focus more on recurring revenue from end users rather than one-time licensing fees from their OEM partners.
Ended the year with $20 million in revenue.

2005

MySQL launched the MySQL Network modeled after the RedHat Network. The MySQL Network is a subscription service targeted at end users that provides updates, alerts, notifications, and product-level support designed to make it easier for companies to manage hundreds of MySQL servers.
MySQL 5 ships and includes many new features to go after enterprise users (e.g. stored procedures, triggers, views, cursors, distributed transactions, federated storage engines, etc.)
Oracle buys Innobase, the 4-person Finnish company behind MySQL's InnoDB storage backend.
Ended the year with $34 million in revenue based on 3400 customers.

2006

Mårten Mickos confirms that Oracle tried to buy MySQL. Oracle' CEO Larry Ellison commented: "We've spoken to them, in fact we've spoken to almost everyone. Are we interested? It's a tiny company. I think the revenues from MySQL are between $30 million and $40 million. Oracle's revenue next year is $15 billion."
Oracle buys Sleepycat, the company that provides MySQL with the Berkeley DB transactional storage engine.
Mårten Mickos announces that they are making MySQL ready for an IPO in 2008 on an projected $100 million in revenues.
8 million active installations.
MySQL has 320 employees in 25 countries, 70 percent of whom work from home.
Raised a $18 million Series C based on a rumored valuation north of $300 million.
MySQL is estimated to have a 33% market share measured in install base and 0.2% market share measured in revenue (the database market was a $15 billion market in 2006).
Ended the year with $50 million in revenue.

2007

Ended the year with $75 million in revenue.

2008

Sun Microsystems acquired MySQL AB for approximately $1 billion.
Michael Widenius (Monty) and David Axmark, two of MySQL AB's co-founders, begin to criticize Sun publicly and leave Sun shortly after.

2009

Mårten Mickos leaves Sun and becomes entrepreneur-in-residence at Benchmark Capital. Sun has now lost the business and spiritual leaders that turned MySQL into a success.
Sun Microsystems and Oracle announced that they have entered into a definitive agreement under which Oracle will acquire Sun common stock for $9.50 per share in cash. The transaction is valued at approximately $7.4 billion.

PlanetMySQL Voting: Vote UP / Vote DOWN

When indexes are created in internal temporary tables

Venu Anuganti — Mon, 08 Mar 2010 11:40:26 +0000

During my previous post on how to improve derived tables performance, I patched the code to add indexes forcefully on internal derived table results, which made a huge difference in the performance. It was just an experiment and a thought to see if it really works without re-writing the queries, so that the logic can be pushed towards the engine rather than query re-write. \

But I got few emails in my inbox today asking whether MySQL really create any keys on internal temporary tables.

The answer is YES; and MySQL does create two keys on internal temporary tables namely ‘group_key‘ and ‘distinct_key‘ on the following conditions:

If there is any aggregate function and/or group-by (group_key)
Distinct column name(group_key)
Distinct in combination with group-by/aggregation functions (distinct_key)

Provided the query results are yielded in temporary table (Using temporary from the explain), else they get optimized away by the existing indexes from the regular table itself. These keys are added to both memory and disk based (MyISAM) internal temporary tables; so it does not matter if the internal temporary table is in memory or disk.

Here is a simple dump of internal temporary table index stats for some of the basic queries related to Information schema [Warning: these queries are really bad, and can't be used for any production use as they are meant for demonstration of different internal keys ]. This is a patch that I might be using for SHOW TEMPORARY TABLES when internal tables are included in the second version. The first version of the patch is already pushed to Maria branch, hoping that it gets pushed to 5.1.

-----------------------------
 TMP TABLE STATS, SESSION: 1
   temp file  : /tmp/#sqlf90_1_1f
   temp type  : MEMORY
   index count: 1
    key 1-1   : distinct_key
    field     : (null)
    key 1-2   : distinct_key
    field     : ENGINE
 query: select count(distinct engine) from information_schema.tables
-----------------------------
 
-----------------------------
 TMP TABLE STATS, SESSION: 1
   temp file  : /tmp/#sqlf90_1_21
   temp type  : MEMORY
   index count: 1
    key 1-1   : group_key
    field     : TABLE_NAME
 query: select table_name, sum(data_length+index_length) from information_schema.tables 
        where table_schema='mysql' group by 1
-----------------------------
 
-----------------------------
 TMP TABLE STATS, SESSION: 1
   temp file  : /tmp/#sqlf90_1_24
   temp type  : MEMORY
   index count: 1
    key 1-1   : group_key
    field     : TABLE_SCHEMA
    key 1-2   : group_key
    field     : TABLE_NAME
    key 1-3   : group_key
    field     : COLUMN_NAME
 query: select tab.table_schema, tab.table_name, column_name, index_name, seq_in_index 
        from Information_schema.tables tab join information_schema.statistics stast 
        using(table_schema,table_name) group by  1,2,3
-----------------------------
 
-----------------------------
 TMP TABLE STATS, SESSION: 1
   temp file  : /tmp/#sqlf90_1_bd
   temp type  : MEMORY
   index count: 1
    key 1-1   : group_key
    field     : TABLE_NAME
    key 1-2   : group_key
    field     : TABLE_TYPE
    key 1-3   : group_key
    field     : ENGINE
    key 1-4   : group_key
    field     : INDEX_SCHEMA
    key 1-5   : group_key
    field     : INDEX_NAME
 query: select tab.table_schema,  tab.table_name, table_type, engine, index_schema, 
        index_name from information_schema.tables tab join information_schema.statistics
        stats using(table_schema, table_name) where table_schema='mysql' group by 
        1,2,3,4,5, 6 order by 4,3,2,1
-----------------------------
 
-----------------------------
 TMP TABLE STATS, SESSION: 1
   temp file  : /tmp/#sqlf90_1_e0
   temp type  : MEMORY
   index count: 1
    key 1-1   : group_key
    field     : TABLE_NAME
 query: select  table_name, sum(data_length+index_length) from information_schema.tables
        where table_schema='mysql' group by 1
-----------------------------

PlanetMySQL Voting: Vote UP / Vote DOWN

Netapp Data ONTAP fail - maxfiles

Trent Hornibrook — Mon, 08 Mar 2010 08:52:20 +0000

Netapp, enterprise network attached storage devices with OS, Data ONTAP has a per volume specific variable called 'maxfiles'. Basically the maximum number of inodes the volume can consume independent of disk utilization.

Unfortunately this variable must be set per volume, it cannot be 'unlimited' and it cannot be downsized.

According to the man page:

DESCRIPTION
maxfiles increases the number of files that a volume can
hold, as close as possible to max. File inodes are stored
in blocks, and the filer may round the requested max num-
ber of files to the nearest block.

Once increased, the value of max can never be lowered, so
the new value must be larger than the current value.

Further increasing this value to be an unlimited-like variable consumes filer RAM and will result in less usable filer RAM after a Data ONTAP OS upgrade.

Moral of the story - also monitor inode usage as well as disk usage on Netapp volumes!

PlanetMySQL Voting: Vote UP / Vote DOWN

my SHOW INNODB STATUS walkthrough

Trent Hornibrook — Mon, 08 Mar 2010 07:34:07 +0000

I am very fortunate to be sent to a Percona innodb low level conference - with one of the guys who has written the High Performance MySQL book. One of the key items will be to dive deep into the Innodb kernel and find out what the hell the thing is doing. To that end I'm going post what I know about the 'show innodb status' output - now my aim from this conference is to pick up on some of the areas that I'm green in to identify and resolve more MySQL performance problems.

So here is my 'show innodb status' walkthrough:

mysql> show innodb status\G

*************************** 1. row ***************************

Type: InnoDB

Name:

Status:

=====================================

100308 17:05:14 INNODB MONITOR OUTPUT

=====================================

Per second averages calculated from the last 14 seconds

The following information is based on stats gathered in the last 14 seconds - Generally a good 30 seconds should pass before the output can be considered an accurate average. There are some stats that are counters since server start however.

----------

SEMAPHORES

----------

OS WAIT ARRAY INFO: reservation count 74964888, signal count 66577404

Mutex spin waits 0, rounds 3978408534, OS waits 51856599

RW-shared spins 16155677, OS waits 7409510; RW-excl spins 19774799, OS waits 3870167

Since server stat, Innodb has had to reserve 74964888 'slots' and innodb has signaled 66577404 threads to proceed - OS waits are very expensive relative to spin waits.

If there is a high concurrency performance issue, there would also be a line under 'OS WAIT' regarding a transaction(s) waiting on a semaphore. Also if there is a transaction, the output also references a c source file - and the name can usually be extracted to find the location of the bottlekneck.

Also, 19774799 times Innodb has used spin waits, and 3870167 Innodb has had to resort to OS waits - OS waits are more expensive.

------------------------

LATEST FOREIGN KEY ERROR

------------------------

100303 12:03:03 Transaction:

TRANSACTION 2 1356465249, ACTIVE 0 sec, process no 16670, OS thread id 1166653776 inserting, thread declared inside InnoDB 500

mysql tables in use 1, locked 1

3 lock struct(s), heap size 368, 1 row lock(s), undo log entries 1

MySQL thread id 6773466308, query id 23928955894 172.24.0.2 readwrite update

insert into `AlertCriteria` (`AlertID`, `Field`, `Value`)

values ('7695501', 'tb', 'WESTMEAD,WATTLE GROVE,HOLSWORTHY,PARRAMATTA,PENDLE HILL,QUAKERS HILL')

Foreign key constraint fails for table `AlertSystem`.`AlertCriteria`:

CONSTRAINT `AlertCriteria_ibfk_1` FOREIGN KEY (`AlertID`) REFERENCES `Alerts` (`AlertID`) ON DELETE CASCADE

Trying to add in child table, in index `AlertID_index` tuple:

DATA TUPLE: 2 fields;

0: len 4; hex 80756c8d; asc ul ;; 1: len 4; hex 04650c45; asc e E;;

But in parent table `AlertSystem`.`Alerts`, in index `PRIMARY`,

the closest match we can find is record:

PHYSICAL RECORD: n_fields 13; compact format; info bits 0

0: len 4; hex 80756c92; asc ul ;; 1: len 6; hex 000250d61ec0; asc P ;; 2: len 7; hex 800000288e0110; asc ( ;; 3: len 5; hex 72656e2331; asc ren#1;; 4: len 23; hex 4e4557414c4552544e414d452d50616c6d204265616368; asc NEWALERTNAME-Palm Beach;; 5: len 4; hex 803497c6; asc 4 ;; 6: len 1; hex 81; asc ;; 7: len 4; hex 80000000; asc ;; 8: len 4; hex 80000000; asc ;; 9: len 6; hex 5765656b6c79; asc Weekly;; 10: len 1; hex 81; asc ;; 11: len 4; hex 80000000; asc ;; 12: len 4; hex 80000000; asc ;;

This output only appears if there has been a foreign key constraint issue since server restart.

The above shows that there was a foreign key constraint that was not met and Innodb aborted the transaction - the bellow junk is the actual parts of the Innodb code where it discovered the foreign key miss match and aborted. (Nothing to be scared about).

---- Last detected deadlock---

The last detected deadlock would appear if there was a deadlock since last server restart- unfortunately this is not the case therefore its not in the output of 'show innodb status'.

-----

------------

TRANSACTIONS

------------

Trx id counter 2 1594638995

Purge done for trx's n:o <>

History list length 928

LIST OF TRANSACTIONS FOR EACH SESSION:

---TRANSACTION 0 0, not started, process no 16670, OS thread id 1206856016

mysql tables in use 1, locked 1

MySQL thread id 6976314925, query id 24543783827 172.24.0.2 readwrite init

UPDATE Subscribers SET EmailAddress = '12618060932336258813', RealEstProps = '106370836 104481172 105731349 105958174 105958282 105969863 105974146 105998765 106010720 106078593 106102921 106150346 106322759 106337015 106340214 106312885 106343690 106159839 106207162 106290970', LastAccess = 1268028314 WHERE EmailAddress = '12618060932336258813'

---TRANSACTION 2 1594638991, not started, process no 16670, OS thread id 1331722576

MySQL thread id 6976315115, query id 24543783821 172.24.0.2 readwrite

---TRANSACTION 2 1594638994, not started, process no 16670, OS thread id 1366600016

MySQL thread id 6976315285, query id 24543783824 172.24.0.2 readwrite

---TRANSACTION 2 1594638982, not started, process no 16670, OS thread id 1161062736

MySQL thread id 6976315282, query id 24543783797 172.24.0.2 readonly

---TRANSACTION 2 1594638980, not started, process no 16670, OS thread id 1406802256

MySQL thread id 6976315278, query id 24543783787 172.24.0.2 readonly

---TRANSACTION 2 1594638978, not started, process no 16670, OS thread id 1079593296

MySQL thread id 6976315275, query id 24543783783 172.24.0.2 readonly

---TRANSACTION 2 1594638983, not started, process no 16670, OS thread id 1378847056

MySQL thread id 6976315277, query id 24543783782 172.24.21.19 readonly

---TRANSACTION 2 1594638977, not started, process no 16670, OS thread id 1226025296

MySQL thread id 6976314965, query id 24543783781 172.24.0.2 readwrite

---TRANSACTION 2 1594638989, not started, process no 16670, OS thread id 1394821456

MySQL thread id 6976315274, query id 24543783819 172.24.0.2 readonly

---TRANSACTION 2 1594638971, not started, process no 16670, OS thread id 1227622736

MySQL thread id 6976315097, query id 24543783748 172.24.0.2 readwrite

---TRANSACTION 2 1594638974, not started, process no 16670, OS thread id 1186355536

MySQL thread id 6976315255, query id 24543783755 172.24.0.2 readwrite

---TRANSACTION 2 1594638970, not started, process no 16670, OS thread id 1250785616

MySQL thread id 6976315261, query id 24543783747 172.24.0.2 readonly

---TRANSACTION 2 1594638967, not started, process no 16670, OS thread id 1384438096

MySQL thread id 6976315262, query id 24543783740 172.24.0.2 readonly

---TRANSACTION 2 1594638990, not started, process no 16670, OS thread id 1105054032

MySQL thread id 6976315258, query id 24543783820 172.24.0.2 readonly

---TRANSACTION 2 1594638962, not started, process no 16670, OS thread id 1326664016

MySQL thread id 6976315256, query id 24543783723 172.24.0.2 readonly

---TRANSACTION 2 1594638959, not started, process no 16670, OS thread id 1189816656

MySQL thread id 6976315251, query id 24543783709 172.24.0.2 readonly

---TRANSACTION 2 1594638955, not started, process no 16670, OS thread id 1337313616

MySQL thread id 6976314577, query id 24543783701 172.24.0.2 readwrite

---TRANSACTION 2 1594638954, not started, process no 16670, OS thread id 1386568016

MySQL thread id 6976315246, query id 24543783698 172.24.0.2 readwrite

---TRANSACTION 2 1594638952, not started, process no 16670, OS thread id 1349560656

MySQL thread id 6976315241, query id 24543783690 172.24.21.19 readonly

---TRANSACTION 2 1594638949, not started, process no 16670, OS thread id 1331456336

MySQL thread id 6976314979, query id 24543783684 172.24.0.2 readwrite

---TRANSACTION 2 1594638944, not started, process no 16670, OS thread id 1263831376

MySQL thread id 6976314994, query id 24543783675 172.24.0.2 readwrite

---TRANSACTION 2 1594638942, not started, process no 16670, OS thread id 1252649296

MySQL thread id 6976315228, query id 24543783669 172.24.0.2 readonly

---TRANSACTION 2 1594638940, not started, process no 16670, OS thread id 1183959376

MySQL thread id 6976315232, query id 24543783666 172.24.0.2 readonly

---TRANSACTION 2 1594638938, not started, process no 16670, OS thread id 1289124176

MySQL thread id 6976315103, query id 24543783663 172.24.0.2 readwrite

---TRANSACTION 2 1594638948, not started, process no 16670, OS thread id 1376184656

MySQL thread id 6976315205, query id 24543783772 172.24.0.2 readwrite

---TRANSACTION 2 1594638933, not started, process no 16670, OS thread id 1324534096

MySQL thread id 6976315224, query id 24543783641 172.24.0.2 readonly

---TRANSACTION 2 1594638931, not started, process no 16670, OS thread id 1272617296

MySQL thread id 6976315222, query id 24543783634 172.24.0.2 readonly

---TRANSACTION 2 1594638930, not started, process no 16670, OS thread id 1201531216

MySQL thread id 6976315221, query id 24543783630 172.24.0.2 readonly

---TRANSACTION 2 1594638929, not started, process no 16670, OS thread id 1355950416

MySQL thread id 6976315219, query id 24543783624 172.24.0.2 readonly

---TRANSACTION 2 1594638953, not started, process no 16670, OS thread id 1192212816

MySQL thread id 6976315217, query id 24543783694 172.24.0.2 readonly

---TRANSACTION 2 1594638919, not started, process no 16670, OS thread id 1225492816

MySQL thread id 6976315037, query id 24543783602 172.24.0.2 readwrite

---TRANSACTION 2 1594638917, not started, process no 16670, OS thread id 1391360336

MySQL thread id 6976315215, query id 24543783600 172.24.21.19 readonly

---TRANSACTION 2 1594638916, not started, process no 16670, OS thread id 1415588176

MySQL thread id 6976315213, query id 24543783599 172.24.0.2 readonly

---TRANSACTION 2 1594638913, not started, process no 16670, OS thread id 1157601616

MySQL thread id 6976315191, query id 24543783798 172.24.0.2 readwrite

---TRANSACTION 2 1594638905, not started, process no 16670, OS thread id 1314949456

MySQL thread id 6976315202, query id 24543783823 172.24.0.2 readonly

---TRANSACTION 2 1594638903, not started, process no 16670, OS thread id 1237207376

MySQL thread id 6976315200, query id 24543783555 172.24.0.2 readwrite

---TRANSACTION 2 1594638901, not started, process no 16670, OS thread id 1242532176

MySQL thread id 6976315195, query id 24543783543 172.24.0.2 readonly

---TRANSACTION 2 1594638896, not started, process no 16670, OS thread id 1268091216

MySQL thread id 6976315189, query id 24543783526 172.24.0.2 readonly

---TRANSACTION 2 1594638892, not started, process no 16670, OS thread id 1234811216

MySQL thread id 6976314982, query id 24543783521 172.24.0.2 readwrite

---TRANSACTION 2 1594638888, not started, process no 16670, OS thread id 1265428816

MySQL thread id 6976315186, query id 24543783692 172.24.0.2 readonly

---TRANSACTION 2 1594638899, not started, process no 16670, OS thread id 1369262416

MySQL thread id 6976315182, query id 24543783538 172.24.21.19 readonly

---TRANSACTION 2 1594638860, not started, process no 16670, OS thread id 1367132496

MySQL thread id 6976315178, query id 24543783465 172.24.0.2 readonly

---TRANSACTION 2 1594638850, not started, process no 16670, OS thread id 1398016336

MySQL thread id 6976315172, query id 24543783443 172.24.0.2 readonly

---TRANSACTION 2 1594638843, not started, process no 16670, OS thread id 1384171856

MySQL thread id 6976315133, query id 24543783760 172.24.0.2 readwrite

---TRANSACTION 2 1594638907, not started, process no 16670, OS thread id 1319475536

MySQL thread id 6976315152, query id 24543783567 172.24.0.2 readonly

---TRANSACTION 2 1594638802, not started, process no 16670, OS thread id 1310157136

MySQL thread id 6976315119, query id 24543783303 172.24.0.2 readonly

---TRANSACTION 2 1594638795, not started, process no 16670, OS thread id 1275013456

MySQL thread id 6976315110, query id 24543783296 172.24.0.2 readonly

---TRANSACTION 2 1594638951, not started, process no 16670, OS thread id 1288857936

MySQL thread id 6976315118, query id 24543783686 172.24.0.2 readonly

---TRANSACTION 2 1594638897, not started, process no 16670, OS thread id 1075738960

MySQL thread id 6976315049, query id 24543783528 172.24.0.2 readwrite

---TRANSACTION 2 1594638771, not started, process no 16670, OS thread id 1234544976

MySQL thread id 6976315095, query id 24543783365 172.24.0.2 readwrite

---TRANSACTION 2 1594638731, not started, process no 16670, OS thread id 1345300816

MySQL thread id 6976315077, query id 24543783797 172.24.0.2 readonly

---TRANSACTION 2 1594638923, not started, process no 16670, OS thread id 1406269776

MySQL thread id 6976315021, query id 24543783608 172.24.0.2 readwrite

---TRANSACTION 2 1594638975, not started, process no 16670, OS thread id 1348229456

MySQL thread id 6976315074, query id 24543783772 172.24.0.2 readonly

---TRANSACTION 2 1594638687, not started, process no 16670, OS thread id 1244129616

MySQL thread id 6976315053, query id 24543783681 172.24.0.2 readonly

---TRANSACTION 2 1594638780, not started, process no 16670, OS thread id 1395087696

MySQL thread id 6976315033, query id 24543783276 172.24.0.2 readonly

---TRANSACTION 2 1594638782, not started, process no 16670, OS thread id 1202329936

MySQL thread id 6976315030, query id 24543783278 172.24.21.19 readonly

---TRANSACTION 2 1594638645, not started, process no 16670, OS thread id 1203394896

MySQL thread id 6976315028, query id 24543782985 172.24.0.2 readonly

---TRANSACTION 2 1594638634, not started, process no 16670, OS thread id 1310689616

MySQL thread id 6976315022, query id 24543782969 172.24.21.19 readonly

---TRANSACTION 2 1594638633, not started, process no 16670, OS thread id 1190082896

MySQL thread id 6976315023, query id 24543783737 172.24.0.2 readonly

---TRANSACTION 2 1594638787, not started, process no 16670, OS thread id 1385503056

MySQL thread id 6976315011, query id 24543783285 172.24.0.2 readonly

---TRANSACTION 2 1594638602, not started, process no 16670, OS thread id 1409997136

MySQL thread id 6976315007, query id 24543782912 172.24.21.19 readonly

---TRANSACTION 2 1594638579, not started, process no 16670, OS thread id 1346365776

MySQL thread id 6976314184, query id 24543782960 172.24.0.2 readwrite

---TRANSACTION 2 1594638935, not started, process no 16670, OS thread id 1269156176

MySQL thread id 6976314961, query id 24543783645 172.24.0.2 readonly

---TRANSACTION 2 1594638458, not started, process no 16670, OS thread id 1429698896

MySQL thread id 6976314942, query id 24543783593 172.24.0.2 readonly

---TRANSACTION 2 1594638926, not started, process no 16670, OS thread id 1081129296

MySQL thread id 6976314918, query id 24543783618 172.24.0.2 readonly

---TRANSACTION 2 1594638427, not started, process no 16670, OS thread id 1402276176

MySQL thread id 6976314858, query id 24543782714 172.24.0.2 readwrite

---TRANSACTION 2 1594638324, not started, process no 16670, OS thread id 1219103056

MySQL thread id 6976314865, query id 24543782531 172.24.0.2 readonly

---TRANSACTION 2 1594638304, not started, process no 16670, OS thread id 1217505616

MySQL thread id 6976314851, query id 24543782334 172.24.0.2 readonly

---TRANSACTION 2 1594638242, not started, process no 16670, OS thread id 1204992336

MySQL thread id 6976314781, query id 24543782210 172.24.0.2 readwrite

---TRANSACTION 2 1594638104, not started, process no 16670, OS thread id 1208985936

MySQL thread id 6976314732, query id 24543782126 172.24.0.2 readonly

---TRANSACTION 2 1594638084, not started, process no 16670, OS thread id 1372457296

MySQL thread id 6976314714, query id 24543781873 172.24.0.2 readonly

---TRANSACTION 2 1594638083, not started, process no 16670, OS thread id 1227090256

MySQL thread id 6976314689, query id 24543783323 172.24.21.19 readwrite

---TRANSACTION 2 1594638152, not started, process no 16670, OS thread id 1285130576

MySQL thread id 6976314673, query id 24543782470 172.24.21.19 readonly

---TRANSACTION 2 1594638133, not started, process no 16670, OS thread id 1236408656

MySQL thread id 6976314670, query id 24543781963 172.24.0.2 readonly

---TRANSACTION 2 1594638689, not started, process no 16670, OS thread id 1378314576

MySQL thread id 6976314661, query id 24543783582 172.24.0.2 readonly

---TRANSACTION 2 1594637919, not started, process no 16670, OS thread id 1294981456

MySQL thread id 6976314582, query id 24543781464 172.24.0.2 readonly

---TRANSACTION 2 1594638993, not started, process no 16670, OS thread id 1230285136

MySQL thread id 6976314565, query id 24543783822 172.24.0.2 readonly

---TRANSACTION 2 1594637889, not started, process no 16670, OS thread id 1176504656

MySQL thread id 6976314558, query id 24543781388 172.24.0.2 readonly

---TRANSACTION 2 1594637695, not started, process no 16670, OS thread id 1188219216

MySQL thread id 6976314435, query id 24543783810 172.24.0.2 readonly

---TRANSACTION 2 1594636477, not started, process no 16670, OS thread id 1379645776

MySQL thread id 6976313272, query id 24543778354 172.24.0.2 readwrite

---TRANSACTION 2 1594638804, not started, process no 16670, OS thread id 1332521296

MySQL thread id 6976313552, query id 24543783311 172.24.0.2 readonly

---TRANSACTION 2 1594636327, not started, process no 16670, OS thread id 1392159056

MySQL thread id 6976313460, query id 24543777763 172.24.0.2 readonly

---TRANSACTION 2 1594638848, not started, process no 16670, OS thread id 1358080336

MySQL thread id 6976313203, query id 24543783542 172.24.21.19 readonly

---TRANSACTION 2 1594635033, not started, process no 16670, OS thread id 1082997072

MySQL thread id 6976312502, query id 24543774736 172.24.0.2 readwrite

---TRANSACTION 2 1594634847, not started, process no 16670, OS thread id 1302436176

MySQL thread id 6976312376, query id 24543780952 172.24.0.2 readwrite

---TRANSACTION 2 1594634012, not started, process no 16670, OS thread id 1280604496

MySQL thread id 6976311989, query id 24543772726 172.24.0.2 readonly

---TRANSACTION 2 1594633833, not started, process no 16670, OS thread id 1390827856

MySQL thread id 6976311765, query id 24543772353 172.24.0.2 readonly

---TRANSACTION 2 1594638984, not started, process no 16670, OS thread id 1180498256

MySQL thread id 6976311498, query id 24543783824 172.24.0.2 readonly

---TRANSACTION 2 1594638855, not started, process no 16670, OS thread id 1312819536

MySQL thread id 6976311467, query id 24543783563 172.24.0.2 readonly

---TRANSACTION 2 1594633289, not started, process no 16670, OS thread id 1084356944

MySQL thread id 6976311315, query id 24543783517 172.24.0.2 readonly

---TRANSACTION 2 1594633524, not started, process no 16670, OS thread id 1077754192

MySQL thread id 6976311161, query id 24543783604 172.24.0.2 readonly

---TRANSACTION 2 1594632775, not started, process no 16670, OS thread id 1387366736

MySQL thread id 6976310798, query id 24543783824 172.24.0.2 readonly

---TRANSACTION 2 1594638274, not started, process no 16670, OS thread id 1336781136

MySQL thread id 6976308416, query id 24543783827 172.24.0.2 readwrite

---TRANSACTION 2 1594637914, not started, process no 16670, OS thread id 1319209296

MySQL thread id 6976305431, query id 24543781455 172.24.0.2 readonly

---TRANSACTION 2 1594637639, not started, process no 16670, OS thread id 1360476496

MySQL thread id 6976286463, query id 24543780813 172.24.0.2 readonly

---TRANSACTION 2 1594601131, not started, process no 16670, OS thread id 1376450896

MySQL thread id 6976275163, query id 24543671866 172.24.0.2 readonly

---TRANSACTION 0 0, not started, process no 16670, OS thread id 1161861456

MySQL thread id 6976021353, query id 24543783828 localhost root

show innodb status

---TRANSACTION 2 1594617250, not started, process no 16670, OS thread id 1284864336

MySQL thread id 6976219191, query id 24543722876 172.24.0.2 readonly

---TRANSACTION 2 1594503382, not started, process no 16670, OS thread id 1229486416

MySQL thread id 6976183067, query id 24543401940 172.24.0.2 readwrite

---TRANSACTION 2 1594615159, not started, process no 16670, OS thread id 1166653776

MySQL thread id 6976103244, query id 24543716049 172.24.0.2 readonly

---TRANSACTION 2 1594261924, not started, process no 16670, OS thread id 1313085776

MySQL thread id 6975951203, query id 24542727637 172.24.21.19 readonly

---TRANSACTION 2 1593727564, not started, process no 16670, OS thread id 1166387536

MySQL thread id 6975473457, query id 24541302242 172.24.21.19 readonly

---TRANSACTION 2 1593159738, not started, process no 16670, OS thread id 1284331856

MySQL thread id 6975003450, query id 24539858051 172.24.21.19 readonly

---TRANSACTION 2 1594638985, not started, process no 16670, OS thread id 1169848656

MySQL thread id 6974686686, query id 24543783806 172.24.0.2 readonly

---TRANSACTION 2 1592767036, not started, process no 16670, OS thread id 1176770896

MySQL thread id 6974686679, query id 24543783798 172.24.0.2 readwrite

---TRANSACTION 2 1592133017, not started, process no 16670, OS thread id 1225759056

MySQL thread id 6974128272, query id 24537203401 172.24.21.19 readonly

---TRANSACTION 0 0, not started, process no 16670, OS thread id 1615534416

MySQL thread id 5962897775, query id 24543752819 localhost mysqlmonitor

---TRANSACTION 2 1594456162, not started, process no 16670, OS thread id 1185290576

MySQL thread id 4250603963, query id 24543259479 Has read all relay log; waiting for the slave I/O thread to update it

The above details some of the ongoing transaction - similar to 'show processlist' "Trx id counter 2 1594638995" Is a counter that increments upon transactions since server start.

"Purge done for trx's n:o <>

"History list length 928" - Again, I'm not entirely sure that this variable means.

---TRANSACTION 0 0, not started, process no 16670, OS thread id 1206856016

mysql tables in use 1, locked 1

MySQL thread id 6976314925, query id 24543783827 172.24.0.2 readwrite init

Information like the above is similar to the output of 'show processlist' at that exact point in time. As I understand it, similar to 'show processlist', if you see many transactions here you have a concurrency performance issue.

--------

FILE I/O

--------

I/O thread 0 state: waiting for i/o request (insert buffer thread)

I/O thread 1 state: waiting for i/o request (log thread)

I/O thread 2 state: waiting for i/o request (read thread)

I/O thread 3 state: waiting for i/o request (write thread)

Pending normal aio reads: 0, aio writes: 0,

ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0

Pending flushes (fsync) log: 0; buffer pool: 0

35818914 OS file reads, 248305163 OS file writes, 47414036 OS fsyncs

4.57 reads/s, 27392 avg bytes/read, 16.57 writes/s, 2.14 fsyncs/s

This shows how Innodb is going reading and writing to disk. There are four threads by default - the buffer pool thread, the log thread, reading and writing. What we see is no pending 'aios' (async io) - which means the server is not waiting on disk. The second last line shows the number of reads, writes and filesystem syncs that have been performed since sever stat - this database server (readb01.kp) is mixed and does slightly more reads than writes.

The last line shows the averages per second since the interval listed at the top of the 'show innodb status' output. So clearly there were some write transactions and not mean read transactions going on.

Again, theres no pending aio's so the server is not IO bound.

-------------------------------------

INSERT BUFFER AND ADAPTIVE HASH INDEX

-------------------------------------

Ibuf: size 1, free list len 131, seg size 133,

4000946 inserts, 4000946 merged recs, 601283 merges

Hash table size 26690981, node heap has 69134 buffer(s)

678.95 hash searches/s, 2176.34 non-hash searches/s

Now I'm a little green on the output of the above. The insert buffer is related to the insert buffer thread and the adaptive hash index in the algorithm Innodb uses to insert data into the tablespace.

---

LOG

---

Log sequence number 327 3878392573

Log flushed up to 327 3878369439

Last checkpoint at 327 3640927886

0 pending log writes, 0 pending chkp writes

36089932 log i/o's done, 1.36 log i/o's/second

The above information pertains to the circle logs in Innodb - This shows how much data has not been flushed to disk that still remains in the innodb circle logs. (3878392573-3878369439) 23134 bytes. The innodb log files are set to around 160Mb to reduce the amount of IO at the cost of start up time upon crash.

----------------------

BUFFER POOL AND MEMORY

----------------------

Total memory allocated 15308894706; in additional pool allocated 45677312

Dictionary memory allocated 658920

Buffer pool size 823168

Free buffers 0

Database pages 754034

Modified db pages 29518

Pending reads 0

Pending writes: LRU 0, flush list 0, single page 0

Pages read 44983219, created 3689633, written 250224524

7.64 reads/s, 0.07 creates/s, 19.64 writes/s

Buffer pool hit rate 1000 / 1000

The total memory is the total memory Innodb has allocated - 15Gb - the additional buffer pool is set to ~400Mb (for data dictionary etc).

Dictionary memory allocated is much less than the additional buffer pool size suggesting that the additional buffer pool memory size can be dropped.

The buffer pool is used to cache the tablespace, store locks and most things innodb so it needs to be set quiet large. What I take into account is the last line which is a ratio output of the usefulness of the innodb buffer pool - the hit rate is currently 1000/1000 or 100% - in other words all transactions, be them read or write can go via the buffer pool instead of missing the buffer pool and going directly to disk.

--------------

ROW OPERATIONS

--------------

1 queries inside InnoDB, 0 queries in queue

1 read views open inside InnoDB

Main thread process no. 16670, id 1156270416, state: sleeping

Number of rows inserted 200754864, updated 1377590681, deleted 77291280, read 722941029741

4.50 inserts/s, 116.85 updates/s, 0.00 deletes/s, 96297.19 reads/s

----------------------------

END OF INNODB MONITOR OUTPUT

============================

1 row in set, 1 warning (0.00 sec)

The '0 queries in the queue' shows that there are currently no transactions in the Innodb kernel waiting to execute. The remaining lines show the type of work Innodb is doing.

PlanetMySQL Voting: Vote UP / Vote DOWN

Tech Coverage, Twitter and the Top 8 Business Pubs

Mark Hinkle — Mon, 08 Mar 2010 06:10:07 +0000

I work for a start-up and like the folks at virtually every start-up we dream of being covered by one of the Big 8 business publications: The Wall Street Journal, The New York Times, Forbes, Fortune, BusinessWeek, The Economist, the Financial Times, and USA Today.

The folks at ITDatabase took a closer look at how they cover the tech industry and surveyed a group of veterans from major tech PR agencies to see what their experiences were like. The resulting report yielded a few interesting tidbits:

The New York Times’ David Pogue (@Pogue) and Kara Swisher (@karaswisher) top the most followed tech journalists on Twitter while NY Times’ Motoko Rich and Bit blogger Jenna Wortham get retweeted much more often (500 and 331 times per post on average respectively).
According to the tech PR veterans surveyed, 46.4% thought that that business press tech coverage is worse than it was five years ago. Only 25% thought it was better.
Business success is not an indicator of press coverage see how Twitter and Facebook rule the total number of clippings along with Google, Apple and Microsoft.

Intriguing research if you are interested in tech PR.

PlanetMySQL Voting: Vote UP / Vote DOWN