NOTE: Any comments in this blog entry is based on my personal thoughts after visiting the Hadoop conference and doesn't represent any current plans within MySQL.

I visited the Hadoop conference today which was a very interesting event. The room was filled to its limit, people were even standing up in lack of chairs. Probably around 300 people or so.

It was interesting to see the wide scope of web-scale problems that could be attacked using Hadoop. The major disruptive feature in Hadoop is the MapReduce solution to solving parallel data analysis problems.

One piece that I started thinking of was how one could introduce the MapReduce into SQL. One presentation of HIVE showed an interesting approach of how to solve this problem. I thought a bit on how one could integrate a MapReduce solution in MySQL and there are certainly a lot of problems to solve but I got a few interesting ideas.

The …

Recently I acquired Sesame Street Volume 1, and on the third DVD in the set I came across one of my favorite Sesame Street songs: “Who are the people in your neighborhood?”

Here’s a sample of one such skit, if you are not familiar with it, or if you want a bit of nostalgia http://youtube.com/watch?v=B9lpUjQvToY (note, play will likely start automatically, so tune your volume appropriately before clicking).

The refrain is “Who are the people in your neighborhood? The people that you meet each day!” I live in a city of 34,000 people just 6 miles northwest of Boston, MA. I know exactly one neighbor, across the street, whom we met because I sent my husband over to get her live band to stop playing loud music at her party at 2 am. I do not know many of the local business owners. I do not know who lives in my neighborhood, yet people live around me. Saying I …

Navicat For MySQL is a GUI for MySQL developers. I've tried a few tools before but somehow got attached to Navicat due to a few nice features that I'm not going to go into right now. Navicat suffers from a couple of annoying bugs and random crashes. I don't know if I can help fix the random ones but if I can at least file the ones I can reproduce, everyone wins. I have the latest as of today version 8.0.23.

Bug [NAL-15328]: Structure Sync Fails to notice encoding differences

Lots of people like PERL, Python, Java or C to write database backfills, data repair, etc. I like writing in all these languages but what I don't like doing is writing the same code over and over again, so I write my DB code in the same language as the environment that I'm using. This means I write backend DB scripts in PHP, I get to reuse common DB paths, classes and functions and improve things when I see them.

But, PHP if not written correctly will use up 2GB of memory easily from loosing reference to arrays or setting globals and forgetting about them; stuff like that. This is especially visible in long running applications.

PEAR is not immune to these memory leaks. So, to get around reference problems in PEAR I do

$skiptrace =& PEAR::getStaticProperty('PEAR_Error', 'skiptrace');

$skiptrace = false;


This prevents this error
PHP Fatal error: Allowed …

Effective April 1, I will join Percona full-time as a consultant. I’ll be helping people build high-performance applications with MySQL, but I’ll also be continuing to develop and improve tools such as Maatkit. This career change has been a long time in progress. I’m really looking forward to it, but at the same time it’s hard to leave my current employer, The Rimm-Kaufman Group (RKG). Working with them has been the best job I’ve ever had.

The details can be found

http://en.oreilly.com/mysql2008/public/schedule/detail/588


I've since moved on from Flickr to a new Job, but Flickr is still allowing me to give this talk. Flickr is so cool!

The talk encompasses capacity planning and scaling for a heavy concurrent write and read environment, and when it makes sense to split resources out to a single application.

There was a proposed project of particular interest to me in the MySQL list for Google's Summer of Code, an Obfuscator. This tool would take a schema/dataset and obfuscate it in such a way that it can be posted on forums or submitted to MySQL in a bug or support request, without divulging any sensitive information.

If you're currently a student and want to work on this project and thus be an active part of the Google Summer of Code 2008, apply quickly (before March 31st) through the GSoC student application form and also join the MySQL SoC mailing list!

The Obfuscator …

Just before the Easter holidays I posted this challenge for a MySQL schema. For lack of submissions to far, I'll leave it open for a little bit longer.

Perhaps you reckon the challenge sucks ;-) In that case please comment and tell why! That'd be good feedback. Otherwise, do take a stab at it. If you did and got stuck, comment about this too. Then others can help and move it forward.

Last week I played with queries from TPC-H benchmarks, particularly comparing MySQL 6.0.4-alpha with 5.1. MySQL 6.0 is interesting here, as there is a lot of new changes in optimizer, which should affect execution plan of TPC-H queries. In reality only two queries (from 22) have significantly better execution time (about them in next posts), but I want to write about is queries that execute slower in new MySQL 6.0 version.

Query is pretty simple

PLAIN TEXT SQL:

1. SELECT sum(l_extendedprice * l_discount) AS revenue
2. FROM lineitem WHERE l_shipdate>= date '1995-01-01'
3. AND l_shipdate <date '1995-01-01' + interval '1' year
4. AND l_discount BETWEEN 0.09 - 0.01 AND 0.09 + 0.01
5. AND l_quantity <24;

with execution plain (in 5.1)

PLAIN TEXT SQL:

1. …

We made a very significant announcement last week, of a collaboration with one of the most (if not the most) security sensitive institutions on earth, the United States government's National Security Agency. They've joined the burgeoning OpenSolaris community, to collaborate with Sun and other community members on the future of ultra-secure operating systems.

To put this in context, community engagement has always been one of the most important ways Sun innovates in the marketplace - we partner with those that have extreme demands (whether it's the world's largest supercomputing facility, or the world's most paranoid security professionals (no offense intended), or the world's largest archival storage facilities), and then we leverage that expertise to create products for the mass market. We let extreme customers teach …