Do you have MyISAM tables you reload with new data?

Do your queries, using that table, get blocked because the table is locked?

Do the waiting queries create idle connections slowing down the table load?

Do you wish you could just replace the table?

Years ago I was told you can replace CSV tables by simply replacing the CSV file. I figured this would also be true of a MyISAM file and it is. I use this perl script to replace MyISAM tables forcast and current observation weather data. The processing and tables are created on another computer. Weather forecasting is CPU and database expensive. I then copy (rsync) the files to the production system and run this script.

#!/usr/bin/perl
################################################################################
################################################################################
# mg_hot_replace_table.pl - Hot Replace a MySQL table.
#
# 2010-05-01 …

I went to MongoSF today - quite an event, and I hope to have a chance to write more about it. This post is about one replication problem and how MongoDB solves it.

If you're using MySQL Replication when your master goes down it is possible for some writes to be executed on the master, but not on the slave, which gets promoted to the master. When Master comes back up it has some updates done to it which cause it to be inconsistent with data on the new Master. In MySQL world we can chose to either ignore this problem (or may be even replay those changes on slaves and hope it works out), re-clone it from the slave or use mk-table-checksum to find inconsistencies and re-sync them with mk-table-sync. Both of these operations can be very expensive for large databases.

MongoDB approach used …

I work in all markets of the database industry, from web & startup through the largest and most established enterprises.  And to be completely honest, the name Ingres has not come up in conversation very much at all.  10 years ago maybe more often, but recently not all that much.  But Ingres has been quietly ticking away.  Despite being largely off the radar, they still have a sizable and loyal customer base, global offices and a focused & dedicated management team.  And importantly they have an open source business model which actually appears to be working.

I wrote last year that their "behind the scenes" status had the potential to change.  Ingres had been very clever and worked out a partnership relationship with Peter Bonzc’s Vectorwise.  And …

For the Friday after the MySQL conference, Oracle had invited all storage engines to the traditional storage engine summit, but this was then canceled (or postponed) in the last minute. Since the engine vendors had already booked the day anyway, we agreed to sponsor the facility so the meeting could take place. In addition to those who had planned to be there, the meeting was also attended by Mikael Ronström, Jonas Oreland and Sanja Byelkin who had their flights cancelled. (Oracle was already represented by Konstantin Osipov.)

Also see http://askmonty.org/wiki/Storage_Engine_Summit_2010 for more complete notes of the summit.

read more

Welcome to Log Buffer. This week’s issue #187 was another group effort. Thanks to all our contributors – you rock!

Suggested by Pythian’s Bradd Piontek, is a post he really liked because he used to write pipelined functions for Dynamic Search queries, – Tom Kyte’s something new I learned about estimated cardinalities. He’s also highlighted something new Tom learned about sqlplus. And the fact that Richard Foote announced the …

I recently discovered that there’s a youtube video of the talk I gave at OpenSQL Camp in Portland in 2009.

This is a whiteboard presentation and is less well developed than the talk I gave a the MySQL conference (I posted those slides two days ago. But since it includes audio it may be easier to understand.

This talk presents the data structure underlying the TokuDB storage engine for MySQL.

There will not be a May meeting of the North Texas MySQL Users Group but I am scrambling to get us a great location for the June meeting. I have left my Job with MySQL/Sun/Oracle and am no longer responsible for MySQL Certifications. That means my access to the Sun office were we have been meeting is ended also.

I am now the Customer Service Manager for Calpont and the InfiniDB storage engine. InfiniDB is column based, multi-threaded, and the community edition is open source. If you run intensive reports against InnoDB or MyIsam databases, you need to investigate this product. And if you regularly run massive queries in data warehousing or business analytic operation that you need to see how the massively parallel process architecture of the enterprise product can make life easier.

Let me premise this post with the statement I think the MySQL documentation is an excellent and highly accurate resource. I think the MySQL docs team do a great job, however like software and people, documentation is not perfect.

As members of the MySQL community you can always contribute to improve the process by reading the documentation and logging any issues as Documentation Bugs.

Some time ago in a discussion with a friend and colleague, we were talking about changes in historical defaults that had been improved finally in MySQL 5.4 The specific discussion was on the new default innodb_buffer_pool_size and we both agreed it increased significantly. One said 1GB, the other said 128MB. Who was right? Well we both were, and we were both inaccurate depending on versions.

Referencing the 5.4 Manual in …

Let's take quick look at the performance of the new InfiniDB Subquery processing available with the 1.1.1 Alpha.  The arrow was added to be sure our timings weren't confused with the axis.

This was against a relatively small dataset, the Star Schema Benchmark with 6 million rows in the fact table.  A base query was run where the outer query...

This post is Part Two of my State of the Internet Operating System. If you haven't read Part One, you should do so before reading this piece.

As I wrote last month, it is becoming increasingly clear that the internet is becoming not just a platform, but an operating system, an operating system that manages access by devices such as personal computers, phones, and other personal electronics to cloud subsystems ranging from computation, storage, and communications to location, identity, social graph, search, and payment. The question is whether a single company will put together a single, vertically-integrated platform that is sufficiently compelling to developers to enable the kind of lock-in we saw during the personal computer era, or whether, Internet-style, we will instead see services from …