Is there anyone using partitioned archive tables in MySQL 5.1 for storing logs (or other voluminous data)?

Storing large amounts of logs in a relational database can bring special challenges.

Logs can take up huge amounts of space on disk, and while disk space is cheap, disk I/O can be expensive, performance-wise. But many logs compress really well, and for this the MySQL archive storage engine (insert-only, no indexes, gzip'ed storage) can be used.

You often want to scan across a few hours or days worth of logs, and indexes are poor for this purpose as the large number of disk seeks can kill performance. And full table scans of years of logs is not all that much fun either. For this, table partioning (supported in MySQL 5.1) is very useful. Store each day or week of logs in a separate table partition, partitioned on day (maybe use the 5.1 event sceduler, or simply a cron job, to add new partitions automatically). Then …

Just a quick note while it is in my head:

When testing software, especially for performance, remember to test on the same hardware you intend to deploy on. One trend I have seen for some time is that testing is often done on whatever happens to be lying around and not otherwise assigned.

While you can certainly test MySQL or an email server on such hardware, you should at least be aware that by providing hardware that does not match what you use in production you will not get an accurate picture, and sometimes the differences can be dramatic. This is especially true of enterprise software with high demands for memory and storage such as databases and MTAs: running such software on an old desktop or five-year-old server will not provide an accurate picture of how the software will perform on a multicore server with several gigabytes of RAM and a RAID array.

My advice is this: if you know you will be deploying the solution …

A client informed me of TheDailyWTF and I've subsequently read it quite frequently. It often features bad SQL of all dialects. It also features stupid business rules, management blunders, bad authentication, bad validation, race conditions and dodgy error messages. You'll certainly remember what not to do and when the forum comments aren't sarcastic and purile then you'll also gain insight into good practice.

If you're completing the survey then please mention your MySQL experience :-)

Jonathan Zuck has written an incendiary piece on the death of the peaceful compromise between free sourcerors and open sourcerors. The great divider? GPLv3.

I found the article profoundly fascinating, as it was evidence of intelligence gone awry. It completely misses the point that GPLv3 is just a license, and only applies to code to which it is newly licensed.

Will Linux suddenly be consumed by the dreaded v3? Nope. Will MySQL? Nope. JBoss? Nope. And so on.

The GPLv3 will only apply if these project maintainers choose to apply it to their code, and there's not a big line waiting for it.

Linus Torvalds says:

(The GPL 3) no longer works in the "fairness" sense. It's purely a firebrand, and only good for the extremist policies of the FSF. It's no longer a nice balance that a lot of people …

Following question came up recently: What happens if you drop or add a partition of an existing already partitioned table with ALTER TABLE? Will it be copied or will just the single partition be dropped and added? How are the index(es) rebuild after such a drop/add?

I have to disagree with the hard-won wisdom Joseph documents in this blog entry. Altough I 100% support his idea of version control for config files. Both his "Use mapfiles" and "Avoid generating configuration files" are 2 practices that I have to disagree with.

Don't use map files, use actual subdirs and map multiple trees on top of eachother, Create trees such as ALL , HOSTNAME , HOSTNAME-X , etc and layer them on top of eachother for the final filesystem. ALL is written first, then hostname and finally hostname-x, so if your CLUSTERNODE13 has a specific config file /etc/murphy that differs from all other nodes it sits in overrides/CLUSTERNODE13/etc/murphy. In the original sugestion with map files.. how would you map different content based on a server to the same filenames ? You'd end up getting a big bunch of …

This is pretty simple approach I often use called to optimize web application performance if problem happens with few pages. If we have "everything is slow" problem looking at slow query logs may be better start.

So what could you do ?

Look at the information shown on the page which comes from database. This is important as this is information you need to have retrieved from database one way or another if you want to keep page look the same. Your task is simply to have this information delivered from the database most efficient way.

Investigate how dynamic is this information If information rarely changes or is static it may be better to pre-create or cache it. There are many …

Hi Kettle fans,

The 2.3.1 release has been dropped! These kinds of things are always a bit troublesome because of the testing I need to do to get it out, the documentation updates, etc, etc. It’s also the right time for me to do bug administration: clean up old stuff, etc. This is not the most enjoyable type of activity and I’m glad it’s over and done with.

Anyway, it’s about time we did something enjoyable again: a webinar! Yes, we have been invited to do a MySQL Webinar next Tuesday. At 10:00 am Pacific, 1:00 pm Eastern or 17:00 GMT for the people living in my timezone. The presentation will be approximately 45 minutes long followed by Q&A.

I think that this is a great …

SQLyog is now free software and seems to run under WINE on my Ubuntu box okay (i think i’m running stock wine from dapper….)

So those of you addicted to SQLyog (i hear there’s a couple), you can use it on linux okay too (and probably other systems that WINE supports)

By allison

Webyog, based in Bangalore, India, has open sourced SQLyog, their GUI MySQL management tool. They're following MySQL's dual licensing model, split between a Community Edition under the GPL and a proprietary Enterprise Edition. In an interview with one of the founders, they estimate about 10% of code in the Enterprise Edition isn't released in the Community Edition.

In the past, I've heard (mainly from friends and collegues based in India or who visit regularly promoting open source) that India is more a consumer than a producer of open source software. Perhaps that's changing.

I also wonder if …