Maatkit does more than just MySQL. I’ve just committed a new version of mk-query-digest, a powerful log analysis tool, with support for Posgtres logs, in both syslog and stderr format. I’m hoping that people will give this a spin in the real world. I have lots of test cases, but that’s never enough; I’m looking for people to crunch their logs and let me know if anything breaks.
A brief tutorial:
# Get it $ wget http://www.maatkit.org/trunk/mk-query-digest # Run it $ perl mk-query-digest --type pglog /path/to/log/file # Learn about it (search for the string "pglog") $ perldoc mk-query-digest
I’m going to close comments on this blog post so I don’t get bug reports in the comments. If you have feedback, please[Read more...]
In the Autumn, I wrote about some experiments I did using KVM and virtual machines to build and test MariaDB binary packages on a number of different platforms. In the period since then I added some polish and refinements, and the system is now running well for some time. We build and test packages for Debian (4 and 5), Ubuntu (8.04 to 10.04), Centos 5, and generic Linux; amd64 and i386 architectures.
To better control the startup and shutdown of the virtual machines, I created
a small wrapper script around KVM called
runvm. This wrapper
encapsulates the steps needed to boot up a virtual machine, run a series of
commands inside it, and shut it down gracefully afterwards. Some special care
is taken in the script to ensure that
I've just publicly released another Postgres-related script, this one called "boxinfo". Basically, it gathers information about a box (server), hence the catchy and original name. It outputs the information it finds into an HTML page, or into a MediaWiki formatted page.
The goal of boxinfo is to have a simple, single script that quickly gathers important information about a server into a web page, so that you can get a quick overview of what is installed on the server and how things are configured. It's also useful as a reference page when you are trying to remember which server was it that had Bucardo version 4.5.0 installed and was running pgbouncer.
As we use MediaWiki internally here at End Point (running with a Postgres backend, naturally), the original (and default) format is HTML with some MediaWiki specific items inside of it.
Because it is meant to[Read more...]
I’m really proud to announce the release of the version 1.0 of mysql-snmp.
mysql-snmp is a mix between the excellent MySQL Cacti Templates and a Net-SNMP agent. The idea is that combining the power of the MySQL Cacti Templates and any SNMP based monitoring would unleash a powerful mysql monitoring system. Of course this project favorite monitoring system is OpenNMS.
mysql-snmp is shipped with the necessary OpenNMS configuration files, but any other SNMP monitoring software can be used (provided[Read more...]
mysqldump --no-data sakila | dump_filter --delete >
This is a little story of a little bug. This gremlin suddenly appeared in a CGI.PM web-based application I work with. To make a long story short, an email was coming out something like this . . .
Subject:Update to Report #389939 by B. bloggins Description:389939 #389939: TPDD Now Deploying to monitoring for the MySQL servers.
. . . when it should have been some thing like this:
Subject: TPDD Update to Report #389939 by B. bloggins
TPDD Now Deploying to monitoring for the MySQL servers.
After about an hour tracking things back, my team and I narrowed it down to this line of code:
We scratched our respective[Read more...]
mk-loadavg tool is a helpful way to gather information about infrequent conditions on your database server (or any other server, really). We wrote it at Percona to help with those repeated cases of things like “every two weeks, my database stops processing queries for 30 seconds, but it’s not locked up and during this time there is nothing happening.” That’s pretty much impossible to catch in action, and these conditions can take months to resolve without the aid of good tools.
In this blog post I’ll illustrate a very simple usage of
mk-loadavg to help in solving a much smaller problem: find out what is happening on the database server during periods of CPU spikes that happen every
A couple of months back, Tim Procter, Sheeri Cabral and I were discussing about how best to diagnose a MySQL server and/or tune its performance, automating the process as much as possible. The Performance Advisors from MySQL Enterprise (http://www.mysql.com/products/enterprise/) do this, but most of our customers don’t have a subscription and Pythian’s collective experience is not necessary reflected by its rules.
In our daily work, we have used Major Heyden’s MySQL Tuner, Mark Leith’s Statpack and our own tools to review a MySQL server configuration parameters. However, all of these tools had limitations in regards of what we wanted to achieve. Our major[Read more...]
This blog post is about things I did on my own free time, not endorsed by my employer.
This blog post is about things I did on my own free time, not endorsed by my employer.
The call for papers for OSDC 2009 is open until 30 June 2009; yes that’s only a few more days. Submit your abstract and do a talk at this fab conference!
This is a grassroots style conference designed by developers for developers. It covers Perl, Python, Ruby/Rails, PHP, Java/Grails and Open Source operating systems as well as some business aspects. If you’d like to cover something else as well that is Open Source themed, please feel free.
The Call for Papers can be found at: http://2009.osdc.com.au/call-for-papers
The important dates are:
StackOverflow is an amazing site for coding questions. It was created by Joel Spolsky of joelonsoftware.com, Jeff Atwood of codinghorror.com, and some other incredibly smart guys who truly care about user experience. I have been a total fan of SO since it went mainstream and it's now a borderline addiction (you can see my StackOverflow badge on the right sidebar).
Update 6/21/09: This server is currently under very heavy load (10-200), even with caching plugins enabled. Please bear with me as I try to[Read more...]
After a long hiatus, I am happy to announce that mylvmbackup version 0.12 has now been released. This release includes a large number of improvements, minor code cleanups, as well as some new functionality. In particular, I would like to thank Matthew Boehm, Tim Stoop, Baron Schwartz, Ville Skyttä and Ronald Bradford for their contributions.
Some notable highlights from the ChangeLog:
install_driver(mysql) failed: Can't load '/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/DBD/mysql/mysql.so' for module DBD::mysql: /usr/lib64/libmysqlclient.so.15: version `libmysqlclient_15' not found (required by /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/DBD/mysql/mysql.so) at /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/DynaLoader.pm line 230.
at (eval line 3
After a lot of mucking around I noticed that the MySQL-shared-community RPM was installed, not the MySQL-shared-compat RPM like my usual installs. So…
rpm -Uvh MySQL-shared-compat-5.1.33-0.rhel5.x86_64.rpm
A few months ago I installed ActivePerl 5.10 on a Windows XP Pro workstation. Next I tried to install DBD::mysql using CPAN, it failed. When I browsed modules via ppm, a GUI in this version of ActiveState Perl on Windows, DBD::mysql was not listed as an available module.
I then downloaded the source code, manually modified the MAKE file, fiddled with Visual Studio NMAKE, compiled it a few times, without success.
Google revealed the existence of Strawberry Perl. So I removed ActiveState Perl, installed Strawberry, ran ppm install DBI, ppm install DBD-mysql. And it worked.
Fast-forward a few weeks, I started playing with EPIC, a plugin for Eclipse that supposedly provides a nice IDE for Perl development and debugging. As a result, I started fiddling with PadWalker, a prerequisites for EPIC. I couldn’t[Read more...]
Maatkit version 3519 is ready for download. There are a lot of changes in this release, many of which are incompatible with previous releases. There are also a lot of important new features. Read on for the details.
First, thanks to everyone who contributed to this month’s release. A lot of people have jumped into Maatkit and started committing code. I attribute this to deliberately forcing a more open policy with decisions being made on the mailing list, rather than the former policy of “Percona pays for development, so they have more say than you do” — a snobby and ill-advised way to treat an open-source project. If you are interested in contributing to Maatkit, please ask. Subversion commit rights are being handed out[Read more...]
“Groovy, baby!”, as international man of mystery likes to say.
It seems converting character and text data to utf8 character set is a common task for MySQL, especially during an upgrade. In fact, I had trouble with it during server and WordPress database upgrade for this blog site. I wrote about it in this post, where I explained how to do it step-by-step using a command line tool such as mysql, taking advantage of some nifty code generation trick with information_schema.
One drawback of that method is that it’s a manual process, therefore time-consuming and error-prone.
I spent some time today to cook up a simple Perl script to automate this task. See below. Remember to change MyDatabase to your database name (there are 3 places that[Read more...]