I've just publicly released another Postgres-related script, this one called "boxinfo". Basically, it gathers information about a box (server), hence the catchy and original name. It outputs the information it finds into an HTML page, or into a MediaWiki formatted page.

The goal of boxinfo is to have a simple, single script that quickly gathers important information about a server into a web page, so that you can get a quick overview of what is installed on the server and how things are configured. It's also useful as a reference page when you are trying to remember which server was it that had Bucardo version 4.5.0 installed and was running pgbouncer.

As we use MediaWiki internally here at End Point (running with a Postgres backend, naturally), the original (and default) format is HTML with some MediaWiki specific items inside of it.

Because it is meant to

  [Read more...]

I’m really proud to announce the release of the version 1.0 of mysql-snmp.

What is mysql-snmp?

mysql-snmp is a mix between the excellent MySQL Cacti Templates and a Net-SNMP agent. The idea is that combining the power of the MySQL Cacti Templates and any SNMP based monitoring would unleash a powerful mysql monitoring system. Of course this project favorite monitoring system is OpenNMS.

mysql-snmp is shipped with the necessary OpenNMS configuration files, but any other SNMP monitoring software can be used (provided

  [Read more...]

Several people have suggested a more flexible approach at mysqldump output in matter of user privileges.
When you dump the data structure for views, triggers, and stored routines, you also dump the permissions related to such objects, with the DEFINER clause.
It would be nice to have such DEFINER clauses removed, or even replaced with the appropriate users in the new database.

The mysqldump filter was created with this need in mind. It allows you to remove all DEFINER clauses and eventually replacing them with a better one.
For example:

mysqldump --no-data sakila | dump_filter --delete >

  [Read more...]

This is a little story of a little bug. This gremlin suddenly appeared in a CGI.PM web-based application I work with. To make a long story short, an email was coming out something like this . . . 

389939
Subject:Update to Report #389939 by B. bloggins Description:389939 #389939: TPDD Now Deploying to monitoring for the MySQL servers.

 . . . when it should have been some thing like this:

Subject: TPDD Update to Report #389939 by B. bloggins

TPDD Now Deploying to monitoring for the MySQL servers.

After about an hour tracking things back, my team and I narrowed it down to this line of code:

$self->send_TXT_email(CGI::param("rep_no"),$rep_object,
                     $subject,$user_ref);

We scratched our respective

  [Read more...]

Maatkit’s mk-loadavg tool is a helpful way to gather information about infrequent conditions on your database server (or any other server, really). We wrote it at Percona to help with those repeated cases of things like “every two weeks, my database stops processing queries for 30 seconds, but it’s not locked up and during this time there is nothing happening.” That’s pretty much impossible to catch in action, and these conditions can take months to resolve without the aid of good tools.

In this blog post I’ll illustrate a very simple usage of mk-loadavg to help in solving a much smaller problem: find out what is happening on the database server during periods of CPU spikes that happen every

  [Read more...]

Why a New Utility?

A couple of months back, Tim Procter, Sheeri Cabral and I were discussing about how best to diagnose a MySQL server and/or tune its performance, automating the process as much as possible. The Performance Advisors from MySQL Enterprise (http://www.mysql.com/products/enterprise/) do this, but most of our customers don’t have a subscription and Pythian’s collective experience is not necessary reflected by its rules.

In our daily work, we have used Major Heyden’s MySQL Tuner, Mark Leith’s Statpack and our own tools to review a MySQL server configuration parameters. However, all of these tools had limitations in regards of what we wanted to achieve. Our major

  [Read more...]

I'm pleased to announce the release of DBD::mysql 4.013. I've been busy with so many interesting tasks at NorthScale which include keeping my open source projects moving along. I made some time this week to conglomerate some patches that have been submitted and close tickets which this release consists of.

Changes from the changelog:

* #49484: PATCH add support for MYSQL_INIT_COMMAND to
DBD::mysql - Thanks Peter John Edwards!
* #48242: 'mysql_bind_type_guessing' doesn't work correctly
with minus sign - Thanks Serguei Trouchelle!
* #45616: t/40blobs.t cannot pass without database access -
ServerError() not declared - Thanks ysth http://ysth.livejournal.com/

Thanks to all who helped with patches on this release!

The files:

file: $CPAN/authors/id/C/CA/CAPTTOFU/DBD-mysql-4.013.tar.gz
















  [Read more...]

Introduction

StackOverflow is an amazing site for coding questions. It was created by Joel Spolsky of joelonsoftware.com, Jeff Atwood of codinghorror.com, and some other incredibly smart guys who truly care about user experience. I have been a total fan of SO since it went mainstream and it's now a borderline addiction (you can see my StackOverflow badge on the right sidebar).

The Story

Update 6/21/09: This server is currently under very heavy load (10-200), even with caching plugins enabled. Please bear with me as I try to

  [Read more...]

Hi all!

I'm pleased to announce a release of DBD::mysql 4.012!

This release includes several fixes and modifications (per ChangeLog):

* Patch to bind_type_guessing from Craigslist, Thanks to Chris! Happiness is no quoted numbers. Fixed ChopBlanks to work with bind_type_guessing

* Patch for win32 strawberry build Thanks to Curtis Jewell! Windows needs love

* Patch for auto-reconnect to set active flag Thanks to Doug Fischer!

* Fixed bug 32464 http://bugs.mysql.com/bug.php?id=32464. See also: https://rt.cpan.org/Ticket/Display.html?id=46308
Add connection flag mysql_no_autocommit_cmd which users of MySQL Proxy can use to prevent 'set autocommit=#' from being issued by the driver. 'perldoc DBD::mysql' to see how to use this












  [Read more...]


install_driver(mysql) failed: Can't load '/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/DBD/mysql/mysql.so' for module DBD::mysql: /usr/lib64/libmysqlclient.so.15: version `libmysqlclient_15' not found (required by /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/DBD/mysql/mysql.so) at /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/DynaLoader.pm line 230.
at (eval line 3


After a lot of mucking around I noticed that the MySQL-shared-community RPM was installed, not the MySQL-shared-compat RPM like my usual installs. So…


rpm -Uvh MySQL-shared-compat-5.1.33-0.rhel5.x86_64.rpm


I just applied the external stored procedure patch to a branch of MariaDB and uploaded it to LaunchPad.You can see the branch at https://code.launchpad.net/~atcurtis/maria/5.1-wl820Note that this is not in any reasonable condition to merge into MariaDB. Hopefully we can engage in dialog as to how we can bring this feature properly to MariaDB, MySQL and Drizzle, hopefully making the plugins

A few months ago I installed ActivePerl 5.10 on a Windows XP Pro workstation. Next I tried to install DBD::mysql using CPAN, it failed. When I browsed modules via ppm, a GUI in this version of ActiveState Perl on Windows, DBD::mysql was not listed as an available module.

I then downloaded the source code, manually modified the MAKE file, fiddled with Visual Studio NMAKE, compiled it a few times, without success.

Google revealed the existence of Strawberry Perl. So I removed ActiveState Perl, installed Strawberry, ran ppm install DBI, ppm install DBD-mysql. And it worked.

Fast-forward a few weeks, I started playing with EPIC, a plugin for Eclipse that supposedly provides a nice IDE for Perl development and debugging. As a result, I started fiddling with PadWalker, a prerequisites for EPIC. I couldn’t

  [Read more...]

Maatkit version 3519 is ready for download. There are a lot of changes in this release, many of which are incompatible with previous releases. There are also a lot of important new features. Read on for the details.

First, thanks to everyone who contributed to this month’s release. A lot of people have jumped into Maatkit and started committing code. I attribute this to deliberately forcing a more open policy with decisions being made on the mailing list, rather than the former policy of “Percona pays for development, so they have more say than you do” — a snobby and ill-advised way to treat an open-source project. If you are interested in contributing to Maatkit, please ask. Subversion commit rights are being handed out

  [Read more...]

Files used in conference presentation here: UC2009 presentationI think todays presentation for Perl Stored Procedures for MySQL was quite successful. The audience was quite engaged and asked questions throughout (yes, I invited questions through the talk). I made sure to mention Eric Herman's Java plugin because even though he didn't get time to submit a talk, the work is noteworthy. Sometimes I

Hi all!

I'm happy to announce the release of DBD::mysql 4.011, uploaded late last night. This release fixes the following (per Changelog):

* Renamed unsafe_bind_type_guessing, fixed some of the logic. This
can be used to deal with bug 43822
(https://rt.cpan.org/Ticket/Display.html?id=43822)
* Patch from Daniel Frett (daniel Dot frett At ccci Dot org -- Thanks!) to fix
issue of binding sever side integer parameters (server-side prepare
statements) resulting in corrupt data, bug 42723
(https://rt.cpan.org/Ticket/Display.html?id=42723)
* Updated documentation, cruft cleanup (as always)

One thing I'd like to fix next is building DBD::mysql on Windows. Anyone who'd like to contribute, I'd be glad to hear from you. I at












  [Read more...]

I have synced the codebase with the 5.1.32 release of MySQL and it appears to work just fine. Sometimes frustrating that Bazaar takes a bizarre amount of time to do a merge.Download link for the source tarball are available from LaunchPad Download.As an experiment, I have built a Mac OS 10.5 installer package (x86 32bit) which I have also placed there. Took a bit of fiddling about to discover how

It is possible to generate SVG graphs directly without requiring access to the Google Chart Servers. Perl provides a wealth of libraries which can create SVG graphs (but most not quite as pretty as the charts which the Google service creates).This could be invaluable for people who want to serve all the information from their own website - for example, SSL encrypted websites - where some 'secured

“Groovy, baby!”, as international man of mystery likes to say.

It seems converting character and text data to utf8 character set is a common task for MySQL, especially during an upgrade. In fact, I had trouble with it during server and WordPress database upgrade for this blog site. I wrote about it in this post, where I explained how to do it step-by-step using a command line tool such as mysql, taking advantage of some nifty code generation trick with information_schema.

One drawback of that method is that it’s a manual process, therefore time-consuming and error-prone.

I spent some time today to cook up a simple Perl script to automate this task. See below. Remember to change MyDatabase to your database name (there are 3 places that

  [Read more...]

In Brooks Johnson's blog posting, Database Science: Are MySQL stored procedures slow?, he mentioned how much slower MySQL's stored procedures are and then compared it with a small piece of .Net code.

Using Perl, a stored procedure which counts to the same value is obviously not going to be as fast as bytecode languages with JIT compilers but it is a lot faster than MySQL's native SQL stored procedures. These perl stored procedures are able to perform dynamic SQL using the familiar DBD::mysql driver without any risk of self-deadlock.

Of course, you can also write stored procedures in Java for many databases but I haven't yet written the neccessary Type 2 JDBC driver to perform a in-thread connection back into the database server to be

  [Read more...]

Actually, what I really wanted to blog about before getting carried away with irony yesterday was an old idea on how to force my developers to use placeholders exclusively for SQL queries in applications. As should be apparent from yesterdays blog entry, I am strongly in favour of using placeholders for interpolating values into SQL queries, due to the great reduction in potential bugs (including, but not limited to, SQL injections).

Basically, wrap the database API so that all database access passes through the wrapper. This can usually be achieved, for example by subclassing DBI (for Perl) and returning such subclasses from the application connection pool, or other similar methods. Probably many large web applications already have such wrappers or use APIs that can be patched or extended appropriately.

Now add code

  [Read more...]

It is sad to see how 9X% (or should that be 99.X%?) of SQL applications are riddled with SQL injection bugs.

There really is no excuse for this. Nobody writes code like this:

sub stupid_sum {
    my ($list) = @_;
    my $string = shift @$list;
    for (@$list) {
      $string .= " + " . $_;
    }
    my $sum = eval($string);
    return $sum;
}

Right? Just because our computers use the Von Neumann architecture, where CPU instructions and data is stored in the same memory, does not mean that we cannot distinguish between code and data (ok, so in TeX we do not, but there is a reason TeX is not pleasant to write applications in).

So when we use

  [Read more...]

Many, if not all, corporations control and monitor employees web surfing and email activities. Sometimes a company may choose to use automatic configuration script to enable employees’ web connection. This can be verified by:

• Internet Explorer: Tools -> Internet Options-> LAN Settings -> “Use automatic configuration script”;
• Firefox 3: Tools -> Options -> Advanced -> Network -> Settings… -> Automatic proxy configuration URL:

Sometimes you need to know a proxy server name or IP address. For instance, if you use Perl and would like to do an automatic binary install of a module, then you will need to define a proxy for cpan to use. This is the command to do that:

set HTTP_proxy=http://my.proxy.server:8000/

So obviously you need to know a proxy server name

  [Read more...]

I'm pleased to announce the release of DBD::drizzle 0.003. In this release, I've made many changes, particularly those to get it working with the stock drizzle client library, which is a bit of a moving target, but that's how it goes with new, quickly-evolving projects.

Caveat: This is an Alpha release.

I've added back the tests from DBD::mysql and modified them to work with Drizzle. Some pass just fine, others report 'out of memory'. I hope to figure this out soon, though this is that last version where I'll use the client library that comes with Drizzle. I'll be changing to using Eric Day's improved client library, libdrizzle, and maybe in the process of changing to that, will squash these bugs.

The files:

  file: $CPAN/authors/id/C/CA/CAPTTOFU/DBD-drizzle-0.003.tar.gz
  size: 89682 bytes
   md5:

  [Read more...]

I spoke with Brian Aker last week about the issues I found with decrement and increment while working on my book "Developing Web Applications with Apache, MySQL, Memcached, and Perl". He suggested I make sure that I was using the latest libmemcached. I checked, and the version of libmemcached that is included with Memcached::libmemcached is 0.21 - this is the version of Memcached::libmemcached I obtained from using CPAN (perhaps CPAN needs a new bundle...). So, I obtained the most recent version of Memcached::libmemcached from the subversion repository at https://perl-libmemcached.googlecode.com/svn, and it includes the lastest libmemcached, version 0.25. To make a long story short, I re-wrote my test script to test each operation individualy and time that, as well as compare the times to Cache::Memcached. The numbers are great and are reason enough  [Read more...]

I'm busy working on my book, "Developing Web Applications with Apache, MySQL, Memcached, and Perl", writing about how Libmemcached (particularly the perl interface to libmemcached, Cache::Memcached::libmemcached/Memcached::libmemcached) is faster. And it is, except when I was writing a test script to compare, I first used Daisuke Maki's script that comes with Cache::Memcached::libmemcached, tools/benchmark.pl (which I modified to only compare Cache::Memcached to Cache::Memcached::libmemcached)

==== Benchmark "Simple get() (scalar)" ====
Rate perl_memcached libmemcached
perl_memcached 6784/s -- -78%
libmemcached 30488/s 349% --
==== Benchmark "Simple get_multi() (scalar)" ====
Rate perl_memcached libmemcached
perl_memcached 1806/s -- -84%
libmemcached








  [Read more...]