I wasn’t prepared for this. After spending months playing with GDB development trees I somehow entirely missed that 7.0 release is getting close, and took me more than an hour to spot it.

My favorite features are python scripting and non-stop debugging. I was toying around with python scripting for a while, and was planning to make backtraces make sense. Having hands that open means that one can see PHP backtraces, when gdb’ing apache, see table names and states when MySQL thread access handler interfaces, or remote IPs and users, when it is writing to network. Process inspection can simply rock, if right tools are created using these new capabilities, and I’m way too excited when I think about those. “Always have debugging symbols” gets way more meaning now.

Another issue I’ve been trying to resolve lately is avoiding long locking …

The other day I found a nice surprise on my inbox. Jay Pipes asked me if I'd like to try fixing a small bug on Drizzle. It looked pretty simple, and the bug report included a big part of the fix. I accepted without a doubt.
I decided to first change trans_prealloc_size from uint32_t to uint64_t. That was done on drizzled/session.h. Then, I went to drizzle/set_var.cc and changed sys_trans_prealloc_size from …

A short time ago in a galaxy nearby, Domas Mituzas wrote about contention profiling with GDB stack traces. Mark Callaghan found the technique useful, and contributed an awk script (in the comments) to aggregate stack traces and identify which things are blocking most threads. I’ve used it myself a time or five. But I’ve found myself wanting it to be fancier, for various reasons. So I wrote a little utility that can aggregate and pretty-print backtraces. It can handle unresolved symbols, and aggregate by only the first N lines of the stack trace. Here’s an example of a mysqld instance that’s really, really frozen up:

bt-aggregate -4 samples/backtrace.txt | head -n12
2396 threads with the following stack trace:
        #0  0x00000035e7c0a4b6 in …

When one wants to script automated replication chain building, certain things are quite annoying, like immutable replication configuration variables. For example, at certain moments log_slave_updates is more than needed, and thats what the server says:

mysql> show variables like 'log_slave_updates';
+-------------------+-------+
| Variable_name     | Value |
+-------------------+-------+
| log_slave_updates | OFF   |
+-------------------+-------+
1 row in set (0.00 sec)

mysql> set global log_slave_updates=1;
ERROR 1238 (HY000): Variable 'log_slave_updates' is a read only variable

Of course, there are few options, roll in-house fork (heheeeee!), restart your server, and keep warming up your tens of gigabytes of cache arenas, or wait for MySQL to ship a feature change in next major release. Then there are evil tactics:

mysql> system gdb -p $(pidof mysqld)
                       -ex "set opt_log_slave_updates=1" -batch …

Over last few years 64-bit x86 platform has became ubiquitous, thus making stupid memory limitations a thing of some forgotten past. 64-bit processors made internal memory structures bigger, but compensated that with twice the amount and twice larger registers.

But there’s one thing that definitely got worse – gcc, the compiler, has a change in default compilation options – it omits frame pointers from binaries in x86_64 architecture. People have advocated against that back in 1997 because of very simple reasons, that are still very much existing today too – frame pointers are needed for efficient stack trace calculations, and stack traces are very very useful, sometimes.

So, this change means that oprofile is not able to give hierarchical profiles, it also means that MySQL crash information will not include most useful bit – …

When I was doing data loading tests, I realized that usually low checksum calculation CPU percentage is actually the blocking factor. See, usually when background writers do the flushing, it gets parallelized, but if active query is forcing a checkpoint, it all happens in ‘foreground’ thread, checksum computation included. This is where more Sun-ish wisdom (these people tune kernel with debugger all the time) comes in:

gdb -p $(pidof mysqld) -ex "set srv_use_checksums=0" --batch

Puff. Everything becomes much faster. Of course, one would be able to restart the server with –skip-innodb-checksums, but that would interrupt the whole process, etc. Of course, proper people would implement tunable parameter (5 lines of code, or so), but anyone with Solaris experience knows how to tune stuff with debuggers, hahaha.

Odd though, I …

I’ve been working with a client recently who has SELinux on his servers.  It has been quite a struggle sometimes.

My colleages tell me that SELinux has a pretty noticeable performance impact.  I am not sure if we have benchmarks to support this; at any rate, the client said it’s OK, we’ll take the performance hit.

There [...]

You have a typical rpm installation of mysql, and the process is crashing. Here are the basic steps needed to find out more info about a crash:

• Configure the OS to be able to create corefiles. (Redhat details), (Solaris details)
  
• Tell mysqld to create a corefile by adding the following options to my.cnf:

[mysqld_safe]
core-file-size=unlimited

[mysqld]
core-fileUsually the corefile will be created in the datadir with a name like core.2921 where 2921 was the pid of the running process. The location is configurable on most OS's.

You'll need the following to study the core file:

• exact mysqld binary that created the core file
…

I finally got the Eclipse debugger to play nicely with MySQL!!!

Steps:
make clean
./configure --with-debug (and I modified my flags with -g as well, tho it's not required)

then under Eclipse I did a 'make all'
used --gdb --one-thread when running
be sure to close the 'registers' pane or else MySQL will crash and require a kill -9

I can now step through my code and see it in action,
it really is doing just what I thought it should!
(tho I found the MySQL GUI tools do a lot of things I didn't know about)

I'll be submitting an early review to my mentor tonight.
I'm not too familiar with submitting a diff so I'll probably attach the full source files as well in case I mess it up.
As of tonight I have surpassed 100 hours working on this project!
In a way I wish all this really was for JUST a t-shirt, adding money to the mix is both …