Backups are an integral and very important part of any system. Backups allow you to recover your data and be up and running again, in the advent of problems such as system crashes, hardware failures or users deleting data by mistake. I had been evaluating backup solution for a while but to be honest I really wasn't satisfied with the solutions available until I came across XtraBackup and I am loving it since. In this post I intend on showing how to do backups and restores using XtraBackup.

Scenario Master – Master replication

MasterA is a client facing server
MasterB is a warm standby server (read only)

MasterB restarted abruptly and when instances were braught back up MasterA (it’s slave) was showing the following error:

MasterA has the following error in show slave status:
Last_IO_Errno: 1236
Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: ‘Could not find first log file name in binary log index file’

Solution:

Slave: stop slave;

Master: flush logs
Master: show master status; — take note of the master log file and master log position

Slave: CHANGE MASTER TO MASTER_LOG_FILE=’log-bin.00000X′, MASTER_LOG_POS=106;
Slave: start slave;

If you have used MySQL for some time you know that mysqld can write binlogs. This is usually used for backup purposes and JITR or for replication purposes so a slave can collect the changes made on the master and apply them locally.

Most of the time apart from configuring how long you keep these binlogs they are pretty much ignored.

Recently I came across an issue.  I have a slave server which is NOT configured read only and which has an additional database used to collect statistics from the replicated database and provided aggregation and business information. The typical sales per country, per product, per day, week, month, year, per whatever etc.  This is the usual datawarehouse type functionality.  It’s done on a slave and not the master so that the load is not passed on down stream to all boxes. …

Thanks to a suggestion from Mohammad Lahlouh, all deleted comments are restored. Thanks!

I was able to do this with the mysql binlog, which is a binary log containing all statements executed on the server. If you face a similar problem, here are the steps to restore the DB:

1. See if you're actually storing binlogs. For me this happened to be the case, although I never actually payed attention (My friend Jens Meijer is so kind to provide hosting for me). In my case they were stored in /var/log/mysql.
2. Grab your latest backup. In my case it was from October 2008.
3. Make sure you also create a new backup, just in case this process goes wrong.
4. Restore your old back.
5. Find the binlog that corresponds to the date of your backup. I did a simple copy of the ~300 binlogs to a different directory.
6. Run the binlog through 'mysqlbinlog' and pipe into a new file. I had to append the …

Recently with a client while running a number of disaster recovery tests I came across a nasty situation which was not part of the original plan and provided a far worse disaster situation then expected.

I should preface this condition with some of the environment conditions.

• MySQL 5.0 Enterprise 5.0.54
• RHEL 5 64bit
• Master and 2 Slaves
• MySQL Data and MySQL Binary Logs/MySQL Error Logs are on separate disk partitions

While running stress tests under high load, we tested the filling of partition containing the logs. This partition included the binary log and MySQL error log.

The observed output was.

• An error message was written to the MySQL error log. See below.
• Application throughput dropped, but did not stop.
• Binary logs stopped occuring.
• MySQL proactively stopped logging but continued to process …

A binlog storage engine was talked about many times already. Being able to JOIN against the binlogs, the just SELECT it, search for entries ... is not possible with current SQL commands in the MySQL Server.

As you have read the previous articles here, I wrote a binlog encoder/decoder for MySQL Proxy and wrapped with with a Lua interface. Some years ago I also wrote a scriptable storage engine which can call lua scripts to return data for SELECTs. Putting the two together gives us: a Binlog Storage Engine.

In the datadir we just create

$ ls -la test/binlog_*.*
-rw-r--r--  1 jan  staff  1342  4 Mai 12:30 test/binlog_000016.lua

mysql> select * from test.binlog_000016;
Query OK, 0 rows affected (0.00 sec) …

I just pushed the code for my replication changes on launchpad:

$ bzr branch lp:~jan-kneschke/mysql-proxy/replication

The presentation should be available ... soon.

One of the first examples is about filtering binlogs. If you want to remove all ALTER statements from the replication stream (or in this case the binlog files) you can just iterate the binlog and copy everything to a new binlog, but the ALTER statement

local binlog = require("mysql.binlog")
local tokenizer = require("proxy.tokenizer")

local f_in = assert(binlog.open("/tmp/binlog-test.log"))
local f_out = assert(binlog.open("/tmp/binlog-test-filtered.log", "w"))
print("filtering /tmp/binlog-test.log to /tmp/binlog-test-filtered.log")

for event in f_in:next() do
    if event.type == "QUERY_EVENT" then
            local tokens = tokenizer.tokenize(event.query.query)
            local first_token = tokenizer.first_stmt_token(tokens) …

Warning... what follows is a murky mess.
It's filed as MySQL bug#43398 (verified!) but it's triggered by cPanel doing evil.

Scenario....Start a mysql server, as normal
Then run mysql_install_db (as root, like you would when you first install MySQL)
See a new binlog file get created, with ownership/group root!Of course you generally wouldn't run mysql_install_db while a server is running, but there's nothing to prevent you (or something else) from doing so!
--bootstrap just shouldn't initialise binlog, then there wouldn't be a issue.

cPanel runs mysql_install_db in its automatic upgrade scripts (dangerous already, automatically upgrading MySQL Server on a system!), it's run every night on cPanel systems even if no upgrade is done, and it behaves exactly as described above. It then chowns the binlog files to mysql:mysql which is of course …

“There are 10 types of people in the world — those who understand binary, and those who don’t.”

The term “binary” in MySQL has many different meanings. How many can you come up with? I have 6, but I am willing to believe there are more!

0) “Binary distribution” is the name for the package that contains a binary. Another use is “binary installation” but that’s pretty much the same usage pattern as “binary distribution”, so I won’t count “binary installation” as a separate usage.
1) “Server binary” or “client binary” is the actual program (mysqld, mysql).
2) “Binary format” is a compressed format. For example, DECIMAL is stored in a “binary format” — each group of nine digits is compressed into 4 bytes.
3) “Binary log” is the change log. You can argue that this is an extension of #3, because the binary log is a compressed …

Protobuf comes with a minor problem: it does not have support for handling "type tagged structures", that is, something reminiscent of objects in OOP lingo, so if one is going to have a heterogeneous sequences of messages, you have to roll it yourself. For that reason, I added a transport frame for the messages in the binary log that wraps each with some extra information. In addition to allowing the binary log to be a sequence of messages, it also adds some integrity-checking data and simplifies some administrative tasks.

The format of each message in the sequences is given in the table in the margin. where the length is a specially encoded …