Internet is full of simple shell backup scripts. Every DBA and system administrators (including myself) has written one. Bash is a bad tool for this job and in this post I will explain why. Let’s find out why the need to write anything arises at first place. There is a lot of great backup tools […]

The post Simple shell script is bad idea appeared first on TwinDB.

Our website twindb.com is built on WordPress software and has always been. A while ago we decided the website needs a better look. But not only that. Managing our old website was quite laborious and manual process which goes strongly against our culture at TwinDB to do things right. Few weeks ago we migrated to […]

The post How we manage WordPress website with Terraform, Chef and GitHub appeared first on TwinDB.

pt-table-checksum perfectly solves problem of checking if master and its slaves are in-sync. It answers the question “Are the slaves consistent with the master?”. However if the answer is “No” pt-table-checksum doesn’t actually tell what exactly is different. [crayon-59f9aff300dfa915077909/] pt-table-sync may give a partial answer. It can print SQL statements to sync the replication cluster. Reading the SQL […]

The post Show differences found by pt-table-checksum appeared first on TwinDB.

Last week I wrote about aDNS discovery feature in Etcd. As a step in the whole process we need to create DNS records in twindb.com zone. CloudFlare provides rich API to work with it. We wrapped it into a Python module twindb_cloudflare and opensourced it:

• Source code: https://github.com/twindb/twindb_cloudflare
• Documentation: https://twindb-cloudflare.readthedocs.io
• PyPi package: https://pypi.python.org/pypi/twindb_cloudflare

In the post I will show how to use twindb_cloudflare module.

CloudFlare API …

Setting up an etcd cluster with DNS discovery may be challenging. There are several building blocks:

• Etcd – a distributed key value store
• Amazon EC2 – cloud computing provider
• Cloudflare – DNS provider
• Chef – for configuring individual nodes

Each of them has their pitfalls, we will guide you through whole process.

DNS discovery

Any clustered system needs a way to maintain a list of nodes in a cluster. Usually you need to specify all cluster members when starting a node. This is the way zookeeper and consul works. Effectively you have redundancy in configuration – the list of nodes is stored on every node. The list must be consistent and it’s difficult to maintain it especially if the cluster …

I use Pingdom's free service to monitor slaptijack.com. Apparently, late Friday night, oom-killer decided that the server needed more memory and took out the MySQL server. To make matters worse, I missed the alarm from Pingdom, and slaptijack.com was down for pretty much all of Saturday. The fact that oom-killer was invoked is annoying, but more on that later.

The beauty of using Pelican rather than Wordpress (or any other database-driven content engine) is one less point of failure for the site. Obviously, I don't have all of slaptijack.com converted to Pelican yet (and perhaps never will), but at least parts of the site were up and working despite MySQL being down. If nothing else, this incident is enough to convince me that moving to Pelican was a good idea.

OOM'ed

…

Not so long ago I had a customer who experienced data loss after MySQL restart. It was really puzzling. MySQL was up & running for many months, but after the customer restarted MySQL server all tables have gone. The tables were still visible in SHOW TABLES output, but they were not readable:

mysql> show tables like 'actor';
+--------------------------+
| Tables_in_sakila (actor) |
+--------------------------+
| actor                    |
+--------------------------+
1 row in set (0.00 sec)

mysql> select * from actor;
ERROR 1146 (42S02): Table 'sakila.actor' doesn't exist
mysql>

To understand what’s happened let make some experiments (WARNING: Don’t do it on production or with valuable data).

Let’s take a healthy MySQL instance with installed sakila database.

While MySQL …

Naturally, I ran into this problem in the middle of the night when the client was asleep and I had no access to the Magento administrative interface. The short story is that after running a re-index of the site, the home page began returning a 404 erro...

Encryption is important component of secure environments. While being intangible property security doesn’t get enough attention when it comes to describing various systems. “Encryption support” is often the most of details what you can get asking how secure the system is. Other important details are often omitted, but the devil in details as we know. In this post I will describe how we secure backup copies in TwinDB.

See the picture. This is what happens when encryption is used incorrectly. The encryption algorithm can be perfect, but poor choice of the mode results in a quite readable encrypted image. This mode is called “Electronic Code Book”, avoid it at all means.

Another bright example of improper encryption use was illustrated in Venona project.

Have you ever tried to install Xtrabackup on Amazon EC2 instance with Oracle’s MySQL 5.6? Dependencies hell strikes when you ask pretty common and reasonable thing – run the GA version of MySQL and backup it with the most popular open-source tool – XtraBackup. From this post you will learn how to resolve the conflicts and make everybody happy.

mysql55-libs conflicts with mysql-community-libs-5.6.22

A fresh Amazon Linux AMI, 2014.09 EC2 instance comes with MySQL 5.5.40 in amzn-updates repository. Today MySQL 5.5 turns five years old. It’s a good and stable version. But many people want to run MySQL 5.6, because it’s better than 5.5, it supports full-text indexes and Oracle ends support of 5.5 this year.

Oracle distributes MySQL releases via YUM repository. Installing MySQL from the YUM repository is a good idea because YUM takes care of …