Some time ago I've noted that one of the tools I use for testing various MySQL and MariaDB cases and to reproduce potential bugs, MySQL-Sandbox, is not updated any more. It turned out that active development switched to its port in Go called dbdeployer. You can find detailed information about dbdeployer and reasons behind developing it provided by its author, Giuseppe Maxia, here and there. See also this post at Percona blog for some quick review of its main features. One of the points of …[Read more]
9 Older Entries »
Version 0.3.0 of dbdeployer has gained the ability of deploying
multiple sandboxes concurrently. Whenever we deploy a group of
sandboxes (replication, multiple) we can use
--concurrent flag, telling dbdeployer that it
should run operations concurrently.
What happens when a single sandbox gets deployed? There are six sets of operations:
- Create the sandbox directory and write down its scripts;
- Run the initialisation script;
- Start the database server;
- Run the pre-grants SQL commands (if any;)
- Load the grants;
- Run the post-grants SQL commands (if any;)
When several …[Read more]
How it happened
A few years ago I started thinking about refactoring MySQL-Sandbox. I got lots of ideas and a name for the project (dbdeployer) but went no further. The initial idea (this was 2013!) was to rewrite the project in Ruby: I had been using Ruby at work and it looked like a decent replacement for Perl. My main problem was the difficulty of installation in an uncontrolled environment. If you have control over your environment (it's your laptop or you are in charge of the server configuration via Puppet or similar) then the task is easy. But if you ever need to deploy somewhere with little or no notice, it becomes a problem: there are servers where Perl is not installed, and is common that the server also have a policy forbidding all scripting languages from being deployed. …[Read more]
Uber’s mission is transportation as reliable as running water, everywhere, for everyone. Last time, we talked about the foundation that powers Uber Engineering. Now, we’ll explore the parts of the stack that face riders and drivers, starting …
The post The Uber Engineering Tech Stack, Part II: The Edge and Beyond appeared first on Uber Engineering Blog.
I was on the quest of searching the Holy Grail of Go programming, and I found something, which I doubt that it is, but close enough – for the first sight.
I have several problems with GO, first, that I write my code on an OSX box, and I’ll run the programs on Linux hosts, so I have to solve the cross compilation; my second problem with Go, that I don’t really like the “There is a GO project folder, and all the GO projects are relying on” approach. It makes using GitHub painful.
The first problem of mine is easily achievable since GO 1.5: we only need a GOOS environment variable and we can compile to different OS-es (see more at Dave Cheney: http://dave.cheney.net/2015/08/22/cross-compilation-with-go-1-5) easily.
The second problem is easily solvable too, just we have to start using the GOPATH variable for every GO project we have.
I don’t really want to use any external dependencies, so I decided to …[Read more]
A few years ago, I wrote the original OmniSQL as a Perl script to automate the execution of queries across many MySQL servers and, in some cases, provide basic aggregation functionality. Shortly after that time, Justin Swanhart released Shard-Query and Domas Mituzas released pmysql, so OmniSQL was retired.
I’m a daily user of pmysql; indeed, we run a heavily patched version. Lately, I find myself needing to extend the functionality even more than we’ve already done. After a bit of evaluation, I decided that C wasn’t the best language to move forward with the new features. Thus, OmniSQL is reborn in Go. While initial testing indicates that it is an order of magnitude slower than pmysql running against 1,000+ instances of MySQL, it’s still well within my tolerance (and there are many areas of …[Read more]
With MySQL, it is relatively easy to create “point in time” restores. All you need is recent(ish) backup and a bunch of saved binary logs. You can restore the backup you have, and when it is completed, you can use mysqlbinlog utility to apply your saved binary logs to the desired state of your database.
I have created a simple go application to make your life easier. You can find it on my GitHub page.
The app works as reading its config file for the MySQL server connection credentials, a local directory where the binary logs will be kept, and the will path of the mysqlbinlog utility.
- It checks the binlogs on the remote server which able to streamed
- Checks the local directory where the binlogs are kept, to check which logs are already there
- The incomplete (file size differs local and on remote server) files will be …
With MySQL it is relatively easy to create point in time restores. All you need is a recent(ish) backup, and a bunch of saved binary logs. You can restore the backup you have, and when it is completed, you can use mysqlbinlog utility to apply your saved binary logs to the desired state of your database. I […]
I am fascinated. Maybe that should be enough, but I guess I have to write a bit more here because we are not on twitter.
I spent a few days to get know Go language, and now I am more than satisfied. I mean, all the project ideas which are floating in my head should be written in Go.
First of all, I have rewritten Mambo-collector to go (https://github.com/banyek/mambo) because I have faced some serious errors when I used it on CentOS 7 – I blame systemd -: If the process was running as root then after a few days of data collecting, killing that process was lead to restart the entire system, which is not a bug, it is a catastrophe. I tried to debug it several ways, but I am not sure where the problem is, it could be at the ‘loghandler’ redirection or any other place in python-daemon, or it is simply there is a buffer inside which overflows – I don’t really care, because mambo was just a proof of concept – what I used in …[Read more]
I’m happy to announce Go packages for MySQL. Particularly exciting is a new query fingerprint function which is very fast and efficient, but I’ll talk about that later. First, go-mysql is currently three simple Go packages for parsing and aggregating MySQL slow logs. If you’ve been following Percona development, you’ve no doubt heard of Percona Cloud Tools (PCT), a somewhat new performance management web service for MySQL.
One tool in PCT is “Query Analytics” which continuously analyzes query metrics from the slow log. The slow log provides the most metrics and therefore the most performance insight into MySQL. percona-agent, the open-source agent for PCT, uses …[Read more]
9 Older Entries »