Showing entries 1 to 10 of 18
8 Older Entries »
Displaying posts with tag: go (reset)
Concurrent sandbox deployment


Version 0.3.0 of dbdeployer has gained the ability of deploying multiple sandboxes concurrently. Whenever we deploy a group of sandboxes (replication, multiple) we can use the --concurrent flag, telling dbdeployer that it should run operations concurrently.

What happens when a single sandbox gets deployed? There are six sets of operations:

  1. Create the sandbox directory and write down its scripts;
  2. Run the initialisation script;
  3. Start the database server;
  4. Run the pre-grants SQL commands (if any;)
  5. Load the grants;
  6. Run the post-grants SQL commands (if any;)

When several …

[Read more]
Meet dbdeployer: the new sandbox maker


How it happened

A few years ago I started thinking about refactoring MySQL-Sandbox. I got lots of ideas and a name for the project (dbdeployer) but went no further. The initial idea (this was 2013!) was to rewrite the project in Ruby: I had been using Ruby at work and it looked like a decent replacement for Perl. My main problem was the difficulty of installation in an uncontrolled environment. If you have control over your environment (it's your laptop or you are in charge of the server configuration via Puppet or similar) then the task is easy. But if you ever need to deploy somewhere with little or no notice, it becomes a problem: there are servers where Perl is not installed, and is common that the server also have a policy forbidding all scripting languages from being deployed. …

[Read more]
The Uber Engineering Tech Stack, Part II: The Edge and Beyond

By Lucie Lozinski

Uber Engineering

Uber’s mission is transportation as reliable as running water, everywhere, for everyone. Last time, we talked about the foundation that powers Uber Engineering. Now, we’ll explore the parts of the stack that face riders …

The post The Uber Engineering Tech Stack, Part II: The Edge and Beyond appeared first on Uber Engineering Blog.

The way I like to compile my Go programs – Makefile

I was on the quest of searching the Holy Grail of Go programming, and I found something, which I doubt that it is, but close enough – for the first sight.

I have several problems with GO, first, that I write my code on an OSX box, and I’ll run the programs on Linux hosts, so I have to solve the cross compilation; my second problem with Go, that I don’t really like the “There is a GO project folder, and all the GO projects are relying on” approach. It makes using GitHub painful.

The first problem of mine is easily achievable since GO 1.5: we only need a GOOS environment variable and we can compile to different OS-es (see more at Dave Cheney: http://dave.cheney.net/2015/08/22/cross-compilation-with-go-1-5) easily.

The second problem is easily solvable too, just we have to start using the GOPATH variable for every GO project we have.

I don’t really want to use any external dependencies, so I decided to …

[Read more]
OmniSQL – Massively Parallel Query Execution

A few years ago, I wrote the original OmniSQL as a Perl script to automate the execution of queries across many MySQL servers and, in some cases, provide basic aggregation functionality. Shortly after that time, Justin Swanhart released Shard-Query and Domas Mituzas released pmysql, so OmniSQL was retired.

I’m a daily user of pmysql; indeed, we run a heavily patched version. Lately, I find myself needing to extend the functionality even more than we’ve already done. After a bit of evaluation, I decided that C wasn’t the best language to move forward with the new features. Thus, OmniSQL is reborn in Go. While initial testing indicates that it is an order of magnitude slower than pmysql running against 1,000+ instances of MySQL, it’s still well within my tolerance (and there are many areas of …

[Read more]
Streaming MySQL binary logs for backup

With MySQL, it is relatively easy to create “point in time” restores. All you need is recent(ish) backup and a bunch of saved binary logs. You can restore the backup you have, and when it is completed, you can use mysqlbinlog utility to apply your saved binary logs to the desired state of your database.

I have created a simple go application to make your life easier. You can find it on my GitHub page.

The app works as reading its config file for the MySQL server connection credentials, a local directory where the binary logs will be kept, and the will path of the mysqlbinlog utility.

  1. It checks the binlogs on the remote server which able to streamed
  2. Checks the local directory where the binlogs are kept, to check which logs are already there
  3. The incomplete (file size differs local and on remote server) files will be …
[Read more]
Streaming MySQL binary logs for backup

With MySQL it is relatively easy to create point in time restores. All you need is a recent(ish) backup, and a bunch of saved binary logs. You can restore the backup you have, and when it is completed, you can use mysqlbinlog utility to apply your saved binary logs to the desired state of your database. I […]

My first impressions about Go language

I am fascinated. Maybe that should be enough, but I guess I have to write a bit more here because we are not on twitter.

I spent a few days to get know Go language, and now I am more than satisfied. I mean, all the project ideas which are floating in my head should be written in Go.

First of all, I have rewritten Mambo-collector to go (https://github.com/banyek/mambo) because I have faced some serious errors when I used it on CentOS 7 – I blame systemd -: If the process was running as root then after a few days of data collecting, killing that process was lead to restart the entire system, which is not a bug, it is a catastrophe. I tried to debug it several ways, but I am not sure where the problem is, it could be at the ‘loghandler’ redirection or any other place in python-daemon, or it is simply there is a buffer inside which overflows – I don’t really care, because mambo was just a proof of concept – what I used in …

[Read more]
Faster fingerprints and Go packages for MySQL

I’m happy to announce Go packages for MySQL. Particularly exciting is a new query fingerprint function which is very fast and efficient, but I’ll talk about that later. First, go-mysql is currently three simple Go packages for parsing and aggregating MySQL slow logs. If you’ve been following Percona development, you’ve no doubt heard of Percona Cloud Tools (PCT), a somewhat new performance management web service for MySQL.

One tool in PCT is “Query Analytics” which continuously analyzes query metrics from the slow log. The slow log provides the most metrics and therefore the most performance insight into MySQL. percona-agent, the open-source agent for PCT, uses …

[Read more]
Why did we develop percona-agent in Go?

We recently open-sourced our percona-agent and if you check out the source code, you’ll find that it is written in the Go programming language (aka Golang). For those not up to speed, the percona-agent is a real-time client-side agent for Percona Cloud Tools.

Our requirements are quite demanding for our agents. This one is software that works on a real production server, so it must be fast, reliable, lightweight and easy to distribute. Surprisingly enough, binaries compiled by Go fit these characteristics.

There are of course alternatives that we considered. On the scripting side: Perl, Python, PHP, Ruby et al. These are not necessarily fast, and the distribution is also interesting. We have enough experience with …

[Read more]
Showing entries 1 to 10 of 18
8 Older Entries »