Writing MySQL Proxy in GO for self-learning: Part 2 — decoding handshake packet

After we built a generic TCP proxy, we can continue with our journey. Today’s goal will be to understand the MySQL Protocol, receive, decode, encode and send to the client the first packet sent by the MySQL Server.

MySQL connections threads

Each client connection to MySQL Server handled by a thread. MySQL is portable, so the underhood threads implementation is system dependent (Windows, macOS, and Linux have their own threads implementation).

What important to us, it’s to understand that a single client connection …

Apache Spark is the de facto framework of the big data world. Any serious organization that’s dealing with big data uses Spark almost exclusively. Though, it has some caveats. For the starter, it’s hard to use. And it’s very confusing to get started with, even for those with a solid …

The post MySQL table to JSON with 10 lines of Spark appeared first on Geeky Hacker.

Writing MySQL Proxy in GO for self-learning: Part 1 — TCP Proxy

Weekend. Quarantine. If you bored like me, and want to learn something new, you can join my journey of writing yet another MySQL Proxy Server.

In the past, I had a crazy idea of writing MySQL Proxy Server. The first reason was to understand the MySQL Communication Protocol to let me write more efficient programs. The second one was to learn more about low-level network programming concepts in general. Finally, I have time at least to start… and you can join me!

The real usage of MySQL Proxy can vary:

• Testing and benchmarking of the infrastructure
• Testing your applications for dealing with MySQL network failures and delays
• Wrap the MySQL usage with own business layer
• Load balancing
• Multiplexing

Prelude

I’m not sure if I can …

Do not use the old mysql api

There are several ways to connect to a MySQL database in PHP. The most common ones are the MySQL API, the MySQLi API and the PDO API (PHP Data Objects). The last two support more features than the old mysql API and are more secure. If you’re using the old “mysql_” functions, you should stop and learn the new PDO API. Those old mysql functions are deprecated and are no longer supported in PHP 7.x.

Bad practice:

<?php  
$con = mysql_connect("localhost", "root", "mypass") or  
    die("Could not connect: " . mysql_error());  
mysql_select_db("tutorials");  
$result = mysql_query("select * from tutorials");  
echo "<h2>Here is a list of the topics:</h2>";  
while ($row = mysql_fetch_array($result)) {  
    echo $row['name']."<br />";  
}  
mysql_close($con);  
?>  

Better practice:

 …

We all do need sometimes to generate raw valid dummy data for our use cases and applications as we start them. Obviously, one can write their own scripts to generate random data, but it is much better to have data, to which human beings can associate with like names, addresses instead of having them filled with random "lorem ipsum" string data :)

While searching for such a tool, I found a site which does exactly this: http://www.generatedata.com/

Documentation: http://benkeen.github.io/generatedata/

This can also be downloaded and installed locally. It supports three types of installations:
- A single, anonymous user account
- A single user account, requires login
- Multiple accounts

Below is the set of wide varied data types it supports for …

New MariaDB/MySQL storage engines such as MyRocks and TokuDB have renewed interest in using engines other than InnoDB. This is great, but also presents new challenges. In this article, I will describe work that I am currently finishing, and which addresses one such challenge.

For example, the left bar in the figure shows what happens to MyRocks replication performance when used with a default install where the replication state table uses InnoDB. The middle bar shows the performance improvement from my patch.

Current MariaDB and MySQL replication uses tables to transactionally record the replication state (eg mysql.gtid_slave_pos). When non-InnoDB storage engines are introduced the question becomes: What engine should be used for the replication table? Any choice will penalise other engines heavily by injecting a cross-engine transaction with every replicated change. Unless all tables can be migrated to the …

New MariaDB/MySQL storage engines such as MyRocks and TokuDB have renewed interest in using engines other than InnoDB. This is great, but also presents new challenges. In this article, I will describe work that I am currently finishing, and which addresses one such challenge.

For example, the left bar in the figure shows what happens to MyRocks replication performance when used with a default install where the replication state table uses InnoDB. The middle bar shows the performance improvement from my patch.

Current MariaDB and MySQL replication uses tables to transactionally record the replication state (eg mysql.gtid_slave_pos). When non-InnoDB storage engines are introduced the question becomes: What engine should be used for the replication table? Any choice will penalise other engines heavily by injecting a cross-engine transaction with every replicated change. Unless all tables can be migrated to the …

I’ve been a fan of GNU Parallel for a while but until recently have only used it occasionally. That’s a shame, because it’s often the simplest solution for quickly solving embarrassingly parallel problems.

My recent usage of it has centered around database export/import operations where I have a file that contains a list of primary keys and need to fetch the matching rows from some number of tables and do something with the data. The database servers are sufficiently powerful that I can run N copies of my script to get the job done far faster (where N is value like 10 or 20).

A typical usage might look like this:

cat ids.txt | parallel -j24 --max-lines=1000 --pipe "bin/munge-data.pl  --db live >> {#}.out

However, I recently found myself scratching my head because parallel was only running 3 jobs rather than the 24 I had specified. …

[Here is a PDF version for readers whose browser does not understand MathML.]

The Simplex Noise (or Improved Perlin Noise) algorithm uses a somewhat mysterious "skew factor" of 3-12. I did not find any really satisfactory explanation for this factor in the descriptions of Simplex Noise that I read. But I managed to work it out nevertheless, which I thought was a fun exercise and worth a quick write-up.

Simplex noise is constructed by assigning random values to each point in a simplex grid. The simplex grid is a tiling of the plane using rhombuses, each rhombus consisting of two equilateral triangles. See the figure on the right.

Given a point (x,y) (expressed in normal rectangular coordinates), we …

Since Ruby on Rails 4.2 has just been released, perhaps now is a good time to review creating a shiny new Rails project. It’s not often I get to create a new project from scratch, but it’s Christmas and I’ve got a bit of downtime — and an itch I’d like to scratch! So, let’s get started.

I’m aiming to build a wee project that keeps track of OmniFocus perspectives. I’ve noticed that people are sharing their perspectives as screenshots and descriptions, in tweets and blog posts. Wouldn’t it be awesome if there was a one-stop-shop for everybody’s perspectives?

A couple of early decisions in terms of the basic starting point:

• Chances are I’ll deploy the app onto Heroku, so it’s a no-brainer to start out by using PostgreSQL in development. (Even …