The Question Recently, a customer asked us:

After importing a new section of user data into our Tungsten cluster, we are seeing perpetually rising replication lag. We are sitting at 8.5hrs estimated convergence time after importing around 50 million rows and this lag is climbing continuously. We are currently migrating some of our users from a NoSQL database into our Tungsten cluster. We have a procedure to write out a bunch of CSV files after translating our old data into columns and then we recursively send them to the write master using the mysql client. Specifically our import SQL is doing LOAD DATA LOCAL INFILE and the reading in a large CSV file to do the import. We have 20k records per CSV file and we have 12 workers which insert them in parallel.

Simple Overview The Skinny

In cases like this, the slaves are having trouble with the database unable to keep up with the apply stage …

The Question Recently, a customer asked us:

Why would heavy disk IO cause the Tungsten Manager and not MySQL to be starved of resources?

For example, we saw the following in the Manager log file tmsvc.log:

2019/06/03 00:50:30 | Pinging the JVM took 29 seconds to respond.
2019/06/03 00:50:30 | Pinging the JVM took 25 seconds to respond.
2019/06/03 00:50:30 | Pinging the JVM took 21 seconds to respond.
2019/06/03 00:50:30 | Pinging the JVM took 16 seconds to respond.
2019/06/03 00:50:30 | Pinging the JVM took 12 seconds to respond.
2019/06/03 00:50:30 | Pinging the JVM took 8 seconds to respond.

The Answer Why a Java application might be slow or freezing

The answer is that if a filesystem is busy being written to by another process, the background I/O will cause the Java JVM garbage collection (GC) to pause.

This problem is not specific to Continuent Tungsten …

Restating MySQL can be really annoying. You just want to disable the goddamn query cache and it takes forever (read 5-10 minutes) to shutdown, not to mention the warm-up time. Yes, with MySQL 5.7 you can do many changes online, so you won’t necessarily be restarting that often, but you still need to do upgrades, occasionally increase redo log size and, admit it, enable skip-grant-tables. Here’s how you can make this process way less painful.

Why is MySQL so slow to restart?

Before we go any further, let me tell you right away that when I’m speaking about MySQL here, I’m actually speaking about InnoDB, or rather, a MySQL server that’s running InnoDB as the main storage engine. And if that’s not your case, do not read any further. You’ve been warned!

Now.. ah yes. Restart. So, restarting MySQL involves two slow stages. I have already mentioned them, but repetition is the mother of skill, so let me say …

This is a little quiz (could be a discussion). I know what we tend to prefer (and why), but we’re interested in hearing additional and other opinions!

Given the way MySQL/MariaDB is architected, what would you prefer to see in a new server, more cores or higher clock speed? (presuming other factors such as CPU caches and memory access speed are identical).

For example, you might have a choice between

• 2x 2.4GHz 6 core, or
• 2x 3.0GHz 4 core

which option would you pick for a (dedicated) MySQL/MariaDB server, and why?

And, do you regard the “total speed” (N cores * GHz) as relevant in the decision process? If so, when and to what degree?

On my post last week, I analysed some of the most common compression tools and formats, and its compression speed and ratio. While that could give us a good idea of the performance of those tools, the analysis would be incomplete without researching the decompression. This is particularly true for database backups as, for those cases where the compression process is performed outside of the production boxes, you may not care too much about compression times. In that case, even if it is relatively slow, it will not affect the performance of your MySQL server (or whatever you are using). The decompression time, however, can be critical, as it may influence in many cases the MTTR of your whole system.

Testing …

This week we are talking about size, which is a subject that should matter to any system administrator in charge of the backup system of any project, and in particular database backups.

I sometimes get questions about what should be the best compression tool to apply during a particular backup system: gzip? bzip2? any other?

The testing environment

In order to test several formats and tools, I created a .csv file (comma-separated values) that was 3,700,635,579 bytes in size by transforming a recent dump of all the OpenStreetMap nodes of the European portion of Spain. It had a total of 46,741,126 rows and looked like this:

171773  38.6048402      -0.0489871      4       2012-08-25 00:37:46     12850816        472193  rubensd
171774  38.6061981      -0.0496867      2       2008-01-19 10:23:21     666916  9250 …

Join 6100 others and follow Sean Hull on twitter @hullsean. There are two ways to speedup UNIONs in a MySQL database. First use UNION ALL if at all possible, and second try to push down your conditions. [mytweetlinks] 1. UNION ALL is much faster than UNION How does a UNION work? Imagine you have two [...]

Read the original article at Five More Things Deadly to Scalability

Join 6000 others and follow Sean Hull on twitter @hullsean. 1. Slow Disk I/O – RAID 5 – Multi-tenant EBS Disk is the grounding of all your servers, and the base of their performance. True with larger and larger main memory, much is available in cache, a server still needs to constantly read from disk [...]

For more articles like these go to Sean Hull's Scalable Startups

Related posts:

1. Mobile Scalability – What is it and why is it important?
2. …

Indexes are a very important part of databases and are used frequently to speed up access to particular data item or items. So before working with indexes, it is important to understand how indexes work behind the scene and what is the data structure that is used to store these indexes, because unless you understand the inner working of an index, you will never be able to fully harness its power.

How I lost 30 pounds in 2010 – a weight loss guide for developers Posted on December 30th, 2010 in blog, personal, zippykid.

2010 was a great year for me, I started a new company, and it’s been profitable since month 2, I closed my consulting company, and most importantly, I lost 30 pounds, and am more mobile/agile than I was in college. I’m 32 and I feel better than I did when I was 24. I’ve talked [...]

View Comments …