Overview The Skinny

In this blog post we will define what a split-brain scenario means in a MySQL database cluster, and then explore how a Tungsten MySQL database cluster reacts to a split-brain situation.

Agenda What’s Here?

• Define the term “split-brain”
• Briefly explore how the Tungsten Manager works to monitor the cluster health and prevent data corruption in the event of a network partition
• Also explore how the Tungsten Connector works to route writes
• Describe how a Tungsten MySQL database cluster reacts to a split-brain situation
• Illustrate various testing and recovery procedures

Split-Brain: Definition and Impact Sounds scary, and it is!

A split-brain occurs when a MySQL database cluster which normally has …

The Question Recently, a customer asked us:

We were wondering if the Connectors would be able to bind to localhost/127.0.0.1 instead of 0.0.0.0:3306? Since the Connector is installed on the application servers, all of the connections are coming from localhost. We would like to limit this exposure so that the 3306 port is not exposed externally. We ask because we are failing PCI checks that are able to access the database port externally.

The Answer YES!

You may set the IP address for the Connector to listen on by using the tpm command option: property=server.listen.address={IP_ADDRESS}

To force the Connector to listen on the localhost only use the following example:

shell> tools/tpm configure alpha --property=server.listen.address=127.0.0.1
shell> tools/tpm update --replace-release

Use the IP …

Overview The Skinny

In this blog post we will discuss how to use the Tungsten Connector keep-alive feature to ensure long-running MySQL & MariaDB/Percona Server client sessions stay connected in a Tungsten Cluster.

Agenda What’s Here?

• Briefly explore how the Tungsten Connector works
• Describe the Connector keep-alives – what are they and why do we use them?
• Discuss why the keep-alive feature is not available in Bridge mode and why
• Examine how to tune the keep-alive feature in the Tungsten Connector

Tungsten Connector: A Primer A Very Brief Summary

The Tungsten Connector is an intelligent MySQL database proxy located between the clients and the database servers, providing a single connection point, while routing queries to …

Overview The Skinny

When it comes to zero downtime, proxies are the first line components of a cluster.

In order to achieve High Availability (HA) for MySQL, MariaDB and Percona Server, a commonly deployed setup consists of configuring load balancers (hardware or software) on top of those proxies.

A Strong Architecture How is Maintenance Made Possible?

With this proxy + load balancer architecture, server maintenance is made possible on any of the proxy hosts, as follows:

• the proxy is stopped
• the load balancer detects the dead proxy and removes it from the pool
• new connection requests go to live proxies

The Problem What Happens to Existing Sessions?

But wait… even though new connections are re-routed correctly, what happens …

Background The Skinny

Performing schema changes often requires extended downtime for applications. This is due to MySQL needing to rebuild tables for common schema change operations. Tools like pt-online-schema-change have been written to try to overcome the downtime associated with schema changes, however they are complex and put a high load on the database. Amazon’s Aurora improves some schema changes operations, but still requires a table rebuild for common operations like adding a column using before or after, or simply to add a column with a default value. Rebuilding a table with millions of rows can take hours and prevent writes to that table the entire time.

How Can Tungsten Clustering Keep Applications Running? How Does It All Work?

…

The Question Recently, a customer asked us:

What is the meaning of this error message found in trepsvc.log?

2019/05/14 01:48:04.973 | mysql02.prod.example.com | [east - binlog-to-q-0] INFO pipeline.SingleThreadStageTask Performing rollback of possible partial transaction: seqno=(unavailable)

Simple Overview The Skinny

This message is an indication that we are dropping any uncommitted or incomplete data read from the MySQL binary logs due to a pending error.

The Answer Safety First

This error is often seen before another error and is an indication that we are rolling back anything uncommitted, for safety. On a master this is normally very little and would likely be internal transactions in the trep_commit_seqno table, for example.

As you may know with the replicator we always extract complete transactions, and so this particular message is …

Overview The Skinny

In this blog post we will discuss how the managed cross-site replication streams work in a Composite Multi-Master Tungsten Cluster for MySQL, MariaDB and Percona Server.

Agenda What’s Here?

• Briefly explore how managed cross-site replication works in a Tungsten Composite Multi-Master Cluster
• Describe the reasons why the default design was chosen
• Explain the pros and cons of changing the configuration
• Examine how to change the configuration of the managed cross-site replicators

Cross-Site Replication A Very Brief Summary

In a standard Composite Multi-Master (CMM) deployment, the managed cross-site replicators pull Transaction History Logs (THL) from every remote cluster’s current master node. …

The Question Recently, a customer asked us:

Why would Tungsten Replicator be slow to apply to MySQL?

The Answer Performance Tuning 101

When you run trepctl status and see:
appliedLatency : 7332.394
like this on a slave, it is almost always due to the inability for the target database to keep up with the applier.

This means that we often need to look first to the database layer for the solution.

Here are some of the things to think about when dealing with this issue:

Architecture and Environment
√ Are you on bare metal?
√ Using the cloud?
√ Dev or Prod?
√ Network speed and latency?
√ Distance the data needs to travel?
√ Network round trip times? Is the …

The Question Recently, a customer asked us:

If we move to using the INI configuration method instead of staging, would password-less SSH still be required?

The Answer The answer is both “Yes” and “No”

No, for installation and updates/upgrades specifically. Since INI-based configurations force the tpm command to act upon the local host only for installs and updates/upgrades, password-less SSH is not required.

Yes, because there are certain commands that do rely upon password-less SSH to function. These are:

• tungsten_provision_slave
• prov-sl.sh
• multi_trepctl
• tpm diag (pre-6.0.5)
• tpm diag --hosts (>= 6.0.5)
• Any tpm-based backup and restore operations that involve a remote node

Summary The Wrap-Up

In …

Overview The Skinny

In this blog post we will discuss how to best integrate various Continuent-bundled cluster monitoring solutions with PagerDuty (pagerduty.com), a popular alerting service.

Agenda What’s Here?

• Briefly explore the bundled cluster monitoring tools
• Describe the procedure for establishing alerting via PagerDuty
• Examine some of the multiple monitoring tools included with the Continuent Tungsten Clustering software, and provide examples of how to send an email to PagerDuty from each of the tools.

Exploring the Bundled Cluster Monitoring Tools A Brief Summary

Continuent provides multiple methods out of the box to monitor the cluster health. The most popular is the suite of Nagios/NRPE scripts (i.e. cluster-home/bin/check_tungsten_*). We also have Zabbix scripts (i.e. cluster-home/bin/zabbix_tungsten_*). Additionally, there is …