Showing entries 11 to 20 of 81
« 10 Newer Entries | 10 Older Entries »
Displaying posts with tag: troubleshooting (reset)
Impact of “tmpdir” change in MySQL replication

Recently we had encountered a strange issue with replication and temp directory(tmpdir) change while working for one major client.

All the servers under this were running with Percona flavor of MySQL versioned 5.6.38 hosted on a Debian 8(Jessie)

The MySQL architecture setup  is as follows one master with 5 direct slaves under it

Through this blog, we will take you through the issue we had faced and how we fixed ultimately.

Client Request:

There was a simple request from our client to add a column and index to a 16GB production table since the table had foreign keys, to avoid complications we decided to go with online DDL instead of pt-online-schema.

When we started to alter, it got failed due to insufficient space in “tmpdir”.MySQL by default would be using “/tmp” for temp table creating, sorting and other temp operation, Since we had only 5.7G left on the “/” …

[Read more]
How to Switch Replica Master of a non-GTID Slave in Percona Cluster ?

Introduction –

Recently i worked on a production issue for one of our client under support .They have a architecture of a three node Galera cluster with one asynchronous slave .

  • Node1 – 172.10.2.11
  • Node2 – 172.10.2.12
  • Node3 – 172.10.2.13
  • Replica – 172.10.2.14

Architecture –

The slave(replica) was configured with node3 as replica master. Unfortunately the node 3 was crashed with an OOM killer ,also server has a low gcache size, so when i am trying to start the node 3 , it went to SST . Here the data size was around 2.6 TB , in general for completion of whole SST and joining the node back to cluster will take around  approximately 12 hours.

As i told earlier, the replication slave was under …

[Read more]
Configuring efficient MySQL Logrotate

I am a Junior DBA at Mydbops. This is my first blog professionally, I would like to brief my encounter with Log-rotate in first few weeks of my work,  Hope it will help other beginners as well. This Blog will cover the following sections.

  • Introduction to Log-rotate

  • Issues Faced

  • Solutions (Fix for the above issues)

  • Best practices

    • How to configure the Log-rotate

    • Operation of Log-rotate

    • Files responsible for the Log_rotate utility.

1.0. Introduction to Log-rotate:

  • Log-rotate is a utility and …

[Read more]
Will IO Size Affect your RDS Performance?​

During our recent consulting with one of our client, We came across an interesting issue on RDS. The baseline is that “Low IO size on your RDS instance can affect your DB performance”.  Yes, It’s IO size, Not IOPS.

We had our production systems running on RDS MySQL with a single master, 3 replicas. All instances are of same type db.m4.4xlarge with same parameter group configuration and the disk size is 1.5 TB. According to the AWS user guide, each of these instances can support up to 4500 (sustained IOPS) guaranteed IOPS.

Find below the Write IOPS graph for all the instances.

It’s understood that Write IOPS / pattern on Master can vary when compared with Slave, due to a lot of factors like binlog row format, log writing etc. But it has to be almost similar for all the slaves given that it …

[Read more]
Database Objects migration to RDS/ Aurora (AWS)

The world of application and its related services are migrating more towards cloud, because of availability, Elasticity, Manageability etc. While moving the entire stack we need to be very cautious while migrating the database part.

Migration of DB servers is not a simple lift and shift operation, Rather it would require a proper planning and more cautious in maintaining data consistency with existing DB server and cloud server by means of native replication or by using any third party tools.

The best way to migrate the existing MySQL database to RDS, in my opinion, is by using “logical backup“. Some of the logical backup tools as below,

Mysqldump — single threaded (widely used)

[Read more]
Troubleshooting MySQL Concurrency Issues with Load Testing Tools Webinar: Q & A

In this blog, I will provide answers to the Q & A for the Troubleshooting MySQL Concurrency Issues with Load Testing Tools webinar.

First, I want to thank everybody for attending my May 23, 2018, webinar on troubleshooting tools for MySQL. The recording and slides for the webinar are available here. Below is the list of your questions that I was unable to answer fully during the webinar.

Q: What do you recommend for benchmarking NDB cluster? Which should be used and how?

A: The issue with benchmarking NDB cluster is not the tool choice, …

[Read more]
ProxySQL Series: Handling resource expensive(bad) Queries in MySQL

This is our fourth blog in the ProxySQL Series

  1. MySQL Replication Read-write Split up
  2. Seamless Replication Switchover Using MHA
  3. Mirroring MySQL Queries

This blog focuses on how to quickly find and address badly written queries using ProxySQL without any downtime and change in application code.

When we get an incident about the high usage on a production master, then mostly it is because of unexpected spike in Traffic (QPS) or slow queries.

Below was the status when we were doing the …

[Read more]
Presentation : MySQL Timeout Variables Explained

MySQL has multiple timeout variables these slides helps to give an overview of the different  timeout variables and their purposes briefly.

Handling long duration SST(timeout) in PXC with systemd

In this blog post, We will be explaining about the timeouts in SST on systemd implementation which we faced recently in Percona XtraDB Cluster  during our Consulting with a client. State Snapshot Transfers (SST) refers to complete data sync from one of the nodes from the cluster to the joining node.

SST will happen for one or more reasons listed below.

  1. Initial sync to join a node to cluster.
  2. Node is out of cluster and lost its ability to join back due to data corruption or inconsistencies and also when the node went far behind the node, …
[Read more]
Basic Internal Troubleshooting Tools for MySQL Server Webinar: Q & A

In this blog, I will provide answers to the Q & A for the Basic Internal Troubleshooting Tools for MySQL Server webinar.

First, I want to thank everybody for attending my February 15, 2018, webinar on troubleshooting tools for MySQL. The recording and slides for the webinar are available here. Below is the list of your questions that I was unable to answer fully during the webinar.

Q: How do we prevent the schema prefix from appearing in the show create view. This is causing issue with restore on another server with a different DB. See the issue …

[Read more]
Showing entries 11 to 20 of 81
« 10 Newer Entries | 10 Older Entries »