How to build a multi-region, multi-master MySQL cloud database back-end capable of serving a global high volume cloud contact center

This global SaaS provider is a Cloud Contact Center solution provider (the leading Salesforce telephony solution), who needed to deliver up-to-date data to clients as quickly as possible. Its cloud-based call center software frees users to make every conversation personal; it supports the creation of exceptional customer experiences that help serve better and sell more. Its technology is 100% cloud-based and device-agnostic for telephony infrastructure.

What is the Challenge?

Active/Active (multi-master) MySQL clustering is needed when there is significant update load on geographically distributed applications. These require fast response times both for updates and reads, and they also need to share data on a global scale.

• Regional distribution of data
• Deliver …

At re:Invent in Las Vegas in December 2019, AWS announced the public preview of RDS Proxy, a fully managed database proxy that sits between your application and RDS. The new service offers to “share established database connections, improving database efficiency and application scalability”.

But one of the benefits that caught my eye is the ability to reduce the downtime in case of an instance failure and a failover. As for the announcement:

In case of a failure, RDS Proxy automatically connects to a standby database instance while preserving connections from your application and reduces failover times for RDS and Aurora multi-AZ databases by up to 66%”

You can read more about the announcement and the new service on the AWS …

I have published enough Debezium MySQL connector tutorials for taking snapshots from Read Replica. To continue my research I wanted to do something for AWS RDS Aurora as well. But aurora is not using binlog bases replication. So we can’t use the list of tutorials that I published already. In Aurora, we can get the binlog file name and its position from its snapshot of the source Cluster. So I used a snapshot for loading the historical data, and once it’s loaded we can resume the CDC from the main cluster.

Requirements:

1. Running aurora cluster.
2. Aurora cluster must have binlogs enabled.
3. Make binlog retention period to a minimum 3 days(its a best practice).
4. Debezium connector should be able to access both the clusters.
5. Make sure you have different security …

Credit: AWS

CDC is becoming more popular nowadays. Many organizations that want to build a realtime/near realtime data pipe and reports are using the CDC as a backbone to powering their real-time reports. Debezium is an opensource product from RedHat and it supports multiple databases (both SQL and NoSQL). Apache Kafka is the core of this.

Managing and scaling Kafka clusters is not easy for everyone. If you are using AWS for your infra then let AWS manage the cluster. AWS has MSK is a managed Kafka service. We are going to configure Debezium with AWS MSK.

Configuration File:

If you are already worked with AWS MSK, then you might be familiar with this configuration file. This is similar to the RDS Parameter group but here you need to upload a with your parameter name and its value. If you are using MSK for the first time, then it’ll make you a bit confused. No worries, I’ll give you the steps to do …

We are living in the DataLake world. Now almost every oraganization wants their reporting in Near Real Time. Kafka is of the best streaming platform for realtime reporting. Based on the Kafka connector, RedHat designed the Debezium which is an OpenSource product and high recommended for real time CDC from transnational databases. I referred many blogs to setup this cluster. But I found just basic installation steps. So I setup this cluster for AWS with Production grade and publishing this blog.

A shot intro:

Debezium is a set of distributed services to capture changes in your databases so that your applications can see those changes and respond to them. Debezium records all row-level changes within each database table in a change event stream, and applications simply read these streams to see the change events in the same order in which they occurred.

Basic Tech Terms:

• Kafka …

We are living in the DataLake world. Now almost every organizations wants their reporting in Near Real Time. Kafka is of the best streaming platform for realtime reporting. Based on the Kafka connector, RedHat designed the Debezium which is an OpenSource product and high recommended for real time CDC from transnational databases. I referred many blogs to setup this cluster. But I found just basic installation steps. So I setup this cluster for AWS with Production grade and publishing this blog.

A shot intro:

Debezium is a set of distributed services to capture changes in your databases so that your applications can see those changes and respond to them. Debezium records all row-level changes within each database table in a change event stream, and applications simply read these streams to see the change events in the same order in which they occurred.

Basic Tech Terms:

• Kafka …

When we think about the cloud, often we consider many of the benefits: scalability, elasticity, agility, and flexible pricing.  As great as these features are, security also remains a business-critical concern. In an on-premise environment, every aspect of security is owned by you.  Looking at the database layer specifically, these include (but are not limited to):

• Data encryption
• Database access control
• Network security
• OS security (both host and guest if in VM environment)
• Physical security

When done properly, that entails a significant amount of work and generally cost.  In the cloud, those aspects are all still relevant and necessary for proper security.  However, under the shared responsibility model, some of that work is offloaded from you and shifted to the cloud provider.  Let’s look at what that model entails and how it is realized …

Our team has continued to pave the way for Continuous MySQL Operations this year with our Continuent Tungsten products: Tungsten Clustering and Tungsten Replicator.

And we’d like to take this opportunity to thank you all – our customers, partners, followers and colleagues – for your support in 2019, and celebrate some of our successes with you…

2019 Continuent Momentum Highlights

• Launched three new Tungsten Clustering & Tungsten Replicator releases
• Introduced the new Tungsten Replicator (AMI) on the Amazon Marketplace
• Were named a 2020 Top Trending Product by Database Trends & Applications Magazine

2019 Continuent Customer Highlights

• 100%: Customer Satisfaction during the most recent customer survey
• 97.5%: our …

AWS is the #1 cloud provider for open-source database hosting, and the go-to cloud for MySQL deployments. As organizations continue to migrate to the cloud, it’s important to get in front of performance issues, such as high latency, low throughput, and replication lag with higher distances between your users and cloud infrastructure. While many AWS users default to their managed database solution, Amazon RDS, there are alternatives available that can improve your MySQL performance on AWS through advanced customization options and unlimited EC2 instance type support. ScaleGrid offers a compelling alternative to hosting MySQL on AWS that offers better performance, more control, and no cloud vendor lock-in and the same price as Amazon RDS. In this post, we compare the performance of MySQL Amazon RDS …

Managing the logs in a centralized repository is one of the most common best practices in the DevOps world. Application logs, system logs, error logs, and any databases logs also will be pushed into your centralized repository. You can use ELK stack or Splunk to visualize the logs to get better insights about it. But as a SQL guy, I wanted to solve this problem with Bigdata ecosystem(use SQL). As a part of that process, we can relationalize unstructured data in AWS Athena with the help of GrokSerDe.

Here S3 is my centralized repository. I know it will not scale like ElasticSearch, but why should I miss this Fun. For this use case, Im going to rationalize the SQL Server Error log in AWS Athena. Let’s take a look at the SQL server’s error log pattern.

2019-09-21 12:53:17.57 Server      UTC adjustment: 0:00
2019-09-21 12:53:17.57 Server      (c) Microsoft Corporation.
2019-09-21 12:53:17.57 Server      All rights reserved. …