A recommendation we often give to our customers is along the lines of “archive old data” to reduce your database size. There is a tradeoff between keeping all our data online and archiving part of it to cold storage.

There could also be legal requirements to keep certain data online, or you might want to query old data occasionally without having to go through the hassle of restoring an old backup.

In this post, we will explore a very useful feature of AWS RDS/Aurora that allows us to export data to an S3 bucket and run SQL queries directly against it.

Archiving Data to S3

Let’s start by describing the steps we need to take to put our data into an S3 bucket in the required format, which is called Apache Parquet.

Amazon states the Parquet format is up to 2x faster to export and consumes up to 6x less storage in S3, compared to other text formats.

1. Create a snapshot of the database (or …

Cloud backup is a strategy increasingly used in organizations. Send copies of your data to the cloud, can help you to prevent a devastating IT crisis and ensure business continuity.

Currently, MySQL Enterprise Backup supports the following types of cloud storage services:

- Oracle Cloud Infrastructure (OCI) Object Storage

- OpenStack Swift or compatible object storage services

- Amazon Simple Storage Service (S3) or compatible storage service

The post Create a Cloud Backup with MySQL Enterprise Backup first appeared on dasini.net - Diary of a MySQL experts.

We are living in the DataLake world. Now almost every oraganization wants their reporting in Near Real Time. Kafka is of the best streaming platform for realtime reporting. Based on the Kafka connector, RedHat designed the Debezium which is an OpenSource product and high recommended for real time CDC from transnational databases. I referred many blogs to setup this cluster. But I found just basic installation steps. So I setup this cluster for AWS with Production grade and publishing this blog.

A shot intro:

Debezium is a set of distributed services to capture changes in your databases so that your applications can see those changes and respond to them. Debezium records all row-level changes within each database table in a change event stream, and applications simply read these streams to see the change events in the same order in which they occurred.

Basic Tech Terms:

• Kafka …

We are living in the DataLake world. Now almost every organizations wants their reporting in Near Real Time. Kafka is of the best streaming platform for realtime reporting. Based on the Kafka connector, RedHat designed the Debezium which is an OpenSource product and high recommended for real time CDC from transnational databases. I referred many blogs to setup this cluster. But I found just basic installation steps. So I setup this cluster for AWS with Production grade and publishing this blog.

A shot intro:

Debezium is a set of distributed services to capture changes in your databases so that your applications can see those changes and respond to them. Debezium records all row-level changes within each database table in a change event stream, and applications simply read these streams to see the change events in the same order in which they occurred.

Basic Tech Terms:

• Kafka …

OblakSoft is pleased to announce general availability of the Cloud Storage Engine for MySQL (ClouSE) version 1.0.  Whether you want to use ready-to-run WordPress for Amazon Cloud / Google Cloud, configure your own WordPress installation with WP2Cloud plugin, or use ClouSE to deploy your own applications, ClouSE lets you tap the power of cloud storage to store your data extremely durably and make serving the content highly scalable.

We would like to thank more than 2,000 registered Beta users and early adopters who helped …

MySQL Enterprise Backup(MEB) has been widely used as the most efficient tool to take backups of huge databases. The storage of backups can be done on-premises – viz, local hard disk, external hard disk drives, network mounted disk. MEB can also stream backups to tape by supporting different Media Management Software like Symantec NetBackup, EMC2 and Oracle Secure Backup. Streaming of backups can also be done using 'ssh' to another machine.

As data keeps growing, instead of expanding the storage, database users are choosing to rely on the growing trend which is the Cloud Storage.

A few facts about Cloud Storage:

1. Offers off-premises storage with AAA(Anytime, Anywhere Access)
2. Sharing of data to make a collaborative effort.
3. Highly fault tolerant, durable and scalable and secure.
4. Cost-effective

To the benefit of our MySQL database users, Mysql …

Shard-Query (source) now supports the MySQL “LOAD DATA INFILE” command.

When you use LOAD DATA LOCAL INFILE a single threaded load from the current process will be performed.  You can specify a path to a file anywhere readable by the PHP script.  This allows loading without using the Gearman workers and without using a shared filesystem.

If you do not specify LOCAL, then the Gearman based loader is used.  You must not specify a path to the file when you omit the LOCAL keyword.  This is because the shared path will the pre-pended to the filename automatically.  The shared path must be a shared or network filesystem (NFS,CIFS,etc) and the files to be loaded must be placed on the shared filesystem for the Gearman based loader to work.  This is because workers may run on multiple nodes and …

ClouSE 1.0b.1.8 is released

OblakSoft is pleased to announce the release of ClouSE Beta version 1.0b.1.8.  This is a minor maintenance release that addresses a couple pain points, while we’re working on a release candidate that will bring the built-in point-in-time recovery and other major features.

Here is the summary of changes:

• Improved caching for Weblobs

Now ClouSE sets the cache control headers for Weblobs.  The cache control headers specify the max-age directive that allows caching content for one year.  This makes proxies and content delivery networks (CDN) more efficient.  The change resolves this support issue.

• Improved error handling on startup

Now database requests fail if …

Updating ClouSE can now be done with one command

With more than a thousand registered users worldwide, guiding customers through the upgrade process is getting hard, so we decided to automate the servicing steps.  Now the update-clouse script can be used to update ClouSE binaries to the current version.  This script ships with the ClouSE binary distribution.

Here is an example of updating ClouSE using the Webmin console.

First, download ClouSE 64-bit binaries from http://www.oblaksoft.com/downloads/.  Then upload and extract the binaries to the /home/ec2-user directory. In Webmin, navigate to Others, then to Upload and Download, then …

Cloud Storage Engine (ClouSE) installation and configuration is now fully automated. Try new ClouSE installer and you’ll have cloud storage access configured in no time.

OblakSoft is pleased to announce the release of Cloud Storage Engine (ClouSE) installer that can be used to deploy and configure ClouSE with your MySQL server. Now ClouSE can be deployed with just one command. The installer will guide the user through the configuration process, prompt for Cloud Storage connection parameters, and install ClouSE into MySQL.

Presented is a step-by-step guide for ClouSE installation and configuration, from downloading the binaries to running the installer. The guide was designed by request of WordPress-to-Cloud users, however it is fully applicable to any generic ClouSE installation.

The slides have notes, they are a little hard to discover with the SlideShare interface, look for a tab below the slide that looks …