Home |  MySQL Buzz |  FAQ |  Feeds |  Submit your blog feed |  Feedback |  Archive |  Aggregate feed RSS 2.0 English Deutsch Español Français Italiano 日本語 Русский Português 中文
Showing entries 1 to 30 of 628 Next 30 Older Entries

Displaying posts with tag: Replication (reset)

"Anemomaster": DML visibility. Your must-do for tomorrow
+0 Vote Up -0Vote Down

Here's our take of master DML query monitoring at Outbrain (presented April 2014). It took a half-day to code, implement, automate and deploy, and within the first hour of work we managed to catch multiple ill-doing services and scripts. You might want to try this out for yourself.

What's this about?

What queries do you monitor on your MySQL servers? Many don't monitor queries at all, and only look up slow queries on occasion, using pt-query-digest. Some monitor slow queries, where Anemometer (relying on pt-query-digest) is a very good tool. To the extreme, some monitor TCP traffic

  [Read more...]
Making the MTR rpl suite GTID_MODE Agnostic
Employee_Team +1 Vote Up -0Vote Down

In MySQL 5.6 we introduced GTID_MODE as a new server option. A global transaction identifier (GTID) is a unique identifier created and associated with each transaction when it is committed on the server of origin (master). This identifier is unique not only to the server on which it originated, but is unique across all servers in a given replication setup. There is a 1-to-1 mapping between all transactions and all GTIDs. For additional information, please refer to the MySQL manual.

Prior to 5.6.17 and 5.7.4, we had GTID specific replication (referred to as “rpl” within the MTR suite) regression tests, and we had to separately run

  [Read more...]
How TokuMX Secondaries Work in Replication
+0 Vote Up -0Vote Down

As I’ve mentioned in previous posts, TokuMX replication differs quite a bit from MongoDB’s replication. The differences are large enough such that we’ve completely redone some of MongoDB’s existing algorithms. One such area is how secondaries apply oplog data from a primary. In this post, I’ll explain how.

In designing how secondaries apply oplog data, we did not look closely at how MongoDB does it. In fact, I’ve currently forgotten all I’ve learned about MongoDB’s implementation, so I am not in a position to compare the two. I think I recall that MongoDB’s oplog idempotency was a key to their

  [Read more...]
MySQL-5.7.4- Change master without stopping slave altogether
Employee_Team +1 Vote Up -0Vote Down
At MySQL, we have been working on simplifying the failover process
making it faster, more flexible and easier to use. In MySQL 5.6 we added
support for Global Transaction Identifiers (GTID), which was a huge leap in the
direction of easing the failover process hiding the details about
replication logs and positions. With MySQL 5.7.4, we are introducing a
new feature that further adds to flexibility and onliness- the user can
only shut down components that he needs to re-configure. 

What we allow with this new feature is to execute CHANGE MASTER TO
command without stopping slave altogether. We realized that stopping
slave altogether is not mandatory in all cases and doing that was more
of a cautious approach to switching master restricting more than what’s
required at times.

Lets dive









  [Read more...]

New! MySQL Utilities release-1.4.2-RC
Employee +2 Vote Up -0Vote Down
The MySQL Utilities Team is pleased to announce the latest release candidate (RC) release of MySQL Utilities. This release includes a number of improvements for useabilty, stability, and a few enhancements. A complete list of all improvements can be found in our release_notes.

New Utilities!


We have also included two new utilities.
  • The mysqlrplsync utility was added, which checks data consistency between servers in a replicated setup. 
  • The mysqlrplms utility was added, which provides round-robin multi-source replication (a slave server continually cycles through multiple masters in order to store a consolidated data set).

How Can I Download MySQL Utilities?


You can download MySQL Utilities 1.4.2 from





  [Read more...]
Tungsten Replicator 3.0 is Cloudera Enterprise 5 Certified
+0 Vote Up -0Vote Down

One of the key platforms I’ve been testing on for the MySQL to Hadoop replication has been Cloudera, largely driven by customer requirements, but it’s also one of the easiest way to get started with Hadoop.

What I’m even more pleased about is the fact that we are proud to announce that Tungsten Replicator 3.0 is certified for use on the new Cloudera Enterprise 5 platform. That means that we’re sure that replicating your data from MySQL to Cloudera 5 and have it work without causing problems or difficulties on the Hadoop

  [Read more...]
New MySQL Utility: Replication Synchronization Checker
+1 Vote Up -0Vote Down
We are very happy to introduce an new MySQL utility called 'mysqlrplsync' that can check the data consistency of an active replication system. This utility is one of the new utilities included in MySQL Utilities release-1.4.2 RC. The other utility is the multi-source replication utility, mysqlrplms.

In a nutshell, the mysqlrplsync utility allows you to check the data consistency between servers in an active replication system. The utility reports missing databases and tables as well as data differences (per table) between the servers. A sophisticated synchronization algorithm that utilizes a table checksum is applied on the active replication servers to locate differences in the data.

Main Features

Besides permitting the data consistency check of an active replication system, here are the main features of the mysqlrplsync utility:



  [Read more...]
Continuent Replication to Hadoop – Now in Stereo!
+0 Vote Up -0Vote Down

Hopefully by now you have already seen that we are working on Hadoop replication. I’m happy to say that it is going really well. I’ve managed to push a few terabytes of data and different data sets through into Hadoop on Cloudera, HortonWorks, and Amazon’s Elastic MapReduce (EMR). For those who have been following my long association with the IBM InfoSphere BigInsights Hadoop product, and I’m pleased to say that it’s working there too. I’ve had to adapt Robert’s original script to work with the different versions of the underlying Hadoop tools and systems to make it compatible. The actual performance and process is unchanged; you just use a different JS-based batchloader script to work with different tools.

Robert has also been simplifying some of the core functionality, such as configuring some fixed pre-determined

  [Read more...]
Fun with Bugs #31 - what's new in MySQL 5.6.17
+1 Vote Up -0Vote Down
MySQL 5.6.17 will probably be announced loudly at or immediately before Percona Live MySQL Conference & Expo next week. But official release announcement via email was made on March 28, release notes and binaries to download are already available, so why not to check them carefully to find out what to expect from this 8th minor release of MySQL 5.6 GA...

First of all, it seems Oracle still does not hesitate to introduce new features and behavior in the process. Just check these major changes:
  • Starting with 5.6.17, MySQL now supports rebuilding regular and partitioned InnoDB tables using



  [Read more...]
MySQL 5.6 GTIDs: Evaluation and Online Migration
+0 Vote Up -0Vote Down

A colleague and I have been looking at GTID on MySQL recently and you may be interested in the blog post that results from that. You can see it here. http://blog.booking.com/mysql-5.6-gtids-evaluation-and-online-migration.html.

 

How to Migrate Your Application to Amazon Virtual Private Cloud without Downtime
+0 Vote Up -0Vote Down
March 27, 2014 By Severalnines

With datacenters being stretched by resource-intensive applications, more and more businesses are outgrowing their existing in-house capacity and having to reconfigure their IT operations. But how do you migrate a busy application to a totally new data center without downtime? How will the application scale in a virtualized cloud environment? And how do you guard against cloud server failures and keep a high level of uptime?

 

 

In this example, we will show you how to migrate a Web application (Wordpress) from a local data center to a AWS VPC. Without downtime even! 

 

Main steps:

      [Read more...]
    MariaDB Eventually Consistent?
    +0 Vote Up -0Vote Down
    Background

    Eventual consistency is a consistency model used in many large distributed databases which requires that all changes to a replicated piece of data eventually reach all affected replicas; conflict resolution is not handled and responsibility is pushed up to the application author in the event of conflicting updates [13].

    Eventual consistency is a specific form of weak consistency; the storage system guarantees that if no new updates are made to the object, eventually all accesses will return the last updated value [14]. If no failures occur, the maximum size of the inconsistency window can be determined based on factors such as communication delays, the load on the system, and the number of replicas involved in the replication scheme [3].

    A few examples of eventually consistent systems:

    • DNS
    • Asynchronous master/slave replication on an RDBMS e.g.
      [Read more...]
    My Favorite MongoDB Replication Feature: Crash Safety
    +0 Vote Up -0Vote Down

    At an extremely high level, replication in MongoDB and MySQL are similar. Both databases have exactly one machine, the primary (or master), that accepts writes from clients. With a single transaction (or atomic operation, in MongoDB’s case), the tables and oplog (or binary log in MySQL) are modified to reflect the change. The log captures what the change is so other secondaries (or slaves) can read the changes and process them, making the slaves identical to the master. (Note that I am NOT talking about multi-master replication.)

    Underneath the covers, their implementations are quite different. And in peeking underneath the covers while developing TokuMX, I learned

      [Read more...]
    Advanced MySQL Replication Architectures and Latest Developments – free webinar
    Employee +1 Vote Up -0Vote Down


    This Thursday (20th March 2014) we’ll be hosted a free webinar covering advanced MySQL Replication topics as well as the latest developments. As always, the webinar is free but you need to register here (http://www.mysql.com/news-and-events/web-seminars/advanced-mysql-replication-architectures-and-latest-developments/" target="_blank) – even if you can’t join live, you’ll then be sent a link to the replay.

    More details on what to expect…

    The biggest Web sites in the


      [Read more...]
    Real-Time Data Loading from MySQL to Hadoop using Tungsten Replicator 3.0 Webinar
    +0 Vote Up -0Vote Down

    To follow-up and describe some of the methods and techniques behind replicating into Hadoop from MySQL in real-time, and how this can be combined into your data workflow, Continuent are running a webinar with me presenting that will go over the details and provide a demo of the data replication process.

    Real-Time Data Loading from MySQL to Hadoop with New Tungsten Replicator 3.0

    Hadoop is an increasingly popular means of analyzing transaction data from MySQL. Up until now mechanisms for moving data between MySQL and Hadoop have been rather limited. The new Continuent Tungsten Replicator 3.0 provides enterprise-quality replication from MySQL to Hadoop. Tungsten Replicator 3.0 is 100% open source, released under a GPL V2 license, and available for download at

      [Read more...]
    No application changes needed: 10 times faster slave with MariaDB 10 parallel replication
    +3 Vote Up -1Vote Down

    Parallel replication is in MariaDB 10.0. I did some benchmarks on the code in 10.0.9. The results are quite good! Here is a graph that shows a 10-times improvement when enabling parallel replication: The graph shows the transaction per second as a function of number of slave worker threads, when the slave is executing events from the master at full speed. The master binlog was generated with sysbench oltp.lua. When the binlog is enabled on the slave and made crash-safe (--sync-binlog=1 --innodb-flush-log-at-trx-commit=1), the slave is about ten times faster at 12 worker threads and above compared to the old single-threaded replication.

    These results are for

      [Read more...]
    The "once and for all" SHOW SLAVE STATUS log files & positions explained
    +1 Vote Up -0Vote Down

    True, GTID is upon us whether via MySQL 5.6 or Tungsten Replicator (and wasn't it in Google Patches since 2009?).

    But some of us are still using standard replication with MySQL 5.5, and the "what's with all these binary log files and positions" question is ever erupting. The output of SHOW SLAVE STATUS confuses people new to it. It confuses me time and again.

    So here's the semi visual guide to interpreting the SHOW SLAVE STATUS.

    About binary logs and relay logs

    A master writes binary logs. These are typically and conventionally called mysql-bin.##### or mysqld-bin.##### (replace ##### with digits).

    A slave connects to its master, and reads entries from the master's binary logs. The slave writes those entries into its own relay logs. These are

      [Read more...]
    Reports exaggerated
    Employee +1 Vote Up -0Vote Down
    I've been letting the blog rest recently, and not so recently as well.  The problem is not a lack of subjects, but a lack of time to do them any justice.  However it is quite sad to see that my last entry was in September 2012, so it is time to post again.

    Of late I have been pondering what I have to say about :
    • Distributed MVCC and write-scaling
    • Different approaches to eventual consistency with replicated RDBMS
    • Various MySQL Cluster related topics
    • Various general rambling and unstructured topics
    However, these will take some time to percolate and calcify.

    In the meantime here are some things I have found interesting recently :






      [Read more...]
    MySQL to Hadoop Step-By-Step
    +0 Vote Up -0Vote Down

    We had a great webinar on Thursday about replicating from MySQL to Hadoop (watch the whole thing). It was great, but one of the questions at the end was ‘is there an easy way to test’.

    Sadly we can’t go giving out convenient ready-to-run downloads of these things because of licensing and and other complexities, so I want to try and make it as simple and straightforward as possible by giving you the directions to complete. I’m going to be point to the Continuent Documentation every now and then so this is not too crowded, but we should get through it pretty easily.

    Major Decisions

    For this to work: 

    • We’ll setup two VMs, one the master
      [Read more...]
    MMUG6: Madrid MySQL Users Group meeting to take place on 20th March 2014
    +1 Vote Up -0Vote Down

    Madrid MySQL Users Group will have its next meeting on 20th March. Details can be found on the group’s Meetup page.

    I will be giving a presentation on MySQL replication hopefully aimed at all levels, but covering some details relevant to larger setups. The meeting will be in Spanish.

    Look forward to seeing you there.

    La próxima reunión de Madrid MySQL Users Group tendrá lugar el jueves 20 de marzo. Se puede encontrar más detalles en la página del grupo.  Ofreceré una presentación sobre replicación de MySQL dirigido a gente de todos los niveles, pero incluirá información relevante a entornos más grandes.  La presentación será en español.

    Espero veros allí.

    Real-Time Replication from MySQL to Cassandra
    +1 Vote Up -0Vote Down

    Earlier this month I blogged about our new Hadoop applier, I published the docs for that this week (http://docs.continuent.com/tungsten-replicator-3.0/deployment-hadoop.html) as part of the Tungsten Replicator 3.0 documentation (http://docs.continuent.com/tungsten-replicator-3.0/index.html). It contains some additional interesting nuggets that will appear in future blog posts.

    The main part of that functionality that performs the actual applier for Hadoop is based around a JavaScript applier engine – there will eventually be docs for that as part of the Batch Applier content (

      [Read more...]
    MySQL Cluster to Hadoop
    Employee +1 Vote Up -0Vote Down

    How do you get data from a MySQL Cluster into Hadoop? Easy, replicate from the cluster to a stand alone MySQL instance and from there use the MySQL Hadoop Applier to HDFS.

    This question came from a long time MySQL user who has jumped into the Big Data world.


    MaxScale has now its own public irc channel
    +1 Vote Up -0Vote Down

    MaxScale is a Proxy for the MySQL protocol built with a modular architecture. The underlying concept of modules allows to extend the MaxScale proxy services. The current version implements Read Write splitting and Connection Load Balancing. Internally MySQL queries go through a SQL parsing phase. This gives MaxScale great capabilities regarding queries routing.

    So if [...]

    No Hadoop Fun for Me at SCaLE 12X :(
    +0 Vote Up -0Vote Down
    I blogged a couple of weeks ago about my upcoming MySQL/Hadoop talk at SCaLE 12X. Unfortunately I had to cancel. A few days after writing the article I came down with an eye problem that is fixed but prevents me from flying anywhere for a few weeks. That's a pity as I was definitely looking forward to attending the conference and explaining how Tungsten replicates transactions from MySQL into HDFS.

    Meanwhile, we are still moving at full steam with Hadoop-related work at Continuent, which is the basis for the next major replication release, Tungsten Replicator 3.0.0. Binary builds and documentation will go up in a few days. There will also be many more public talks about Hadoop support, starting in

      [Read more...]
    On the road again - FOSSAsia
    +0 Vote Up -0Vote Down

    On the road again - FOSSAsia

    It has been a few busy months until now. I have moved from Italy to Thailand, and the move has been my first priority, keeping me from attending FOSDEM and interacting with social media. Now I start catching my breath, and looking around for new events to attend. But before I get into this, let’s make a few things clear:

    • I am still working for Continuent. Actually, it’s because of my company flexibility that I could move to a different country (a different continent, 6 time zones away) without much trouble. Thanks, Continuent! (BTW: Continuent is hiring! )
    • I am still involved with MySQL activities, events, and community matters. I just happen to be in a different time zone, where direct talk with
      [Read more...]
    Using MASTER_GTID_WAIT() to avoid stale reads from slaves in replication
    +2 Vote Up -0Vote Down

    I have just implemented MASTER_GTID_WAIT() in MariaDB 10.0. This can be used to give a very elegant solution to the problem of stale reads in replication read-scaleout, without incuring the overheads normally associated with synchronous replication techniques. This idea came up recently in a discussion with Stephane Varoqui, and is similar to the concept of Lamport logical clock described in this Wikipedia article.

    I wanted to describe this, hoping to induce people to test and maybe start using this, as it is a simple but very neat idea, actually.

    A very typical use of MariaDB/MySQL replication is

      [Read more...]
    Anonymizing Data During Replication
    +0 Vote Up -0Vote Down

    If you happen to work with personal data, chances are you are subject to SOX (Sarbanes-Oxley) whether you like it or not.

    One of the worst aspects of this is that if you want to be able to analyse your data and you replicate out to another host, you have to find a way of anonymizing the information. There are of course lots of ways of doing this, but if you are replicating the data, why not anonymize it during the replication?

    Of the many cool features in Tungsten Replicator, one of my favorites is filtering. This allows you to process the stream of changes that are coming from the data extracted from the master and perform operations on it. We use it a lot in the replicator for ignoring tables, schemas and columns, and for ensuring that we have the correct information within the THL.

    Given this, let’s use it to anonymize the data as it is being

      [Read more...]
    MySQL 5.6 GA one year – What is next?
    +3 Vote Up -0Vote Down

    MySQL 5.6 has been GA for just over a year now. See MySQL 5.6.10 Release Notes.  Congratulations on your birthday! That is quite a long time. I was using it earlier in production because it worked and could do things that 5.5 could not do, but earlier versions were to use at your own risk, and indeed if prodded incorrectly would fall on the floor. That is fair enough because they were work in progress, yet if you poked them the right way they did a very good job.  Those dev versions have been long since upgraded which is good so they do not need quite as much care and attention.

    So from where I see 5.6 it works very well. One big change that has made a large difference but which I think a lot of people may not really understand or use is the

      [Read more...]
    Fun with MySQL and Hadoop at SCaLE 12X
    +0 Vote Up -0Vote Down
    It's my pleasure to be presenting at SCaLE 12X on the subject of real-time data loading from MySQL to Hadoop.  This is the first public talk on work at Continuent that enables Tungsten Replicator to move transactions from MySQL to HDFS (Hadoop Distributed File System).  I will explain how replication to Hadoop works, how to set it up, and offer a few words on constructing views of MySQL data using tools like Hive.

    As usual with replication everything we are doing on Hadoop replication is open source.  Builds and documentation will be

      [Read more...]
    MaxScale, ProxySQL and MySQL Proxy
    +2 Vote Up -0Vote Down

    At FOSDEM 2014 ProxySQL and MaxScale were both presented. Both are proxy that can help build sophisticated MariaDB/MySQL architectures. But currently what is the most used proxy with MySQL? It is HAproxy. HAproxy is a level 4 proxy that has no knowledge of the MySQL protocol. Being low level makes it very fast but it [...]

    Showing entries 1 to 30 of 628 Next 30 Older Entries

    Planet MySQL © 1995, 2014, Oracle Corporation and/or its affiliates   Legal Policies | Your Privacy Rights | Terms of Use

    Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.