Home |  MySQL Buzz |  FAQ |  Feeds |  Submit your blog feed |  Feedback |  Archive |  Aggregate feed RSS 2.0 English Deutsch Español Français Italiano 日本語 Русский Português 中文
Showing entries 1 to 19

Displaying posts with tag: mydumper (reset)

mydumper [less] locking
+1 Vote Up -0Vote Down
In this post I would like to review how my dumper for MySQL works from the point of view of locks. Since 0.6 serie we have different options, so I will try to explain how they work

As you may know mydumper is multithreaded and this adds a lot of complexity compared with other logical backup tools as it also needs to coordinate all threads with the same snapshot to be consistent. So let review how mydumper does this with the default settings.

By default mydumper uses 4 threads to dump data and 1 main thread

Main Thread
  • FLUSH TABLES WITH READ LOCK
Dump Thread X
  • START TRANSACTION WITH CONSISTENT SNAPSHOT;
  • dump non-InnoDB tables
Main Thread
  • UNLOCK TABLES
Dump Thread X
  • dump InnoDB tables
As you can see in this case we need FTWRL for two things, coordinate transaction’s snapshots and dump  [Read more...]
Introducing backup locks in Percona Server
+1 Vote Up -0Vote Down

TL;DR version: The backup locks feature introduced in Percona Server 5.6.16-64.0 is a lightweight alternative to FLUSH TABLES WITH READ LOCK and can be used to take both physical and logical backups with less downtime on busy servers. To employ the feature with mysqldump, use mysqldump --lock-for-backup --single-transaction. The next release of Percona XtraBackup will also be using backup locks automatically if the target server supports the feature.

Now on to the gory details, but let’s start with some history.

In the beginning…

In the beginning there was FLUSH TABLES, and users messed with their MyISAM tables under a live server and were not ashamed. Users could do nice things like:

mysql> FLUSH TABLES;
# execute myisamchk, myisampack, backup / restore some


  [Read more...]
New mydumper 0.6.1 release offers performance and usability features
+1 Vote Up -0Vote Down

One of the tasks within Percona Remote DBA is to ensure we have reliable backups with minimal impact. To accomplish this, one of the tools our team uses is called mydumper. We use mydumper for logical backups because of several nice features. Some of them are:

  • multithreaded, producing very fast backups compared to mysqldump
  • almost no locking, if not using non innodb tables
  • built-in compression
  • separate files for each table, making it easy to restore single tables or schema, and the possibility to hardlink files reducing the space needed to keep history of backups. Also this feature give you the possibility to restore with more than one thread.

The

  [Read more...]
how MySQL engineering broke the backups
+0 Vote Up -0Vote Down

MySQL has exceptional track of record by introducing minor fixes that cause major breakages. Though usually I could blame naiveté of engineers, who did not really ever have to deal with production implications, but lately I can start sensing various business implications against open-source offerings.

As an original author of mydumper I really cannot get out of my mind that 5.5 and 5.6 metadata locking changes are there to screw with anyone who is building a backup solution using stable snapshot views of MySQL (for example, mysqldump –single-transaction, the golden standard of backing things up in MySQL world).

As seen in a bug #71017 (palindrome!) filed by my esteemed colleague Eric, newly

  [Read more...]
Vote for MySQL[plus] awards 2011 !
+5 Vote Up -0Vote Down

First of all, I wish you a happy new year.
Many things happened last year, it was really exciting to be involved in the MySQL ecosystem.
I hope this enthusiasm will be increased this year, up to you !

To start the year, I propose the MySQL[plus] Awards 2011
It will only take 5 minutes to fill out these polls.
Answer with your heart first and then with your experience with some of these tools or services.

Polls will be closed January 31, so, vote now !
For “other” answers, please,  let me a comment with details.

Don’t hesitate to submit proposal for tools or services in the comments.






  [Read more...]
Mydumper 0.5.1 released
+0 Vote Up -0Vote Down
After much feedback from the community who have been using mydumper I have created the first bugfix release of the 0.5 series of mydumper.  Changes in this release are as follows:

  • Fix MySQL 5.0 compile issue

  • Make the metadata file visible (after muliple requests)

  • Add --no-lock option to mydumper

  • First --daemon snapshot is now at execution instead of the first timed interval

  • Fix CentOS 5.x compile issue (also affected Mac OSX)

  • Fix libmysqlclient search bug

  • Add cppcheck target

  • Fix errors flaged by cppcheck

  • Add option to turn off docs build

  • Add status output at end of CMake


To obtain this release you can download it from Launchpad.

Many thanks to everyone who has been trying it and giving feedback.
Mydumper 0.5.1 released
+1 Vote Up -0Vote Down

After much feedback from the community who have been using mydumper I have created the first bugfix release of the 0.5 series of mydumper.  Changes in this release are as follows:

  • Fix MySQL 5.0 compile issue
  • Make the metadata file visible (after muliple requests)
  • Add –no-lock option to mydumper
  • First –daemon snapshot is now at execution instead of the first timed interval
  • Fix CentOS 5.x compile issue (also affected Mac OSX)
  • Fix libmysqlclient search bug
  • Add cppcheck target
  • Fix errors flaged by cppcheck
  • Add option to turn off docs build
  • Add status output at end of CMake

To obtain this release you can download it from Launchpad.

Many thanks to everyone who has been trying it and giving

  [Read more...]
mydumper, now with continuous backup mode!
+0 Vote Up -0Vote Down
It has happened to the best of us.  Our MySQL servers are running nicely when the tea-lady in the data centre trips over a wire, knocks a cup hot of java down the UPS batteries and fries every server in the data centre.  OK, so that doesn't happen very often, but there have been cases where data centre failures have physically damaged servers.

So, if this happens with your MySQL server, what do you do?  You can setup new servers in a new data centre and hope you have an up-to-date backup.  But unless you had remote slaves these backups probably won't be as current as they could be.  When you get to the point where you have many servers keeping slaves for every one just for backups can be difficult to maintain.  Many DBAs have different backup strategies to cover this potential problem.

This is where the new mydumper



  [Read more...]
mydumper, now with continuous backup mode!
+5 Vote Up -0Vote Down

It has happened to the best of us.  Our MySQL servers are running nicely when the tea-lady in the data centre trips over a wire, knocks a cup hot of java down the UPS batteries and fries every server in the data centre.  OK, so that doesn’t happen very often, but there have been cases where data centre failures have physically damaged servers.

So, if this happens with your MySQL server, what do you do?  You can setup new servers in a new data centre and hope you have an up-to-date backup.  But unless you had remote slaves these backups probably won’t be as current as they could be.  When you get to the point where you have many servers keeping slaves for every one just for backups can be difficult to maintain.  Many DBAs have different backup strategies to cover this potential problem.

This is where the new mydumper 0.5.0 comes in.  It has a

  [Read more...]
Mydumper 0.2.3 released!
+0 Vote Up -0Vote Down
Today marks the release of mydumper 0.2.3.  Mydumper is a multi-threaded high-performance data dumper (and loader) for MySQL and Drizzle written in C. This is a minor bugfix release whilst I work on the upcoming 0.5 version. The end goal here will be that there is always a 'stable' and 'development' version. 0.2 will be the first stable version and will only have bug fixes. 0.5 will be the first 'development' version where the next planned set of new features will hit.

So, the changes since 0.2.2 are:

  • Drizzle support now fully works again

  • Fixes so mydumper compiles in FreeBSD (thanks to Kirill A. Korinskiy)


If you wish to try this version the source can be downloaded here.
    Mydumper 0.2.3 released!
    +0 Vote Up -0Vote Down

    Today marks the release of mydumper 0.2.3.  Mydumper is a multi-threaded high-performance data dumper (and loader) for MySQL and Drizzle written in C. This is a minor bugfix release whilst I work on the upcoming 0.5 version. The end goal here will be that there is always a ‘stable’ and ‘development’ version. 0.2 will be the first stable version and will only have bug fixes. 0.5 will be the first ‘development’ version where the next planned set of new features will hit.

    So, the changes since 0.2.2 are:

    • Drizzle support now fully works again
    • Fixes so mydumper compiles in FreeBSD (thanks to Kirill A. Korinskiy)

    If you wish to try this version the source can be downloaded

      [Read more...]
    Mydumper now with MyISAM consistent snapshots!
    +0 Vote Up -0Vote Down
    Mydumper 0.2.2 has been released today with a number of fixes and new features.  The one that most people have been asking for is consistent snapshots for non-InnoDB tables (such as MyISAM).  We have been able to achieve this without locking the database for the entire backup using the following method:

  • Flush tables with read lock (and start transaction with consistent snapshot on all threads)

  • Dump non-InnoDB

  • Start InnoDB dump

  • When non-InnoDB dump has finished (whilst InnoDB is dumping) unlock tables





  •   [Read more...]
    Mydumper now with MyISAM consistent snapshots!
    +4 Vote Up -0Vote Down

    Mydumper 0.2.2 has been released today with a number of fixes and new features.  The one that most people have been asking for is consistent snapshots for non-InnoDB tables (such as MyISAM).  We have been able to achieve this without locking the database for the entire backup using the following method:

  • Flush tables with read lock (and start transaction with consistent snapshot on all threads)
  • Dump non-InnoDB
  • Start InnoDB dump
  • When non-InnoDB dump has finished (whilst InnoDB is dumping) unlock tables
  • Profit
  • I have also started work on Drizzle support this week.  The Drizzle support is not entirely complete

      [Read more...]
    Mydumper now with myloader!
    +0 Vote Up -0Vote Down
    It has only been a few days since the 0.2.0 release of mydumper but there have been some big changes since then.  I will try and go over them all here.

    Mydumper 0.2.1


    Mydumper 0.2.1 has been released today, many thanks for all those who have been testing the trunk source, the feedback has gone a long way to making fixes and improvements to mydumper.  You can download the source for it here.

    Myloader


    A few days ago Mark Callaghan asked about restoring mydumper backups. 



      [Read more...]
    Mydumper now with myloader!
    +6 Vote Up -0Vote Down

    It has only been a few days since the 0.2.0 release of mydumper but there have been some big changes since then.  I will try and go over them all here.

    Mydumper 0.2.1

    Mydumper 0.2.1 has been released today, many thanks for all those who have been testing the trunk source, the feedback has gone a long way to making fixes and improvements to mydumper.  You can download the source for it here.

    Myloader

    A few days ago Mark Callaghan asked about restoring mydumper backups.  There is of course the great

      [Read more...]
    MySQL Data Dumper 0.2.0 released!
    +0 Vote Up -0Vote Down


    A couple of years ago Domas Mituzas created a tool which could basically be thought of as basically a lightweight multi-threaded mysqldump.  By this I mean it can retrieve data from multiple tables simultaneously and can even break a table down into parts for simultaneous retrieval.  Sometime around 2010 I started hacking on mydumper too but stopped whilst working at Rackspace (Drizzle was way more than full-time for me).

    Back when Domas first blogged about it he was managing to dump his sample data over 10x faster than mysqldump!

    Since the 0.1 series Domas has fixed a lot of problems and since starting





      [Read more...]
    MySQL Data Dumper 0.2.0 released!
    +5 Vote Up -0Vote Down

    Photo by Pockafwye under a CC by NC SA 2.0 license

    A couple of years ago Domas Mituzas created a tool which could basically be thought of as basically a lightweight multi-threaded mysqldump.  By this I mean it can retrieve data from multiple tables simultaneously and can even break a table down into parts for simultaneous retrieval.  Sometime around 2010 I started hacking on mydumper too but stopped whilst working at Rackspace (Drizzle was way more than full-time for me).

    Back when Domas first blogged about it he was managing to dump his sample data over 10x faster than mysqldump!

    Since the 0.1 series Domas has fixed a lot of problems and since starting at

      [Read more...]
    SkySQL – The Return of the Jedi
    +0 Vote Up -0Vote Down
    I apologise to those reading Planet MySQL who will see this twice, this is aimed at a different audience to my personal blog.

    Rackspace and Drizzle

    If you have read my last ‘Last Week in Drizzle‘ post you will know that Rackspace are no longer supporting Drizzle. They have done a fantastic job so far and have decided to pass the baton to other companies. As for the staff, they wished to redeploy us to other teams which is something I personally was not keen on. I would rather remain within the MySQL/Drizzle sphere which I would have no longer been able to do effectively inside Rackspace any


      [Read more...]
    after the conference, mydumper, parallelism, etc
    +5 Vote Up -0Vote Down

    Though slides for my MySQL Conference talks were on the O’Reilly website, I placed them in my talks page too, for both dtrace and security presentations.

    I also gave a lightning talk about mydumper. Since my original announcement mydumper has changed a bit. It supports writing compressed files, detecting and killing slow queries that could block table flushes, supports regular expressions for table names, and trunk is slowly moving towards understanding that storage engines differ :)

    I’ve been using mydumper quite a lot in my deployments (and observing 10x faster dumps). Now, the sad part is how to do

      [Read more...]
    Showing entries 1 to 19

    Planet MySQL © 1995, 2014, Oracle Corporation and/or its affiliates   Legal Policies | Your Privacy Rights | Terms of Use

    Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.