Home |  MySQL Buzz |  FAQ |  Feeds |  Submit your blog feed |  Feedback |  Archive |  Aggregate feed RSS 2.0 English Deutsch Español Français Italiano 日本語 Русский Português 中文
Showing entries 1 to 30 of 255 Next 30 Older Entries

Displaying posts with tag: TokuDB (reset)

When (and how) to move an InnoDB table outside the shared tablespace
+0 Vote Up -0Vote Down

In my last post, “A closer look at the MySQL ibdata1 disk space issue and big tables,” I looked at the growing ibdata1 problem under the perspective of having big tables residing inside the so-called shared tablespace. In the particular case that motivated that post, we had a customer running out of disk space in his server who was looking for a way to make the ibdata1 file shrink. As you may know, that file (or, as explained there, the set of ibdata files composing the shared tablespace) stores all InnoDB tables created when innodb_file_per_table is disabled, but also other InnoDB structures, such as undo logs and data dictionary.

For example, when you run a transaction involving

  [Read more...]
An Updated Description of Clustering Keys for TokuDB
+0 Vote Up -0Vote Down

Covering indexes can result in orders of magnitude performance improvements for queries. Bradley’s presentation on covering indexes describes what a covering index is, how it can effect performance, and why it works. However, the definition of a covering index can get cumbersome since MySQL limits the number of columns in a key to 16 (32 on MariaDB).

Tokutek introduced multiple clustering indexes into MySQL to address these problems. Zardosht describes the multiple clustering indexes feature and how clustering indexes differ from covering indexes. Zardosht

  [Read more...]
Troubleshooting TokuDB Corruption
+1 Vote Up -0Vote Down

I recently ran across some TokuDB table corruption, which was not easily identifiable at first, and the error log entry was not too verbose either, so I wanted to share that experience here.

Basically, TokuDB crashed, and then mysqld had problems restarting afterward. Just for reference, the error log had the following in the stack trace:

/usr/lib64/mysql/plugin/ha_tokudb.so(+0x71c48)[0x7fb25be75c48]
/usr/lib64/mysql/plugin/ha_tokudb.so(+0x71cbd)[0x7fb25be75cbd]
/usr/lib64/mysql/plugin/ha_tokudb.so(_Z29toku_deserialize_bp_from_disk...
/usr/lib64/mysql/plugin/ha_tokudb.so(_Z23toku_ftnode_pf_callback...
/usr/lib64/mysql/plugin/ha_tokudb.so(_Z30toku_cachetable_pf_pinned_pair..
/usr/lib64/mysql/plugin/ha_tokudb.so(_Z24toku_ft_flush_some_child...
/usr/lib64/mysql/plugin/ha_tokudb.so(_Z28toku_ftnode_cleaner_callback...
  [Read more...]
Troubleshooting TokuDB ERROR 1126 – API Version Mismatch or bitmap_free
+1 Vote Up -0Vote Down

When setting up TokuDB, you may encounter error 1126.

I’ve seen 2 recent invocations of it, so I wanted to share them both here in case you run into this issue:

MariaDB [(none)]> install soname 'ha_tokudb';
ERROR 1126 (HY000): Can't open shared library
'/usr/lib/mysql/plugin/ha_tokudb.so' (errno: 2, undefined symbol: bitmap_free)
MariaDB [(none)]> install soname 'ha_tokudb';
ERROR 1126 (HY000): Can't open shared library 'ha_tokudb.so'
(errno: 8, API version for STORAGE ENGINE plugin TokuDB not
supported by this version of the server)

The latter is a bit more descriptive, but the former is fairly cryptic.

Given the latter, as you may have guessed it, if you run into either of these, you have the wrong version of ha_tokudb.so in your plugin directory – that is, it is not the correct version

  [Read more...]
Disabling Transparent Hugepages for TokuDB
+0 Vote Up -0Vote Down

If you want to use TokuDB with MariaDB, MySQL, or Percona Server, you will need to disable support for transparent hugepages in Linux.

Fortunately, this is very easy to check, and to change.

An easy way to check is with:

cat /sys/kernel/mm/transparent_hugepage/enabled

This will return something like:

[always] madvise never

Note the word surrounded by “[]” is what this option is set to. So the above is set to “always”. To disable it, we want it set to “never”.

I’ve found the easiest way to change/set this is to add the below to your /etc/rc.local file (and then reboot your system):

if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
   echo never >
  [Read more...]
Enabling TokuDB in MariaDB is a Breeze
+0 Vote Up -0Vote Down

TokuDB is gaining more and more popularity and many people are finding it very helpful for certain cases.

Using/enabling it in some distributions can be quite a pain, but enabling it in MariaDB is easy as 1-2-…, well, only 1-2, since that’s all there is to it!

1. Ensure you have the correct MariaDB version:

  • MariaDB 5.5.36+
  • MariaDB 10.0.9+
  • Note: Linux 64-bit systems only – specific packages include: Ubuntu, Debian, Fedora, CentOS, Red Hat
  • Note: If using the Linux tarball – it must be the version built with glibc 2.14+

2. Run this command:

INSTALL SONAME 'ha_tokudb';

or update my.cnf file with:

[mysqld]
plugin-load=ha_tokudb

There is one requirement from

  [Read more...]
Examining the TokuDB MySQL storage engine file structure
+1 Vote Up -0Vote Down

As we know different storage engines in MySQL have different file structures. Every table in MySQL 5.6 must have a .frm file in the database directory matching the table name. But where the rest of the data resides depends on the storage engine.

For MyISAM we have .MYI and .MYD files in the database directory (unless special settings are in place); for InnoDB we might have data stored in the single table space (typically ibdata1 in the database directory) or as file per table (or better said file per partition) producing a single file with .ibd extension for each table/partition. TokuDB as of this version (7.1.7) has its own innovative approach to storing the table contents.

I have created the table in the database test having the following

  [Read more...]
Why TokuDB hates Transparent HugePages
+0 Vote Up -0Vote Down

If you try to install the TokuDB storage engine on a modern Linux distribution it might fail with following error message:

2014-07-17 19:02:55 13865 [ERROR] TokuDB will not run with transparent huge pages enabled.
2014-07-17 19:02:55 13865 [ERROR] Please disable them to continue.
2014-07-17 19:02:55 13865 [ERROR] (echo never > /sys/kernel/mm/transparent_hugepage/enabled)

You might be curious why TokuDB refuses to start with Transparent HugePages. Are they not a good thing… allowing smaller kernel page tables and less TLB misses when accessing data in the buffer pool? I was curious, so I asked Tim Callaghan this very question.

This problem originates with TokuDB using jemalloc memory allocator, which uses a particular trick to deal with memory fragmentation. The classical problem with memory



  [Read more...]
Reference architecture for a write-intensive MySQL deployment
+0 Vote Up -0Vote Down

We designed Percona Cloud Tools (both hardware and software setup) to handle a very high-intensive MySQL write workload. For example, we already observe inserts of 1bln+ datapoints per day. So I wanted to share what kind of hardware we use to achieve this result.

Let me describe what we use, and later I will explain why.

Server:

  • Chassis: Supermicro SC825TQ-R740LPB 2U Rackmount Chassis
  • Motherboard: Supermicro X9DRI-F dual socket
  • CPU: Dual Intel Xeon Ivy Bridge E5-2643v2 (6x 3.5Ghz cores, 12x HT cores, 25M L3)
  • Memory: 256GB (16x 16GB 256-bit quad-channel) ECC registered DDR3-1600
  • Raid: LSI MegaRAID 9260-4i 4-port 6G/s hardware RAID controller, 512M buffer
  • MainStorage: PCIe SSD HGST FlashMAX II 4.8TB
  • Secondary
  [Read more...]
TokuDB tips: MySQL backups
+0 Vote Up -0Vote Down

In my recent post, “TokuDB gotchas: slow INFORMATION_SCHEMA TABLES,” I saw a couple questions and tweets asking if we use TokuDB in production. Actually I mentioned it in that post and we also blogged about it in a couple of other recent posts:

So, yes, we are using Percona Server + TokuDB as a main storage engine in Percona Cloud Tools to store timeseries data.

And, yes, Percona

  [Read more...]
TokuDB gotchas: slow INFORMATION_SCHEMA TABLES
+0 Vote Up -0Vote Down

We are using Percona Server + TokuDB engine extensively in Percona Cloud Tools and getting real usage operational experience with this engine. So I want to share some findings we came across, in hope it may help someone in their work with TokuDB.

So, one problem I faced is that SELECT * FROM INFORMATION_SCHEMA.TABLES is quite slow when I have thousands tables in TokuDB. How slow? For example…

select * from information_schema.tables limit 1000;
...
1000 rows in set (18 min 31.93 sec)

This is very similar to what InnoDB faced a couple years back. InnoDB solved it by adding

  [Read more...]
Percona Server 5.6.19-67.0 with TokuDB (GA) now available
+1 Vote Up -0Vote Down

Percona is glad to announce the release of Percona Server 5.6.19-67.0 on July 1, 2014. Download the latest version from the Percona web site or from the Percona Software Repositories.

Based on MySQL 5.6.19, including all the bug fixes in it, Percona Server

  [Read more...]
Percona Server with TokuDB (beta): Installation, configuration
+1 Vote Up -0Vote Down

My previous post was an introduction to the TokuDB storage engine and aimed at explaining the basics of its design and how it differentiates from InnoDB/XtraDB. This post is all about motivating you to give it a try and have a look for yourself. Percona Server is not officially supporting TokuDB as of today, though the guys in the development team are working hard on this and the first GA release of Percona Server with TokuDB is looming on the horizon. However, there’s a beta version available now. For the installation tests in this post I’ve used the latest version of

  [Read more...]
Getting to know TokuDB for MySQL
+0 Vote Up -0Vote Down

During last April’s Percona Live MySQL Conference and Expo, TokuDB celebrated it’s first full-year as an open source storage engine. I still remember reading the official announcement and the expectations it created one year ago. The premises were very interesting as it had the potential of helping MySQL manage “big data” in a way InnoDB just couldn’t. It also provided additional interesting features like “hot schema changes,” all the while making our dear flash storages last longer.

While I’ve kept an eye on the evolution of TokuDB this past year, I reckon I haven’t given

  [Read more...]
Best Practices for Partitioned Collections and Tables in TokuDB and TokuMX
+0 Vote Up -1Vote Down

In my last post, I gave a technical explanation of the performance characteristics of partitioned collections in TokuMX 1.5 (which is right around the corner) and partitioned tables in relational databases. Given those performance characteristics, in this post, I will present some best practices when using this feature in TokuMX or TokuDB. Note that these best practices are designed for TokuMX and TokuDB only, which

  [Read more...]
Percona Server 5.6.17-66.0 is now available
+0 Vote Up -0Vote Down

 

Percona is glad to announce the release of Percona Server 5.6.17-66.0 on June 11, 2014. Downloads are available here and from the Percona Software Repositories.

Based on MySQL 5.6.17, including all the bug fixes in it, Percona Server 5.6.17-66.0 is the current GA

  [Read more...]
Understanding the Performance Characteristics of Partitioned Collections
+0 Vote Up -0Vote Down

In TokuMX 1.5 that is right around the corner, the big feature will be partitioned collections. This feature is similar to partitioned tables in Oracle, MySQL, SQL Server, and Postgres. A question many have is “why should I use partitioned tables?” In short, it’s complicated. The answer depends on your workload, your schema, and your database of choice. For example, this Oracle related post states “Anyone with un-partitioned databases over 500 gigabytes is courting disaster.” That’s not true for TokuDB or TokuMX. Nevertheless,

  [Read more...]
MariaDB 10.0.11 Overview and Highlights
+5 Vote Up -0Vote Down

MariaDB 10.0.11 was recently released, and is available for download here:

https://downloads.mariadb.org/mariadb/10.0.11/

This is the second GA release of MariaDB 10.0, and 12th overall release of MariaDB 10.0.

This is primarily a bug-fix release.

Here are the main items of note:

  • Updated TokuDB engine to version 7.1.6
  • Updated Spider storage engine to version 3.2 (now Gamma)
  • Updated XtraDB storage engine to version 5.6.17-65.0
  • Updated InnoDB storage engine to version 5.6.17
  • Updated
  •   [Read more...]
    Thoughts on Small Datum – Part 3
    +0 Vote Up -0Vote Down

    Background: If you did not read my first blog post about why I am sharing my thoughts on the benchmarks published by Mark Callaghan on Small Datum you may want to skim through it now for a little context: Thoughts on Small Datum – Part 1”

    ~~~~~~~~~~~~~~~~~~~~~~~~

    Last time, in Thoughts on Small Datum – Part 2 I shared my cliff notes and a graph on Mark Callaghan’s (@markcallaghan) March 11th insertion rate benchmarks using flash storage media. In those tests he compares MySQL (http://www.mysql.com/) outfitted with the

      [Read more...]
    Maybe You Should Try Taking a Walk in My Shoes
    +0 Vote Up -0Vote Down

    The title of this post should really be, “Maybe He Should Try Taking a Walk in Your Shoes.”

    The he I’m referring to is economist and author, Tim Harford. The you is the people who use NewSQL and NoSQL approaches to mine big data with database platforms like MySQL (http://www.mysql.com" target="_blank) and MongoDB (or, preferably, our high-performance distributions of them, TokuDB and TokuMX).

    Why should Mr. Harford take that walk? Well, he recently

      [Read more...]
    Thoughts on Small Datum – Part 2
    +0 Vote Up -0Vote Down

    If you did not read my first blog post about Mark Callaghan’s (@markcallaghan) benchmarks as documented in his blog, Small Datum, you may want to skim through it now for a little context.

    ——————-

    On March 11th, Mark, a former Google and now Facebook database guru, published an insertion rate benchmark comparing MySQL (http://www.mysql.com) outfitted with the InnoDB storage engine with two NoSQL alternatives — basic MongoDB and TokuMX (the Tokutek high-performance

      [Read more...]
    Percona Live 2014 Impressions
    +0 Vote Up -0Vote Down

    Three weeks ago I had the privilege of attending my first Percona Live MySQL conference, which was incredible! In particular, there were two things that I found impressive about the conference.

    First, was the amount of knowledge sharing and support that MySQL users provide each other; it truly is a community. Coming from EMC, I’ve attended several conferences in the past, but I’ve always considered them more of a marketing focused event, mostly spent doing product launches and company roadmaps and not much time fostering knowledge sharing and informal get-togethers: Percona Live was different. There were well thought out tutorials, information packed presentations, and keynotes rife with practical knowledge culled from the real world. I had many great conversations at our booth with people that have evaluated TokuDB or TokuMX or were planning to as soon as they got back into

      [Read more...]
    Thoughts on Small Datum – Part 1
    +0 Vote Up -0Vote Down

    A little background…

    When I ventured into sales and marketing (I’m an engineer by education) I learned I would often have to interpret and simply summarize the business value that is sometimes hidden in benchmarks. Simply put, the people who approve the purchase of products like TokuDB® and TokuMX™ appreciate the executive summary.

    Therefore, I plan to publish a multipart series here on TokuView where I will share my simple summaries and thoughts on business value for the benchmarks Mark Callaghan (@markcallaghan), a former Google and now Facebook database guru, is publishing on his blog, Small Datum.

    I’m going to start with his first benchmark post and work my way forward to

      [Read more...]
    How Tokutek uses the Random Query Generator framework to test TokuDB
    +0 Vote Up -0Vote Down

    During a typical release cycle for TokuDB at Tokutek, we spend time qualifying and hardening the product using numerous tools.  For example, we run stress and unit tests directly on the Fractal Tree indexes, MySQL Test Runner (MTR) tests on the storage engine as well as numerous performance benchmarks to prevent regressions. In addition, we have recently been implementing the Random Query Generator (RQG) framework internally here at Tokutek to more exhaustively stress TokuDB.  My name is Joel Epstein and I am a Quality Assurance Engineer here at Tokutek who has been integrating RQG into the overall test plan strategy.

    At

      [Read more...]
    MariaDB 10.0.10 Overview and Highlights
    +2 Vote Up -0Vote Down

    MariaDB 10.0.10 was recently released, and is available for download here:

    https://downloads.mariadb.org/mariadb/10.0.10/

    This is the first GA ("Generally Availability", aka "recommended for production systems") release of MariaDB 10.0, and 11th overall release of MariaDB 10.0.

    Since this is the initial 10.0 GA release, this is primarily a bug-fix and polishing release.

    Here are the main items of note:

    read more

    Why TokuDB does not use the ‘uint3korr’ function
    +0 Vote Up -0Vote Down

    The ‘uint3korr’ function inside of the mysqld server extracts a 3 byte unsigned integer from a memory buffer. One use is for ‘mediumint’ columns which encode their value in 3 bytes. MySQL 5.6 and MariaDB 10.0 claims to have optimized this function for x86 and x86_64 processors. There is a big comment that says:

    Attention: Please, note, uint3korr reads 4 bytes (not 3)!
    It means, that you have to provide enough allocated space.

    The ‘uint3korr’ optimization may be fast, but it is not valgrind safe. Here is an example where valgrind detects TokuDB reading beyond the end of a buffer when it uses the ‘uint3korr’ function.

    ==3899== Thread 36:
    ==3899== Invalid read of size 4
    ==3899== at 0xB76C089: tokudb_uint3korr(unsigned char const*) (hatoku_defines.h:533)
    ==3899== by 0xB795C5E:
      [Read more...]
    Percona Live 2014 Recap
    +0 Vote Up -0Vote Down

    The MySQL community continues to amaze me, everyone is friendly and always willing to answer questions and help others. It’s always nice to meet people in-person that I’ve met virtually in the last 12 months. Percona Live 2014 MySQL Conference and Expo in Santa Clara just wrapped up and I wanted to share my highlights of the event before I unplug for the weekend.

    • I heard several stories of people’s experiences with TokuDB, open sourcing 12 months ago has been a huge success for Tokutek. We have a growing community in our tokudb-user Google Group, please share your experiences and help others.
    • In his “Galera Cluster New Features” presentation, Sappo Jaakola of
      [Read more...]
    Uninitialized data in the TokuDB recovery log
    +0 Vote Up -0Vote Down

    A TokuDB MySQL test run with valgrind reported an uninitialized data error when writing into the TokuDB recovery log.

    ==1032== Syscall param write(buf) points to uninitialised byte(s)
    ==1032== at 0x3EFA60E4ED: ??? (in /lib64/libpthread-2.12.so)
    ==1032== by 0xB894038: toku_os_full_write(int, void const*, unsigned long) (file.cc:249)
    ==1032== by 0xB83248A: write_outbuf_to_logfile(tokulogger*, __toku_lsn*) (logger.cc:513)
    ==1032== by 0xB83326C: toku_logger_maybe_fsync(tokulogger*, __toku_lsn, int, bool) (logger.cc:836)
    ==1032== by 0xB8327DE: toku_logger_fsync_if_lsn_not_fsynced(tokulogger*, __toku_lsn) (logger.cc:586)
    ==1032== by 0xB8493E6: toku_txn_maybe_fsync_log(tokulogger*, __toku_lsn, bool) (txn.cc:600)
    ==1032== by 0xB7B4EBB: toku_txn_commit(__toku_db_txn*, unsigned int, void (*)(__toku_txn_progress*, void*), void*, bool, bool) (ydb_txn.cc:198)
    ==1032==
      [Read more...]
    Lock Escalation and Big Transactions in TokuDB and TokuMX
    +0 Vote Up -0Vote Down

    We have seen TokuDB lock escalation stall the execution of SQL operations for tens of seconds. To address this problem, we changed the lock escalation algorithm used by TokuDB and TokuMX so that the cost of lock escalation only affects big transactions. We also eliminated a serialization point when running lock escalation.

    Transactions in TokuDB and TokuMX accumulate locks on key ranges while they execute. These locks allow multiple transactions to run concurrently. The locks are released when the transaction commits or aborts.

    The locks are stored in an in memory data structure that contains a set of key range and transaction identifier pairs. Since the locks are stored in memory and we want to support arbitrarily large transactions, an algorithm is needed to kick in when the amount of memory used to store locks exceeds a maximum limit. The

      [Read more...]
    Percona Server with TokuDB: Packing 15TB into local SSDs
    +0 Vote Up -0Vote Down

    Two weeks ago we released an Alpha release of Percona Server with TokuDB. Right now I am on a final stage of evaluation of TokuDB for using in our project Percona Cloud Tools and it looks promising.

    What is the most attractive in TokuDB? For me it is compression, but not just compression: TokuDB provides great performance over compressed data.

    In my synthetic tests I saw a compression ratio of 10:1 (TokuDB LZMA to InnoDB uncompressed), in the real production data it is less, 6:1, but still impressive.

    In our servers we have 4 x SSD Crucial M500 960GB combined in RAID5, which give 2877.0 GB of usable space. With TokuDB we should be


      [Read more...]
    Showing entries 1 to 30 of 255 Next 30 Older Entries

    Planet MySQL © 1995, 2014, Oracle Corporation and/or its affiliates   Legal Policies | Your Privacy Rights | Terms of Use

    Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.