Home |  MySQL Buzz |  FAQ |  Feeds |  Submit your blog feed |  Feedback |  Archive |  Aggregate feed RSS 2.0 English Deutsch Español Français Italiano 日本語 Русский Português 中文
Showing entries 1 to 30 of 272 Next 30 Older Entries

Displaying posts with tag: TokuView (reset)

How TokuMX Secondaries Work in Replication
+0 Vote Up -0Vote Down

As I’ve mentioned in previous posts, TokuMX replication differs quite a bit from MongoDB’s replication. The differences are large enough such that we’ve completely redone some of MongoDB’s existing algorithms. One such area is how secondaries apply oplog data from a primary. In this post, I’ll explain how.

In designing how secondaries apply oplog data, we did not look closely at how MongoDB does it. In fact, I’ve currently forgotten all I’ve learned about MongoDB’s implementation, so I am not in a position to compare the two. I think I recall that MongoDB’s oplog idempotency was a key to their

  [Read more...]
How Tokutek uses the Random Query Generator framework to test TokuDB
+0 Vote Up -0Vote Down

During a typical release cycle for TokuDB at Tokutek, we spend time qualifying and hardening the product using numerous tools.  For example, we run stress and unit tests directly on the Fractal Tree indexes, MySQL Test Runner (MTR) tests on the storage engine as well as numerous performance benchmarks to prevent regressions. In addition, we have recently been implementing the Random Query Generator (RQG) framework internally here at Tokutek to more exhaustively stress TokuDB.  My name is Joel Epstein and I am a Quality Assurance Engineer here at Tokutek who has been integrating RQG into the overall test plan strategy.

At

  [Read more...]
Why TokuDB does not use the ‘uint3korr’ function
+0 Vote Up -0Vote Down

The ‘uint3korr’ function inside of the mysqld server extracts a 3 byte unsigned integer from a memory buffer. One use is for ‘mediumint’ columns which encode their value in 3 bytes. MySQL 5.6 and MariaDB 10.0 claims to have optimized this function for x86 and x86_64 processors. There is a big comment that says:

Attention: Please, note, uint3korr reads 4 bytes (not 3)!
It means, that you have to provide enough allocated space.

The ‘uint3korr’ optimization may be fast, but it is not valgrind safe. Here is an example where valgrind detects TokuDB reading beyond the end of a buffer when it uses the ‘uint3korr’ function.

==3899== Thread 36:
==3899== Invalid read of size 4
==3899== at 0xB76C089: tokudb_uint3korr(unsigned char const*) (hatoku_defines.h:533)
==3899== by 0xB795C5E:
  [Read more...]
Percona Live 2014 Recap
+0 Vote Up -0Vote Down

The MySQL community continues to amaze me, everyone is friendly and always willing to answer questions and help others. It’s always nice to meet people in-person that I’ve met virtually in the last 12 months. Percona Live 2014 MySQL Conference and Expo in Santa Clara just wrapped up and I wanted to share my highlights of the event before I unplug for the weekend.

  • I heard several stories of people’s experiences with TokuDB, open sourcing 12 months ago has been a huge success for Tokutek. We have a growing community in our tokudb-user Google Group, please share your experiences and help others.
  • In his “Galera Cluster New Features” presentation, Sappo Jaakola of
  [Read more...]
Uninitialized data in the TokuDB recovery log
+0 Vote Up -0Vote Down

A TokuDB MySQL test run with valgrind reported an uninitialized data error when writing into the TokuDB recovery log.

==1032== Syscall param write(buf) points to uninitialised byte(s)
==1032== at 0x3EFA60E4ED: ??? (in /lib64/libpthread-2.12.so)
==1032== by 0xB894038: toku_os_full_write(int, void const*, unsigned long) (file.cc:249)
==1032== by 0xB83248A: write_outbuf_to_logfile(tokulogger*, __toku_lsn*) (logger.cc:513)
==1032== by 0xB83326C: toku_logger_maybe_fsync(tokulogger*, __toku_lsn, int, bool) (logger.cc:836)
==1032== by 0xB8327DE: toku_logger_fsync_if_lsn_not_fsynced(tokulogger*, __toku_lsn) (logger.cc:586)
==1032== by 0xB8493E6: toku_txn_maybe_fsync_log(tokulogger*, __toku_lsn, bool) (txn.cc:600)
==1032== by 0xB7B4EBB: toku_txn_commit(__toku_db_txn*, unsigned int, void (*)(__toku_txn_progress*, void*), void*, bool, bool) (ydb_txn.cc:198)
==1032==
  [Read more...]
Lock Escalation and Big Transactions in TokuDB and TokuMX
+0 Vote Up -0Vote Down

We have seen TokuDB lock escalation stall the execution of SQL operations for tens of seconds. To address this problem, we changed the lock escalation algorithm used by TokuDB and TokuMX so that the cost of lock escalation only affects big transactions. We also eliminated a serialization point when running lock escalation.

Transactions in TokuDB and TokuMX accumulate locks on key ranges while they execute. These locks allow multiple transactions to run concurrently. The locks are released when the transaction commits or aborts.

The locks are stored in an in memory data structure that contains a set of key range and transaction identifier pairs. Since the locks are stored in memory and we want to support arbitrarily large transactions, an algorithm is needed to kick in when the amount of memory used to store locks exceeds a maximum limit. The

  [Read more...]
My Favorite MongoDB Replication Feature: Crash Safety
+0 Vote Up -0Vote Down

At an extremely high level, replication in MongoDB and MySQL are similar. Both databases have exactly one machine, the primary (or master), that accepts writes from clients. With a single transaction (or atomic operation, in MongoDB’s case), the tables and oplog (or binary log in MySQL) are modified to reflect the change. The log captures what the change is so other secondaries (or slaves) can read the changes and process them, making the slaves identical to the master. (Note that I am NOT talking about multi-master replication.)

Underneath the covers, their implementations are quite different. And in peeking underneath the covers while developing TokuMX, I learned

  [Read more...]
Big trouble with zero-length character columns in TokuDB
+0 Vote Up -0Vote Down

What good is a zero-length character column in a MySQL table? A zero-length character column has type of ‘char(0)’. If it is nullable, then it can at least store one bit. If it is not nullable, then the value for this column in all rows is a null string. IMO, not very useful. However, the MySQL Reference Manual says that there are valid uses for such a column, so TokuDB should support it. Unfortunately, we recently found and fixed a bug related to zero length character columns in TokuDB.

A Random Query Generator (RQG) trial generated a table with a ‘char(0)’ column and caused TokuDB to crash when executing an alter

  [Read more...]
March 20 Webinar: How to Scale MySQL for Big Data Applications
+0 Vote Up -0Vote Down

You may think that you have to buy, install, and get up to speed on a new database if you want to work with large amounts of data, but you can do more than you think with the MySQL you already have.
Register Now!

SPEAKER: Jon Tobin, Tokutek
DATE: Thursday, March 20th
TIME: 1pm ET

Without having to change your application or do special tuning you can increase performance and save significant time and money when you need to scale.

Join Tokutek’s Jon Tobin as he demonstrates how to use MySQL or MariaDB in Big Data applications by simply upgrading the storage engine with TokuDB, and how to effectively evaluate TokuDB for increased performance, compression and agility.

During this webinar you will learn:

  • How to dramatically increase performance without having to rewrite



  [Read more...]
Announcing TokuDB v7.1.5
+0 Vote Up -0Vote Down

Yesterday we released TokuDB v7.1.5, which includes the following important features and fixes:

  • Upgraded MySQL and MariaDB to version 5.5.36.
  • Six months of performance improvements to our underlying Fractal Tree indexing.
  • Fixes for bugs, stalls, and behavioral issues reported by users.
  • Fixes for issues identified by the addition of Random Query Generator (RQG) testing.
  • Fixes for issues identified by the addition of Valgrind testing.
  • Full details on the changes in TokuDB v7.1.5 can be found in the release notes section of the TokuDB User’s Guide, available from our documentation page.

As always, you can download the Community and Enterprise

  [Read more...]
Tokutek and Percona Live 2014
+1 Vote Up -0Vote Down

I’ve been a little behind in recent blogging efforts, and realized that in less than a month we’ll be back at Percona Live: MySQL Conference and Expo 2014, aka PLMCE. Last year’s PLMCE was my first, as well as the event where Tokutek announced the open sourcing of TokuDB.

It’s hard to believe that a year has gone by, but the customer adoption in both enterprise and community users has been awesome. TokuDB is available from our website for both MySQL and MariaDB, and is also

  [Read more...]
How TokuMX was Born
+2 Vote Up -0Vote Down

With TokuMX 1.4 coming out soon, with (teaser) wonderful improvements made to sharding and updates (and plenty of other goodies), I’ve recently reminisced about how we got TokuMX to this point. We (actually, really John) started dabbling with integrating Fractal Tree® indexes into MongoDB in the summer of 2012, where we (really, he) prototyped using Fractal Tree indexes only for secondary indexes. As cool as that prototype was, it

  [Read more...]
The Effects of Database Heap Storage Choices in MongoDB
+0 Vote Up -0Vote Down

William Zola over at MongoDB gave a great talk called “The (Only) Three Reasons for Slow MongoDB Performance”. It reminded me of an interesting characteristic of updates in MongoDB. Because MongoDB’s main data store is a flat file and secondary indexes store offsets into the flat file (as I explain here), if the location of a document changes, corresponding entries in secondary indexes must also change. So, an update to an unindexed field that causes the document to move also causes modifications to every secondary index, which, as William points out, can be expensive. If a document has indexed an array, this

  [Read more...]
January 28 Webinar: Get More Out of MySQL with TokuDB
+0 Vote Up -0Vote Down

You love MySQL and MariaDB for its ease of deployment, but what if you could increase performance and save significant time and money when your application starts to scale without having to change your applications?
Register Now!

SPEAKER: Tim Callaghan, VP of Engineering at Tokutek
DATE: Tuesday, January 28th
TIME: 1pm ET

Join this interactive webinar with Tokutek’s VP of Engineering, Tim Callaghan, as he walks through the potential pitfalls when using MySQL or MariaDB for Big Data applications, and how to effectively use TokuDB to increase performance, reduce database size and achieve true schema agility.

Attend this webinar to learn:

  • How easy it is to install and configure TokuDB with MySQL or MariaDB
  • How to dramatically increase performance without having to rewrite code



  •   [Read more...]
    December 17 Webinar: Use Your MySQL Knowledge to Become a MongoDB Guru
    +0 Vote Up -0Vote Down

    Use your MySQL expertise to analyze the strengths and weaknesses of MongoDB.

    SPEAKER: Tim Callaghan, VP of Engineering at Tokutek
    DATE: Tuesday, December 17th
    TIME: 1pm ET
    Register Now!

    MongoDB is a popular NoSQL DBMS that shares the ease-of-use and quick setup that made MySQL famous. But is MongoDB really up to the job? Is it right for your applications? If you understand MySQL well, you know how database systems work.

    Join Tim Callaghan, VP/Engineering at Tokutek as he recaps his and CEO of Continuent, Robert Hodges, session from 2013′s Percona Live London. Learn how to lean on your knowledge of topics like schema design, query optimization, indexing, sharding, and high availability to analyze the strengths and weaknesses of MongoDB. System design is all about asking the right




      [Read more...]
    What does the ‘Incorrect key file for table’ error mean?
    +0 Vote Up -0Vote Down

    What does it mean if MySQL returns the ‘Incorrect key file for table’ error for one of my queries? The answer is complicated and depends on which storage engine is returning the error. We have debugged two cases which we describe here.

    File system out of space

    When running the random query generator, one of the queries failed.

    Query: SELECT * FROM (mysql . general_log AS table1 INNER JOIN INFORMATION_SCHEMA . INNODB_BUFFER_PAGE AS table2 
    ON ( table2 . SPACE = table1 . user_host ) ) ORDER BY table1 . thread_id LIMIT 168 
    failed: 126 Incorrect key file for table '/data/mysql7/performance_schema_vardir/tmp/#sql_6b8_17.MYI'; 
    try to repair it

    Since this query requires a sort, MySQL creates a hidden temporary table called ‘#sql_6b8_17.MYI’ to hold the intermediate results. While the query was executing, some

      [Read more...]
    Put your MySQL Knowledge to Good Use with Tim Callaghan at Percona Live-London, November 12
    +0 Vote Up -0Vote Down

    Attending Percona Live in London next week?

    Don’t miss the chance to hear Tokutek’s Vice President of Engineering, Tim Callaghan, discuss how to use your MySQL knowledge to become an instant MongoDB Guru and the advantages of using Fractal Tree&#174 indexes in MySQL and MongoDB. Tim will be speaking about these topics in two separate sessions at 12:00pm and 5:00pm on November 12.

    For more information on these sessions and Percona Live-London, visit https://www.percona.com/live/london-2013/users/tim-callaghan.

    Introducing TokuMX Transactions for MongoDB Applications
    +1 Vote Up -1Vote Down

    Since our initial release last summer, TokuMX has supported fully ACID and MVCC multi-statement transactions. I’d like to take this post to explain exactly what we’ve done and what features are now available to the user.

    But before beginning, an important note: we have implemented this for non-sharded clusters only. We do not support distributed transactions across different shards.

    At a high level, what have we done?

    We have taken MongoDB’s basic transactional behavior, and extended it. MongoDB is transactional with respect to one, and only one, document. MongoDB guarantees single document atomicity. Journaling provides durability

      [Read more...]
    November 6 Webinar: 5 Pitfalls to Avoid with MySQL and Big Data
    +0 Vote Up -0Vote Down

    You love MySQL for its ease of deployment – but are you worried about how your application will perform when it starts to scale?

    SPEAKER: Gerry Narvaja, Tokutek
    DATE: Wednesday, November 6th
    TIME: 1pm ET
    Register Now!

    Join this interactive webinar with Gerry Narvaja of Tokutek as he walks through the potential pitfalls when using MySQL for Big Data applications, how you can avoid unnecessary tolls on time and resources and tips on how to get the most out of your MySQL applications with open source TokuDB.

    Attend this webinar to learn how to:

    • dramatically increase performance without having to rewrite code
    • reduce the total cost of your servers and



      [Read more...]
    Problems with Multiple XA Storage Engines in MySQL 5.6
    +2 Vote Up -0Vote Down

    While integrating TokuDB into MySQL 5.6, we found that MySQL 5.6 does not support more than one XA storage engine. For example, there is an assert in the ha_recover function that fires when the total number of XA storage engines is greater than one. After disabling this assert, we found lots of bugs in the MySQL 5.6 implementation of the TC_LOG_MMAP class, which is used when running with the binlog turned off.

    There are two alternatives that we know of to fix this problem in MySQL 5.6:

    • First, we could merge code from MariaDB 5.5 into MySQL 5.6. The advantage of this approach is that we have been running this code with TokuDB in MariaDB 5.5 for a long time, so we have confidence in its correctness.
    • Second, we found that MySQL 5.7.2 has made changes to allow multiple XA storage engines. This is great news for TokuDB since we have one
      [Read more...]
    Interactive Debugging of Transaction Conflicts with TokuDB
    +0 Vote Up -0Vote Down

    I am developing a concurrent application that uses TokuDB to store its database. Sometimes, one of my SQL statements returns with a ‘lock wait timeout exceeded’ error. How do I identify the cause of this error? First, I need to understand a little bit about how TokuDB transactions use locks. Then, I need to understand how to use the MySQL information schema to look at the current state of the locks.

    Transactions and Locks

    TokuDB uses key range locks to implement serializable transactions. These locks are acquired as the transaction progresses. The locks are released when the transaction commits or aborts.

    TokuDB stores these locks in a data structure called the lock tree. The lock tree stores the set of range locks granted to each transaction. In addition, the lock tree stores the set of locks that are not granted due to a conflict

      [Read more...]
    Announcing TokuDB v7.1
    +0 Vote Up -0Vote Down

    Today we released TokuDB v7.1, which includes the following important features and fixes:

    • Added ability for users to view lock information via information_schema.tokudb_trx, information_schema.tokudb_locks, and information_schema.tokudb_lock_waits tables.
    • Changed the default compression to zlib and default basement node size to 64K.
    • Changed default analyze time to 5 seconds.
    • Added server variable to control amount of memory allocated for each bulk loader. In prior TokuDB versions each loader allocated 50% of the available TokuDB cache.
    • Changed table close behavior such that all data for the table remains in the cache (and is not flushed immediately).
    • Removed user reported stalls due to cache pressure induced by the bulk loader, lock tree escalation, and a particular open table stall.
    • Several bugs and behavioral issues
      [Read more...]
    Introducing TokuMX Clustering Indexes for MongoDB
    +1 Vote Up -0Vote Down

    Since introducing TokuMX, we’ve discussed benefits that TokuMX has for existing MongoDB applications that require no changes. In this post, I introduce an extension we’ve made to the indexing API: clustering indexes, a tool that can tremendously improve query performance. If I were to speak to someone about clustering indexes, I think the conversation could go something like this…

    What is a Clustering Index?

    A clustering index is an index that stores the entire document, not just the defined key.

    A common example is

      [Read more...]
    A TokuDB Stall Caused by Conflicting Transactions When Opening a Table
    +0 Vote Up -0Vote Down

    One of our customers reported that ‘create table select from’ statements stall for a period of time equal to the TokuDB lock timeout.  This indicated a lock conflict between multiple transactions.  In addition, other MySQL clients that were opening unrelated tables were also stalled.  This indicated that some shared mutex is held too long.  We discuss details about this bug and how it was fixed.  The bug fix will be distributed in TokuDB 7.1.0.

    Example
    Suppose that we set the tokudb lock timeout to 60 seconds just to exaggerate the stall.

    mysql> set global tokudb_lock_timeout=60000;
    Query OK, 0 rows affected (0.00 sec)

    We then create a simple table.

    mysql> create table s (id int primary key);
    Query OK, 0 rows affected (0.02







      [Read more...]
    October 8 Webinar: Getting Started with MySQL
    +0 Vote Up -0Vote Down

    Save Time & Money – Do It Right The First Time

    SPEAKER: Gerry Narvaja, Tokutek
    DATE: Tuesday, October 8th
    TIME: 1pm ET

    If you are thinking of using the leading open source database in a project, learn just how easy it is to get started with MySQL, and how important it is to do it right. A few simple decisions early in development, and knowing when and how to effectively use the TokuDB performance engine, can save significant time and money down the road. Key metrics for MySQL success include performance, scalability, database size, and agility. Register Now!

    Attend this webinar to learn:

    • The basics of installing and configuring MySQL and TokuDB
    • How to maximize performance for long-term scalability


      [Read more...]
    How we built TokuMX
    +0 Vote Up -0Vote Down

    When I get to talk to people about TokuMX and how it’s an optimized MongoDB, I sometimes get follow-up questions like:

    • “Is it an in-memory proxy?”
    • “Write optimized? So you buffer all of the writes in memory and lose them on crash?”
    • “Did you re-implement the server and match the protocol?”

    None of these things describe TokuMX, but it demonstrates that there are many schools of thought on how to optimize databases, and MongoDB in particular. I’d like to elaborate more on what TokuMX really is and how we built it. First, let’s talk about what MongoDB is.

    MongoDB consists of a server process that stores data and executes queries, mongod, a sharding router process, mongos, as well as a wire protocol for interacting with these

      [Read more...]
    TokuMX Hot Backup – Part 3
    +0 Vote Up -0Vote Down

    Last week I described TokuDB’s new Hot Backup feature.  This week we are going to briefly discuss the same feature, but as it was added to TokuMX, our version of MongoDB.

    Since the Hot Backup library is essentially a shim between MySQL and the Linux kernel, intercepting file system calls for the life of the process, it should be easy to add this to any other system, including TokuMX.  Indeed with our addition of transactions and logging to TokuMX we can gain a consistent backup of any data set at any time.

    Unlike MySQL, where system tables use the non-transactional MyISAM storage engine, TokuMX uses internal

      [Read more...]
    A TokuDB Stall Caused by a Big Transaction and How It was Fixed
    +1 Vote Up -0Vote Down

    One of our customers sometimes observed lots of simple insertions taking far longer than expected to complete. Usually these insertions completed in milliseconds, but the insertions sometimes were taking hundreds of seconds. These stalls indicated the existence of a serialization bug in the Fractal Tree index software, so the hunt was on. We found that these stalls occurred when a big transaction was committing and the Fractal Tree index software was taking a checkpoint. This problem was fixed in both TokuDB 7.0.3 and TokuMX 1.0.3. Please read on as we describe some details about this bug and how we fixed it. We describe some of the relevant Fractal Tree index algorithms first.

    What is a Big Transaction?

    Each transaction builds a rollback log as it performs Fractal Tree index operations. The rollback log is maintained in

      [Read more...]
    TokuDB Hot Backup – Part 2
    +0 Vote Up -0Vote Down

    In my last post, I discussed the existing backup solutions for MySQL.  At the end I briefly discussed why the backup solutions for InnoDB do not apply to TokuDB.  Now I’m going to outline the backup solution we created.  Our solution works for both TokuDB and InnoDB.  It also has no knowledge of the log files and does not require any changes to either storage engine.  In fact, the library could be used with almost any process; it has no knowledge of what types of files are being backed up.

    Shims

    Tokutek’s Hot Backup is essentially a shim between the mysqld process and the operating system (Linux only, at this point.)  It is a separately compiled C++ library that simply gets linked into the mysqld application at the end of the respective build process.  We ship this library

      [Read more...]
    Lock Diagnostics and Index Usage Statistics in TokuMX v1.2.1
    +0 Vote Up -0Vote Down

    TokuMX v1.2.1 introduces two simple new features to help you understand the performance characteristics of your database: lock diagnostics and index usage statistics. We’d like to take you through a few examples of what these features are and how to use them.

    Lock Diagnostics

    Since we introduced TokuMX, one of the most frequent complaints has been about “lock not granted” errors.  These arise when a long-running operation takes document-level locks, and other clients timeout while waiting to acquire the same locks.

    This is a new problem in TokuMX that doesn’t exist in MongoDB, because MongoDB

      [Read more...]
    Showing entries 1 to 30 of 272 Next 30 Older Entries

    Planet MySQL © 1995, 2014, Oracle Corporation and/or its affiliates   Legal Policies | Your Privacy Rights | Terms of Use

    Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.