A key feature of our new TokuDB v6.0 release, which I have been blogging about this week, is compression. Compression is always on in TokuDB, and the compression we’ve achieved in the past has been quite good. See a previous post on the 18x compression achieved by TokuDB v5.0 on one benchmark. In our latest release, we’ve updated the way compression works and got 50% improvement on compression.

I decided to present numbers on the same set of data as the old post, so see that post for experimental details.

But first, what are the changes? TokuDB compresses large blocks

Master/slave replication is an important tool that gets used in many ways: distributing read loads among many slaves for performance, using a slave for backups so the master can handle live load, geographically distributed disaster recovery, etc. The Achilles’ Heal of slave performance is that slave workloads are single-threaded. The master can have many clients inserting, updating, querying, whereas the slave has only one insertion client: the master. InnoDB single-client performance is much slower than its multi-client performance, which means that the bottleneck in a master/slave system is often the rate at which a slave can keep up.

If the master has an average transactions per second (tps) that is higher than what the slave can handle, the slave will fall further and further behind. If the slaves are being used to distribute read workload, for example, the

We are excited to announce TokuDB® v6.0, the latest version of Tokutek’s flagship storage engine for MySQL and MariaDB.

This version offers feature and performance enhancements over previous releases, support for XA (two-phase transactional commits), better compression, and reduced performance variability associated with checkpointing. This release also brings TokuDB support up to date on MySQL v5.1, MySQL v5.5 and MariaDB v5.2. There’s a lot of great technical stuff under the hood in this release and I’ll be reviewing the improvements one-by-one over the course of this week.

I’ll be posting more details about the new features and performance, so here’s an overview of what’s in store.

Looks like we’ll be having some more fun at the Percona Live MySQL Conference! In addition to our booth and my colleague Tim’s talk, my lightning talk was accepted. The title is “OLTP and OLAP – Have Your Cake and Eat it Too!” The lightning talks, given in a TBD order, will start Wednesday evening (April 11th) at around 6:30 pm.

Below is the abstract I submitted.

On Monday, I took a break from planning for the upcoming Percona Live MySQL Conference (where we have a session, lightning talk, booth, and other misc activities planned) to go attend the UK-Massachusetts Innovation Economies Conference at the MIT Media Lab. The event featured Gov. Deval Patrick, MIT Media Lab Director Joi Ito, industry experts such as

Tokutek and Percona are giving away free tickets to the Percona Live MySQL Conference and Expo (worth $995 each), and you can win them! We’re also giving away copies of High Performance MySQL, 3rd Edition (worth $55 each).

This year’s event is the best ever, with a better lineup of talks and speakers than ever before.  It’s the one event you should not miss if you’re at all interested in MySQL.  We really want you to be there — and that’s why we’re joining with Percona to give away free tickets! It’s easy to enter:

• Follow our Twitter feed, and retweet us when we mention

Given all the focus and hype on Big Data, I was excited to have the chance at the recent O’Reilly Strata Show to sit down with Jeff Kelly, one of the top rated “Big Data” analysts, to give a MySQL perspective. Below is my interview with Jeff Kelly and David Floyer.

http://siliconangle.tv/video/cube-strata-conference-2012-lawrence-schwartz

In the segment, you’ll find a number of topics. These include indexing technology, NoSQL vs. MySQL, when to

We had the privilege this past week to be invited to be part of the 2012 O’Reilly Strata “Making Data Work” Conference. Some of our photos from the event are here. At the event, we were excited to have Tokutek described in front of the approximately 2,500 attendees during the keynote sessions.

Overall, the diversity of topics discussed at the conference was impressive, spanning databases, developer tools, data visualization techniques, customer stories, and business implications. The full agenda is

Ensuring sufficient disk I/O to catch copyright violations at network speed.

Evidenzia GmbH & Co. KG

Issues addressed:

• Storage growth, including maxed-out disk I/O utilization
• Performance issues and business impact due to slow selects
• Inability to revise data schema on the fly

The Company: Evidenzia GmbH & Co. KG is one of the leading partners of the software, movie and music industry when it comes to tracing copyright infringements

So there I was, gesticulating in front of the chalkboard, lecturing to 120 students. Topic: the beauty of advanced data structures. The door opens and the department chair walks up the aisle onto the stage and comes up to me. 242 eyes follow her progress through the room. She whispers to me that I must stop speaking in English. Officially, the course must be conducted entirely in Spanish.

Entirely in Spanish… I turn back to face the students. 120 students are waiting for me to speak. I realize that I don’t even know how to say simple constructions like “x plus y” let alone “advanced data structure” or “order log log amortized time complexity.”

Is this a newfangled anxiety dream? Can this replace the tried and true: I’m in front of the class but I forgot to put on my pants? Can this replace the classic: I’m

We are excited to announce that we’ve been named as one of ten finalists selected for the startup showcase at the O’Reilly Strata “Making Data Work” Conference at the end of this month in Santa Clara, California. The startup showcase will be held on February 29th, starting at 6:30 pm.

The conference offers a great overview of the big data space, with tracks on Data Science, Business and Industry,

While it might not have been New England’s weekend on the Big Gridiron, it was certainly New England’s day for Big Data at the New England Database Summit on Friday at MIT.

The summit was well attended, with 350 registrants and keynotes from prominent MySQL users such as Mark Callaghan. The coverage was quite broad, with presentations running the gamut from grad students (complete with bodyguards and intimidating academic

I’ll be speaking on April 11th at 4:30 pm in Room 4 in at the Percona Conference and Expo Talk. The topic will be “Creating a Benchmark Infrastructure That Just Works.”

Throughout my career I’ve been involved with maintaining the performance of database applications and therefore created many benchmark frameworks. At Tokutek, an important part of my role is measuring the performance of our storage engine over time and versus competing solutions. There is nothing proprietary about

iiBench measures the rate at which a database can insert new rows while maintaining several secondary indexes. We ran this for 1 billion rows with TokuDB and InnoDB starting last week, right after we launched TokuDB v5.2. While TokuDB completed it in 15 hours, InnoDB took 7 days.

The results are shown below. At the end of the test, TokuDB’s insertion rate remained at 17,028 inserts/second whereas InnoDB had dropped to 1,050 inserts/second. That is a difference of over 16x. Our complete set of benchmarks for TokuDB v5.2 can be found here.

TokuDB® v5.2, the latest version of Tokutek’s flagship storage engine for MySQL and MariaDB, is now available.

This version offers performance enhancements over previous releases, especially for multi-client scale up and point queries, and extends the cases where ALTER TABLE is non-blocking, in particular adding Hot Column Rename.

TokuDB v5.2 maintains all our established advantages: fast trickle load, fast bulk load, fast range queries through clustering indexes, hot schema changes, great compression, no fragmentation, and full MySQL compatibility for ease of installation. See our benchmark page for details.

Multi-client workloads

In TokuDB v5.2, we have reworked our locking scheme to better support multi-client workloads, and as

 
Thanks again to Sheeri Cabral  for having me at the Boston MySQL Meetup on Monday for the talk on “Fractal Tree® Indexes – Theoretical Overview and Customer Use Cases.” The crowd was very interactive, and I appreciated that over 50 people signed up for the event and left some very positive comments and reviews.

In addition, the conversation spilled over late into the night as we made our way over to nearby Mead

FictionPress

Issues addressed:

• Support complex and efficient indexes at 100+ million rows.
• Predicable and consistent performance regardless of data size growth.
• Fast recovery.

Ensuring Predictable Performance at Scale

The Company:  FictionPress operates both FictionPress.com and FanFiction.net and is home to over 6 million works of fiction, with millions of writers/readers

The MySQL council is looking for candidates for 2012.  Based on community feedback, this will be an open nomination process (and you can nominate yourself).

The MySQL council advocates for the MySQL community, and needs community leaders to help address issues such as keeping the bug database open and keeping the user conference from fragmenting.  The council tries to help solve issues that the community faces with Oracle, IOUG, or anyone else.  The council comprises four to six members representing a mix of consultants, volunteers, community activists, developers, and vendors.  The council is not meant to replace any existing grass roots organizations, rather it may be helpful for gaining visibility for their issues.

As a council member, you would be asked to participate in meetings (they’ve been about once a month by phone, but there may be a

It’s almost the end of the year – that means holiday cards, shopping, cooking, parties, and the inevitable year-end top lists (including gems like this one).

In the spirit of end of year list making, we fed our 60+ blogs this year through Google Analytics to find out what our own top ten blogs were (outside of product announcements). So if you missed an episode of the View (TokuView that is) we’ve got a Tokutek Top Ten for you (spoiler alert – they are mostly technical):

10. Cage Match: OldSQL, NoSQL and NewSQL – References to

Limelight Networks

Issue addressed: Managing metadata at exabyte scale

Delivering Agile Storage in the Cloud with Billions of Assets

The Company: Founded in 2001, Limelight Networks, Inc (NASDAQ: LLNW) is an Internet platform and services company that integrates the most business-critical parts of the online content value chain. Limelight’s cloud-based services enable customers to profit from the shift of content and advertising to the online world, from the explosive growth of mobile and connected devices, and from the migration of IT applications and

At next month’s Boston MySQL Meetup, I will give a talk: “Fractal Tree Indexes – Theoretical Overview and Customer Use Cases.” The meetup is 7 pm Monday, January 9th, 2012, and will be held at MIT Building E51 Room 337e (corner of Ames & Amherst St, Cambridge, MA). Thanks to host Sheeri Cabral for the invitation.

Most databases employ B-trees to achieve a good tradeoff between the ability to update data quickly and to search it quickly. It

As a storage engine developer, I am excited for MySQL 5.6. Looking at http://dev.mysql.com/tech-resources/articles/whats-new-in-mysql-5.6.html, there has been plenty of work done to improve the performance of reads in MySQL for all storage engines (provided they take advantage of the new APIs).

What would be great to add is API improvements to increase the performance of writes, and more specifically, updates. For many applications that perform updates, such as applications that do click counting or impression counting, there are significant opportunities for improving write performance.

Take the following example of click counting (or impression counting). You have a website and want to save the number of times links on your website have been clicked. Your table

With the release of TokuDB v5.0 last March, we delivered a powerful and agile storage engine that broke through traditional MySQL scalability and performance barriers. As deployments of TokuDB have grown more varied, one request we have repeatedly heard from customers and prospects, especially in areas such as online advertising, social media, and clickstream analysis, is for improved performance for multi-client workloads.

Tokutek is now pleased to announce limited beta availability for TokuDB v5.2. The latest version of our flagship product offers a significant improvement over TokuDB v5.0 in multi-client scaling as well as performance gains in point queries, range queries, and trickle load speed. There are a host of other smaller changes and improvements that are detailed in our release notes (available to beta participants).

TokuDB continues to

Thanks to everyone who signed up and attended the webinar I gave this week with Tim Callaghan on Scaling MySQL. For those who missed it and are interested, the video and slides are now posted here.

A brief description of the webinar is also below.

MySQL implementations are often kept relatively small, often just a few hundred GB or less. Anything beyond this quickly leads to painful operational problems such as poor insertion rates, slow queries, hours to days offline for schema changes, prolonged downtime for dump/reload, etc. The promise of scalable MySQL has remained largely unfulfilled, until TokuDB.

TokuDB v5.0 delivers

• Exceptional

 
This version includes support for “SELECT … FOR UPDATE” as well as displaying table “create time” and “last update time” via “SHOW TABLE STATUS”. The release also addresses a number of other bugs and fixes such as “point update” and “replace into” deadlocks. For more details, see the release notes section of the user’s guide, available here. To get the latest version of TokuDB (free for evaluation, development, testing, qualification, experimentation, and POCs ) click here. Starting with this release, we’ve also streamlined the installation process by creating a combined tarball that includes both MySQL and TokuDB.

Note that we’ll be hosting

I’ll be talking about How Fractal Trees Work  today at MIT in the Computational Research In Boston and Beyond (CRIBB) seminar (http://www-math.mit.edu/crib/2011/nov4.html). The talk is at 12:30 in the Stata Center room 32-141.  Pizza available before.

This talk will be academically-oriented (not much marketing).  The abstract is as follows:

Most storage systems employ B-trees to achieve a good tradeoff between the ability to update data quickly and to search it quickly.  It turns out that B-trees are far from the optimimum in this tradeoff space. I’ll talk about Fractal Tree indexes, which were developed in a collaboration between MIT, Stony Brook, and Rutgers.  I’ll talk about how they work, and what their performance bounds are.  My startup, Tokutek, is

MySQL implementations are often kept relatively small, often just a few hundred GB or less. Anything beyond this quickly leads to painful operational problems such as poor insertion rates, slow queries, hours to days offline for schema changes, prolonged downtime for dump/reload, etc. The promise of scalable MySQL has remained largely unfulfilled, until TokuDB.

Time: 2PM EST / 11AM PST

REGISTER TODAY

TokuDB v5.0 delivers

• Exceptional Agility — Hot Schema Changes allow read/write operations during index creation or column/field addition
• Unmatched Speed — Fractal Tree indexes

I had the pleasure of being invited to blog at the MassTLC unConference on Friday. The event was a full day of diverse topics and discussions ranging from the latest in recipe sharing sites, to entrepreneurial CEO war stories, to hot trends in venture investing. An excerpt covering Big Data from my MassTLC blog is below.

Big Data and Analytics in MA

Hosted by Steve O’Leary of Aeris Partners and Bob Zurek (@bzurek) of Oracle

First question – what is Big Data? While often debated, Steve had a working definition of “big” in terms of Volume, Velocity and Variety. Fritz Knabe of IBM noted that Big Data can come from even the most unexpected places, such

Many database management tasks become difficult as you move from millions of rows and gigabytes of data to billions of rows and terabytes of data. Such tasks include ingesting data while maintaining indexes; changing schemas without downtime; and supporting connections, replication, and backup. For some scaling problems (connections and replication), MySQL® is better than most of the competition. For others, such as indexing, schema changes, and backup, MySQL has typically been harder to use. Fortunately, the tasks MySQL does well are in its core, whereas the tasks that are more difficult can be solved with storage engine plug-ins.

I recently gave a talk at Oracle Open World 11, a copy of which can be found here. This presentation discusses how MySQL’s storage engines have

I’ve been benchmarking and testing TokuDB for a few months now. One goal of benchmarking is to understand what is limiting the performance of a particular configuration. I frequently use “show engine [innodb/tokudb] status;” from within the MySQL command line client as part of my research.

As I run most of my benchmarks on InnoDB as well as TokuDB, I noticed that there are significant differences in the way each present status information. InnoDB returns a single row, with various sections and carriage returns to maintain readability. In contrast, TokuDB presents one piece of status information per row (currently 139 rows as of TokuDB v5.0.5). This is an important distinction if you want to parse, compare, or store discrete status values. Here is sample output from each engine. I’ve cut out portions of each to maintain readability.

InnoDB plugin