This week, I was at Percona Live, and it was a lot of fun! I even got to give a talk on write optimization techniques (not just ours), that I’m told will be online soon.

But if you missed that, or even if you didn’t, I’m still in the valley until Monday night. I’ll be speaking very briefly, and fielding questions this Monday, April 16th, at 6PM, at the Silicon Valley NewSQL group’s meetup in Sunnyvale. It’s shaping up to be a great crowd—Amazon, Microsoft, Clustrix, VoltDB, Drizzle, and many others will be there. So if you’re at all

Checkpointing — which involves periodically writing out dirty pages from memory — is central to the design of crash recovery for both TokuDB and InnoDB. A key issue in designing a checkpointing system is how often to checkpoint, and TokuDB takes a very different approach from InnoDB. How often and how much InnoDB checkpoints is complicated, but under certain workloads it can be relatively infrequent. In contrast, TokuDB runs a complete checkpoint starting one minute after the last one ended.

Frequent checkpoints make for fast recovery. Once MySQL crashes, the storage engine needs to replay the log to get back to a correct state. The length of the log is a function of the time since the last checkpoint for TokuDB and a more complicated function of the workload for InnoDB. And replaying the log is single threaded. So

A key feature of our new TokuDB v6.0 release, which I have been blogging about this week, is compression. Compression is always on in TokuDB, and the compression we’ve achieved in the past has been quite good. See a previous post on the 18x compression achieved by TokuDB v5.0 on one benchmark. In our latest release, we’ve updated the way compression works and got 50% improvement on compression.

I decided to present numbers on the same set of data as the old post, so see that post for experimental details.

But first, what are the changes? TokuDB compresses large blocks

Master/slave replication is an important tool that gets used in many ways: distributing read loads among many slaves for performance, using a slave for backups so the master can handle live load, geographically distributed disaster recovery, etc. The Achilles’ Heal of slave performance is that slave workloads are single-threaded. The master can have many clients inserting, updating, querying, whereas the slave has only one insertion client: the master. InnoDB single-client performance is much slower than its multi-client performance, which means that the bottleneck in a master/slave system is often the rate at which a slave can keep up.

If the master has an average transactions per second (tps) that is higher than what the slave can handle, the slave will fall further and further behind. If the slaves are being used to distribute read workload, for example, the

We are excited to announce TokuDB® v6.0, the latest version of Tokutek’s flagship storage engine for MySQL and MariaDB.

This version offers feature and performance enhancements over previous releases, support for XA (two-phase transactional commits), better compression, and reduced performance variability associated with checkpointing. This release also brings TokuDB support up to date on MySQL v5.1, MySQL v5.5 and MariaDB v5.2. There’s a lot of great technical stuff under the hood in this release and I’ll be reviewing the improvements one-by-one over the course of this week.

I’ll be posting more details about the new features and performance, so here’s an overview of what’s in store.

Looks like we’ll be having some more fun at the Percona Live MySQL Conference! In addition to our booth and my colleague Tim’s talk, my lightning talk was accepted. The title is “OLTP and OLAP – Have Your Cake and Eat it Too!” The lightning talks, given in a TBD order, will start Wednesday evening (April 11th) at around 6:30 pm.

Below is the abstract I submitted.

On Monday, I took a break from planning for the upcoming Percona Live MySQL Conference (where we have a session, lightning talk, booth, and other misc activities planned) to go attend the UK-Massachusetts Innovation Economies Conference at the MIT Media Lab. The event featured Gov. Deval Patrick, MIT Media Lab Director Joi Ito, industry experts such as

Tokutek and Percona are giving away free tickets to the Percona Live MySQL Conference and Expo (worth $995 each), and you can win them! We’re also giving away copies of High Performance MySQL, 3rd Edition (worth $55 each).

This year’s event is the best ever, with a better lineup of talks and speakers than ever before.  It’s the one event you should not miss if you’re at all interested in MySQL.  We really want you to be there — and that’s why we’re joining with Percona to give away free tickets! It’s easy to enter:

• Follow our Twitter feed, and retweet us when we mention

Given all the focus and hype on Big Data, I was excited to have the chance at the recent O’Reilly Strata Show to sit down with Jeff Kelly, one of the top rated “Big Data” analysts, to give a MySQL perspective. Below is my interview with Jeff Kelly and David Floyer.

http://siliconangle.tv/video/cube-strata-conference-2012-lawrence-schwartz

In the segment, you’ll find a number of topics. These include indexing technology, NoSQL vs. MySQL, when to

We had the privilege this past week to be invited to be part of the 2012 O’Reilly Strata “Making Data Work” Conference. Some of our photos from the event are here. At the event, we were excited to have Tokutek described in front of the approximately 2,500 attendees during the keynote sessions.

Overall, the diversity of topics discussed at the conference was impressive, spanning databases, developer tools, data visualization techniques, customer stories, and business implications. The full agenda is

Ensuring sufficient disk I/O to catch copyright violations at network speed.

Evidenzia GmbH & Co. KG

Issues addressed:

• Storage growth, including maxed-out disk I/O utilization
• Performance issues and business impact due to slow selects
• Inability to revise data schema on the fly

The Company: Evidenzia GmbH & Co. KG is one of the leading partners of the software, movie and music industry when it comes to tracing copyright infringements

We are excited to announce that we’ve been named as one of ten finalists selected for the startup showcase at the O’Reilly Strata “Making Data Work” Conference at the end of this month in Santa Clara, California. The startup showcase will be held on February 29th, starting at 6:30 pm.

The conference offers a great overview of the big data space, with tracks on Data Science, Business and Industry,

While it might not have been New England’s weekend on the Big Gridiron, it was certainly New England’s day for Big Data at the New England Database Summit on Friday at MIT.

The summit was well attended, with 350 registrants and keynotes from prominent MySQL users such as Mark Callaghan. The coverage was quite broad, with presentations running the gamut from grad students (complete with bodyguards and intimidating academic

I’ll be speaking on April 11th at 4:30 pm in Room 4 in at the Percona Conference and Expo Talk. The topic will be “Creating a Benchmark Infrastructure That Just Works.”

Throughout my career I’ve been involved with maintaining the performance of database applications and therefore created many benchmark frameworks. At Tokutek, an important part of my role is measuring the performance of our storage engine over time and versus competing solutions. There is nothing proprietary about

iiBench measures the rate at which a database can insert new rows while maintaining several secondary indexes. We ran this for 1 billion rows with TokuDB and InnoDB starting last week, right after we launched TokuDB v5.2. While TokuDB completed it in 15 hours, InnoDB took 7 days.

The results are shown below. At the end of the test, TokuDB’s insertion rate remained at 17,028 inserts/second whereas InnoDB had dropped to 1,050 inserts/second. That is a difference of over 16x. Our complete set of benchmarks for TokuDB v5.2 can be found here.

TokuDB® v5.2, the latest version of Tokutek’s flagship storage engine for MySQL and MariaDB, is now available.

This version offers performance enhancements over previous releases, especially for multi-client scale up and point queries, and extends the cases where ALTER TABLE is non-blocking, in particular adding Hot Column Rename.

TokuDB v5.2 maintains all our established advantages: fast trickle load, fast bulk load, fast range queries through clustering indexes, hot schema changes, great compression, no fragmentation, and full MySQL compatibility for ease of installation. See our benchmark page for details.

Multi-client workloads

In TokuDB v5.2, we have reworked our locking scheme to better support multi-client workloads, and as

 
Thanks again to Sheeri Cabral  for having me at the Boston MySQL Meetup on Monday for the talk on “Fractal Tree® Indexes – Theoretical Overview and Customer Use Cases.” The crowd was very interactive, and I appreciated that over 50 people signed up for the event and left some very positive comments and reviews.

In addition, the conversation spilled over late into the night as we made our way over to nearby Mead

FictionPress

Issues addressed:

• Support complex and efficient indexes at 100+ million rows.
• Predicable and consistent performance regardless of data size growth.
• Fast recovery.

Ensuring Predictable Performance at Scale

The Company:  FictionPress operates both FictionPress.com and FanFiction.net and is home to over 6 million works of fiction, with millions of writers/readers

It’s almost the end of the year – that means holiday cards, shopping, cooking, parties, and the inevitable year-end top lists (including gems like this one).

In the spirit of end of year list making, we fed our 60+ blogs this year through Google Analytics to find out what our own top ten blogs were (outside of product announcements). So if you missed an episode of the View (TokuView that is) we’ve got a Tokutek Top Ten for you (spoiler alert – they are mostly technical):

10. Cage Match: OldSQL, NoSQL and NewSQL – References to

Limelight Networks

Issue addressed: Managing metadata at exabyte scale

Delivering Agile Storage in the Cloud with Billions of Assets

The Company: Founded in 2001, Limelight Networks, Inc (NASDAQ: LLNW) is an Internet platform and services company that integrates the most business-critical parts of the online content value chain. Limelight’s cloud-based services enable customers to profit from the shift of content and advertising to the online world, from the explosive growth of mobile and connected devices, and from the migration of IT applications and

At next month’s Boston MySQL Meetup, I will give a talk: “Fractal Tree Indexes – Theoretical Overview and Customer Use Cases.” The meetup is 7 pm Monday, January 9th, 2012, and will be held at MIT Building E51 Room 337e (corner of Ames & Amherst St, Cambridge, MA). Thanks to host Sheeri Cabral for the invitation.

Most databases employ B-trees to achieve a good tradeoff between the ability to update data quickly and to search it quickly. It

As a storage engine developer, I am excited for MySQL 5.6. Looking at http://dev.mysql.com/tech-resources/articles/whats-new-in-mysql-5.6.html, there has been plenty of work done to improve the performance of reads in MySQL for all storage engines (provided they take advantage of the new APIs).

What would be great to add is API improvements to increase the performance of writes, and more specifically, updates. For many applications that perform updates, such as applications that do click counting or impression counting, there are significant opportunities for improving write performance.

Take the following example of click counting (or impression counting). You have a website and want to save the number of times links on your website have been clicked. Your table

With the release of TokuDB v5.0 last March, we delivered a powerful and agile storage engine that broke through traditional MySQL scalability and performance barriers. As deployments of TokuDB have grown more varied, one request we have repeatedly heard from customers and prospects, especially in areas such as online advertising, social media, and clickstream analysis, is for improved performance for multi-client workloads.

Tokutek is now pleased to announce limited beta availability for TokuDB v5.2. The latest version of our flagship product offers a significant improvement over TokuDB v5.0 in multi-client scaling as well as performance gains in point queries, range queries, and trickle load speed. There are a host of other smaller changes and improvements that are detailed in our release notes (available to beta participants).

TokuDB continues to

Thanks to everyone who signed up and attended the webinar I gave this week with Tim Callaghan on Scaling MySQL. For those who missed it and are interested, the video and slides are now posted here.

A brief description of the webinar is also below.

MySQL implementations are often kept relatively small, often just a few hundred GB or less. Anything beyond this quickly leads to painful operational problems such as poor insertion rates, slow queries, hours to days offline for schema changes, prolonged downtime for dump/reload, etc. The promise of scalable MySQL has remained largely unfulfilled, until TokuDB.

TokuDB v5.0 delivers

• Exceptional

 
This version includes support for “SELECT … FOR UPDATE” as well as displaying table “create time” and “last update time” via “SHOW TABLE STATUS”. The release also addresses a number of other bugs and fixes such as “point update” and “replace into” deadlocks. For more details, see the release notes section of the user’s guide, available here. To get the latest version of TokuDB (free for evaluation, development, testing, qualification, experimentation, and POCs ) click here. Starting with this release, we’ve also streamlined the installation process by creating a combined tarball that includes both MySQL and TokuDB.

Note that we’ll be hosting

I’ll be talking about How Fractal Trees Work  today at MIT in the Computational Research In Boston and Beyond (CRIBB) seminar (http://www-math.mit.edu/crib/2011/nov4.html). The talk is at 12:30 in the Stata Center room 32-141.  Pizza available before.

This talk will be academically-oriented (not much marketing).  The abstract is as follows:

Most storage systems employ B-trees to achieve a good tradeoff between the ability to update data quickly and to search it quickly.  It turns out that B-trees are far from the optimimum in this tradeoff space. I’ll talk about Fractal Tree indexes, which were developed in a collaboration between MIT, Stony Brook, and Rutgers.  I’ll talk about how they work, and what their performance bounds are.  My startup, Tokutek, is

MySQL implementations are often kept relatively small, often just a few hundred GB or less. Anything beyond this quickly leads to painful operational problems such as poor insertion rates, slow queries, hours to days offline for schema changes, prolonged downtime for dump/reload, etc. The promise of scalable MySQL has remained largely unfulfilled, until TokuDB.

Time: 2PM EST / 11AM PST

REGISTER TODAY

TokuDB v5.0 delivers

• Exceptional Agility — Hot Schema Changes allow read/write operations during index creation or column/field addition
• Unmatched Speed — Fractal Tree indexes

I’ve been benchmarking and testing TokuDB for a few months now. One goal of benchmarking is to understand what is limiting the performance of a particular configuration. I frequently use “show engine [innodb/tokudb] status;” from within the MySQL command line client as part of my research.

As I run most of my benchmarks on InnoDB as well as TokuDB, I noticed that there are significant differences in the way each present status information. InnoDB returns a single row, with various sections and carriage returns to maintain readability. In contrast, TokuDB presents one piece of status information per row (currently 139 rows as of TokuDB v5.0.5). This is an important distinction if you want to parse, compare, or store discrete status values. Here is sample output from each engine. I’ve cut out portions of each to maintain readability.

InnoDB plugin

Review of Thursday’s Cloud Events in Boston

Everyone is well aware by now of the EC2 outage that Amazon had back in April and it would have surprised no one if that high profile had put a damper on cloud adoption. But judging what we heard yesterday at Boston’s two cloud events (MassTLC’s Cloud Computing Summit and Vilna’s Moving Your Data to the Cloud Panel), cloud solutions can work just fine. For example, there was the customer story told by Douglas Kim, Managing Director, Global Head, PaaS & Cloud Computing at