Home |  MySQL Buzz |  FAQ |  Feeds |  Submit your blog feed |  Feedback |  Archive |  Aggregate feed RSS 2.0 English Deutsch Español Français Italiano 日本語 Русский Português 中文
Showing entries 1 to 26

Displaying posts with tag: openark kit (reset)

common_schema & openark-kit in the media: #DBHangOps, OurSQL
+0 Vote Up -0Vote Down


I had the pleasure of joining into @DBHangOps today, and speak about common_schema and openark-kit. What was meant to be a 15 minute session turned to be 50 -- sorry, people, I don't talk as much at home, but when it comes to my pet projects...

I also realized I was missing on a great event: DBHangOps is a hangout where you can chat and discuss MySQL & related technologies with friends and colleagues, with whom you typically only meet at conferences. I will certainly want to attend future events.

Thanks to John Cesario and Geoffrey Anderson who invited me to talk, and to the friends and familiar faces who attended; I was happy to talk about my work, and very interested in

  [Read more...]
opeark-kit revision 196 released
+1 Vote Up -0Vote Down

This is a long due maintenance release of openark-kit. This release includes bugfixes and some enhancements, mainly to oak-online-alter-table.

oak-online-alter-table Changes / bug fixes include:

  • Support for keyword-named columns
  • Use of FORCE INDEX due to lack of MySQL's ability for figure out the chunking key at all times
  • --sleep-ratio option added; allows for sleep time proportional to execution time (as opposed to constant sleep time with --sleep)
  • Support for chunk-retry (in case of deadlock) via --max-lock-retries)
  • Fixed order of cleanup
  • Fixed bug with verbose messages with non-integer
  [Read more...]
Speaking at Percona Live 2013: common_schema, lightning talks
+3 Vote Up -0Vote Down

In two weeks time I will be giving these talks at Percona Live:

  • common_schema: DBA's framework for MySQL: an introduction to common_schema, my evolving server side solutions project. This will be a revised version of the talk I gave at Percona Live London; I have felt some weaknesses during that talk, which I've thrown out, letting room for cool stuff. I will discuss common_schema's various views, interesting and useful routines, the power of QueryScript, and a brief intro to the newcomer
  [Read more...]
common_schema over traditional scripts
+1 Vote Up -0Vote Down

If you are familiar with both openark kit and common_schema, you'll notice I've incorporated some functionality already working in openark kit into common_schema, essentially rewriting what used to be a Python script into SQL/QueryScript.

What was my reasoning for rewriting good code? I wish to explain that, and provide with a couple examples.

I'm generally interested in pushing as much functionality into the MySQL server. When using an external script, one:

  • Needs the right dependencies (OS, Perl/Python version, Perl/Python modules).
  • Needs to provide with connection params,
  • Needs to get acquainted with a lot
  [Read more...]
State of InnDB Online DDL in MySQL 5.6.8-RC
+2 Vote Up -0Vote Down

5.6.8-rc is out, and so I'm following up on InnoDB's online DDL new feature: the ability to SELECT, INSERT, DELETE, UPDATE a table even while an ALTER TABLE is executing on same table.

The brief summary

Not as advertised; many things can't be done.

The longer review

I'm using 5.6.8-rc 64bit binary distribution for Linux, installed via mysqlsandbox. My hardware is irrelevant, but the fact I'm testing on my laptop assists me in that ALTER TABLE operations take a while, so that I'm able to easily type commands in two terminals and have the time to watch them being executed. Query cache is disabled.

I'm using the sakila sample database, and in particular I'm working with the rental table. Here's the table definition:

  [Read more...]
Experimenting with 5.6 InnoDB Online DDL (bugs included)
+2 Vote Up -1Vote Down

MySQL 5.6 offers the groundbreaking online DDL operations for InnoDB. Most common use cases will enjoy this feature, and the need for online alter table scripts will decrease. This is a killer feature!

I've put this new feature to the usability test. How did it go? Not too well, I'm afraid.

[Updates to this text inline], also see this followup.

sakila & DDL

sakila is still a very useful database. I say "still" because it is not very large, and computing power is getting stronger; yet on my laptop some operations can still take many seconds to

  [Read more...]
How common_schema split()s tables - internals
+1 Vote Up -0Vote Down

This post exposes some of the internals, and the SQL behind QueryScript's split. common_schema/QueryScript 1.1 introduces the split statement, which auto-breaks a "large" query (one which operates on large tables as a whole or without keys) into smaller queries, and executes them in sequence.

This makes for easier transactions, less locks held, potentially (depending on the user) more idle time released back to the database. split has similar concepts to oak-chunk-update and

  [Read more...]
+3 Vote Up -0Vote Down

Have just read INSERT, Don’t DELETE by Aaron Brown, and have some lengthy response, which is why I write this post instead of commenting on said post.

I wish to offer my counter thought and suggest that DELETEs are probably the better choice.

Aaron suggests that, when one wishes to purge rows from some table, a trick can be used: instead of DELETEing unwanted rows, one can INSERT "good" rows into a new table, then switch over with RENAME (but please read referenced post for complete details).

I respectfully disagree on several points discussed.


The fact one needs to block writes during the time of creation of new table is problematic: you need to essentially turn off parts of your application. The posts suggests one could

  [Read more...]
Webinar review: Zero-Downtime Schema Changes In MySQL
+1 Vote Up -0Vote Down

Yesterday I attended the Zero-Downtime Schema Changes In MySQL webinar by Baron Schwartz, Percona (do you say "attended" for something you listened to from your home office?)

I was keen to learn about possible enhancements and improvements of pt-online-schema-change over oak-online-alter-table. Here are my impressions:

The base logic of pt-online-schema-change is essentially the same as of oak-online-alter-table. You create a ghost/shadow table, create complex triggers, copy in chunks, freeze and swap. Both work on any type of

  [Read more...]
Documentation in SQL: CALL for help()
+1 Vote Up -0Vote Down

Documentation is an important part of any project. On the projects I maintain I put a lot of effort on documentation, and, frankly, the majority of time spent on my projects is on documentation.

The matter of keeping the documentation faithful is a topic of interest. I'd like to outline a few documentation bundling possibilities, and the present the coming new documentation method for common_schema. I'll talk about any bundling that is NOT man pages.

High level: web docs

This is the initial method of documentation I used for openark kit and mycheckpoint. It's still valid for mycheckpoint. Documentation is web-based. You need Internet access to read it. It's in HTML

  [Read more...]
Slides for “openark-kit: MySQL utilities for everyday use”
+1 Vote Up -0Vote Down

Today I have delivered my talk, openark-kit: MySQL utilities for everyday use, at the O’REILLY  MySQL Conference 2011.

The slides are uploaded to the O’Reilly site, and I’m attaching them here as well. Feel free to download the PDF: openark-kit-mysqlconf11.pdf

I wish to thank all who attended my talk!


Checking for AUTO_INCREMENT capacity with single query
+3 Vote Up -0Vote Down

Darn! This means oak-show-limits becomes redundant. Am I not supposed to speak about it on my coming presentation? Bad timing!

You have AUTO_INCREMENT columns. How far are you pushing the limits? Are you going to run out of AUTO_INCREMENT values soon? Perhaps you wonder whether you should ALTER from INT to BIGINT?

The answer is all there in INFORMATION_SCHEMA. The TABLES table shows the current AUTO_INCREMENT value per table, and the COLUMNS table tells us all about a column’s data type.

It takes some

  [Read more...]
Would you be my friend on mysqlconf? (tempting offer inside)
+1 Vote Up -1Vote Down

I’m still throwing papers to the trash and starting all over, fixing, rewriting and improving my talk at mysqlconf 2011, where I will be presenting openark-kit: MySQL utilities for everyday use.

However I’ve got something up my sleeve: a benefit many can enjoy, that’ll make me a respectful, popular and sought after speaker. While others may try and lure you with such earthly temptations as a 20% off discount, I am in a position to offer you a more spiritual gift: my friendship!

See, if you become my friend, I can offer you a 25% discount on the MySQL conference. Yes, that’s 5% more than my competitors! The only thing I ask in return is

  [Read more...]
Speaking at the O’Reilly MySQL Conference 2011
+5 Vote Up -0Vote Down

I’m very pleased and humbled to announce that my submission to the upcoming O’Reilly MySQL Conference, April 2011, has been accepted.

I will present a 45 minute session titled openark-kit: MySQL utilities for everyday use.

In this session, I will present some of the tools in the openark kit. We’ll discuss some limitations of the MySQL server, and how openark kit tools overcome those limitations and provide with solutions to common maintenance and audit problems.

This will be a technical session and will discuss various topics of the MySQL server: security, execution plans, replication, triggers and more. I do not intend to discuss all tools, nor to cover the various options. Instead, I’ll present the “behind the

  [Read more...]
oak-hook-general-log: your poor man’s Query Analyzer
+0 Vote Up -0Vote Down

The latest release of openark kit introduces oak-hook-general-log, a handy tool which allows for some analysis of executing queries.

Initially I just intended for the tool to be able to dump the general log to standard output, from any machine capable to connect to MySQL. Quick enough, I realized the power it brings.

With this tool, one can dump to standard output all queries using temporary tables; or using a specific index; or doing a full index scan; or just follow up on connections; or… For example, the following execution will only log queries which make for filesort:

oak-hook-general-log --user=root --host=localhost --password=123456
  [Read more...]
openark-kit (rev. 170): new tools, new functionality
+3 Vote Up -0Vote Down

I’m pleased to announce a new release of the openark kit. There’s a lot of new functionality inside; following is a brief overview.

The openark kit is a set of utilities for MySQL. They solve everyday maintenance tasks, which may be complicated or time consuming to work by hand.

It’s been a while since the last announced release. Most of my attention was on mycheckpoint, building new features, writing documentation etc. However my own use of openark kit has only increased in the past few months, and there’s new useful solutions to common problems that have been developed.

I’ve used and improved many tools over this time, but doing the final cut, along with proper documentation, took

  [Read more...]
Thoughts and ideas for Online Schema Change
+6 Vote Up -0Vote Down

Here’s a few thoughts on current status and further possibilities for Facebook’s Online Schema Change (OSC) tool. I’ve had these thoughts for months now, pondering over improving oak-online-alter-table but haven’t got around to implement them nor even write them down. Better late than never.

The tool has some limitations. Some cannot be lifted, some could. Quoting from the announcement and looking at the code, I add a few comments. I conclude with a general opinion on the tool’s abilities.

“The original table must have PK. Otherwise an error is returned.”

  [Read more...]
openark-kit, Facebook Online Schema Change, and thoughts on open source licenses
+3 Vote Up -0Vote Down

MySQL@Facebook team have recently published an Online Schema Change code for non blocking ALTER TABLE operations. Thumbs Up!

The code is derived from oak-online-alter-table, part of openark-kit, a toolkit I’m authoring. Looking at the documentation I can see many ideas were incorporated as well. And of course many things are different, a lot of work has been put to it by MySQL@Facebook.

openark-kit is currently released under the new BSD license, and, as far as I can tell (I’m not a lawyer), Facebook’s work has followed the license to the letter. It is a strange thing to see your code incorporated into another project. While I knew work has begun on the tool by

  [Read more...]
EXPLAIN: missing db info
+0 Vote Up -0Vote Down

I’m further developing a general log hook, which can stream queries from the general log.

A particular direction I’m taking is to filter queries by their type of actions. For example, the tool (oak-hook-general-log) can be instructed to only stream out those queries which involve creation of a temporary table; or those which cause for a filesort, or full index scan, etc.

This is done by evaluating of query execution plans on the fly. I suspect the MySQL query analyzer (http://www.mysql.com/why-mysql/white-papers/mysql_wp_queryanalyzer.php) roughly does the same (as a small part of what it does).

It’s almost nothing one cannot do with sed/awk. However, I bumped into a couple of problems:

  • The general log (and the
  •   [Read more...]
    oak-hook-general-log: streaming general log
    +0 Vote Up -0Vote Down

    I’m seeking input on a new openark kit utility I’ve started to implement.

    The tool, oak-hook-general-log, will hook up to a MySQL (>= 5.1) server, and stream the general log into standard output. It looks like this:

    bash$ python src/oak/oak-hook-general-log.py --socket=/tmp/mysql.sock --user=root
    2010-03-21 10:18:42     root[root] @ localhost []       79      1       Query   SELECT COUNT(*) FROM City
    2010-03-21 10:18:48     root[root] @ localhost []       79      1       Query   DELETE FROM City WHERE id=1000
    2010-03-21 10:18:54     root[root] @ localhost []       79      1       Query   SHOW PROCESSLIST
    2010-03-21 10:19:06     root[root] @ localhost []       79     
      [Read more...]
    mk-schema-change? Check out ideas from oak-online-alter-table
    +1 Vote Up -0Vote Down

    In response to Mark Callaghan’s post mk-schema-change.

    I apologize for not commenting on the post itself, I do not hold a Facebook account. Anyway this is a long write, so it may as well deserve a post of its own.

    Some of the work Mark is describing already exists under openark kit’s oak-online-alter-table. Allow me to explain what I have gained there, and how the issue can be further pursued. There is relevance to Mark’s suggestion.

    oak-online-alter-table uses a combination of locks, chunks and triggers to achieve an almost non-blocking ALTER TABLE effect. I had a very short opportunity to

      [Read more...]
    RPM builds for openark kit
    +3 Vote Up -0Vote Down

    Thanks to Lenz Grimmer, openark kit is now available in RPM format.

    .DEB or python packages are available, as usual, on project page on Google Code.

    Thank you, Lenz!

    New and Noteworthy in openark kit
    +0 Vote Up -0Vote Down

    A new release of openark kit is out. Some interesting additions.changes are:

    openark kit is a set of utilities for MySQL, helping in easing out everyday’s work. Let’s look more closely at the changes.

      [Read more...]
    Online ALTER TABLE now available in openark kit
    +0 Vote Up -0Vote Down

    A new utility in openark kit allows for online ALTER TABLE operation. That is, the modification of table structure without locking down the entire table for the duration of the operation. The oak-online-alter-table utility works under the following restrictions:

    • The table has at least one single-column UNIQUE KEY [*]
    • Altered table shares a single-column UNIQUE KEY with the original table [*]
    • No ‘AFTER’ triggers are defined on the table (the utility creates its own triggers for the duration of the operation)
    • The table has no FOREIGN KEYs [*][#]
    • Table name is no longer than 57 characters

    [*]: Restriction is scheduled to be removed

      [Read more...]
    Blocking user accounts
    +0 Vote Up -0Vote Down

    A long time missing feature in MySQL is temporarily blocking accounts: denying a user to log in, without affecting any other of her privileges.

    There is no such privilege as ‘LOGIN’ in the grants table, as the ability to log in is the most basic one MySQL allows. This basic privilege is called USAGE.

    I’ll present a hack around this, one which oak-block-account implements. Before presenting the hack, lets lay down some requirements:

    • A user can be blocked from logging in to MySQL.
    • Such a blocked user can later be ‘released’, re-enabling him to log in.
    • It should be possible to determine if a certain user is currently blocked or not.

    A first attempt to answer the above requirements is to change the account’s password. As a naive

      [Read more...]
    Announcing openark kit
    +0 Vote Up -0Vote Down

    It is my great pleasure to announce the availability of the openark kit, a set of lightweight utilities for MySQL, which eases every day tasks.

    The available tools are:

      [Read more...]
    Showing entries 1 to 26

    Planet MySQL © 1995, 2014, Oracle Corporation and/or its affiliates   Legal Policies | Your Privacy Rights | Terms of Use

    Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.