Home |  MySQL Buzz |  FAQ |  Feeds |  Submit your blog feed |  Feedback |  Archive |  Aggregate feed RSS 2.0 English Deutsch Español Français Italiano 日本語 Русский Português 中文
Showing entries 1 to 15

Displaying posts with tag: storage engine api (reset)

An old note on the Storage Engine API
+1 Vote Up -0Vote Down

Whenever I stick my head into the MySQL storage engine API, I’m reminded of a MySQL User Conference from several years ago now.

Specifically, I’m reminded of a slide from an early talk at the MySQL User Conference by Paul McCullagh describing developing PBXT. For “How to write a Storage Engine for MySQL”, it went something like this:

  • Develop basic INSERT (write_row) support – INSERT INTO t1 VALUES (42)
  • Develop full table scan (rnd_init, rnd_next, rnd_end)  - SELECT * from t1
  • If you’re sane, stop here.
  • A lot of people stop at step 3. It’s a really good place to stop too. It avoids most of the tricky parts that are unexpected, undocumented and unlogical (yes, I’m inventing words here).

    MySQL vs Drizzle plugin APIs
    +1 Vote Up -0Vote Down

    There’s a big difference in how plugins are treated in MySQL and how they are treated in Drizzle. The MySQL way has been to create a C API in front of the C++-like (I call it C- as it manages to take the worst of both worlds) internal “API”. The Drizzle way is to have plugins be first class citizens and use exactly the same API as if they were inside the server.

    This means that MySQL attempts to maintain API stability. This isn’t something worth trying for. Any plugin that isn’t trivial quickly surpasses what is exposed via the C API and has to work around it, or, it’s a storage engine and instead you have this horrible mash of C and C++. The byproduct of this is that no core server features are being re-implemented as plugins. This means

      [Read more...]
    The EXAMPLE storage engine
    +2 Vote Up -0Vote Down

    The Example storage engine is meant to serve mainly as a code example of the stub of a storage engine for example purposes only (or so the code comment at the start of ha_example.cc reads). In reality however, it’s not very useful. It likely was back in 2004 when it could be used as a starting point for starting some simple new engines (my guess would be that more than a few of the simpler engines started from ha_example.cc).

    The sad reality is the complexity of the non-obviousness of the bits o the storage engine API you actually care about are documented in ha_ndbcluster.cc, ha_myisam.cc and ha_innodb.cc. If you’re doing something that isn’t already done by one of those three engines: good luck.

    Whenever I looked at ha_example.cc I always wished there was something

      [Read more...]
    The MEMORY storage engine
    +3 Vote Up -0Vote Down

    I recently wrote about Where are they now: MySQL Storage Engines and The MERGE storage engine: not dead, just resting…. or forgotten. Today, it’s the turn of the MEMORY storage engine – otherwise known as HEAP.

    This is yet another piece of the MySQL server that sits largely unmaintained and unloved. The MySQL Manual even claims that it supports encryption… with the caveat of having to use the SQL functions for encryption/decryption rather than in the engine itself (so,

      [Read more...]
    Where are they now: MySQL Storage Engines
    +3 Vote Up -0Vote Down

    There was once a big hooplah about the MySQL Storage Engine Architecture and how it was easy to just slot in some other method of storage instead of the provided ones. Over the years I’ve repeatedly mentioned how this

      [Read more...]
    Is your Storage Engine buggy or the database server?
    +5 Vote Up -0Vote Down

    If your storage engine returns an error from rnd_init (or doStartTableScan as it’s named in Drizzle) and does not save this error and return it in any subsequent calls to rnd_next, your engine is buggy. Namely it is buggy in that a) an error may not be reported back to the user and b) everything may explode horribly when rnd_next is called after rnd_init returned an error.

    Unless it is running on MariaDB 5.2 or (soon, when the patch hits the tree) Drizzle.

    Monty (Widenius, not Taylor) wrote a patch for MariaDB based on my bug report that

      [Read more...]
    A more complete look at Storage Engine API
    +3 Vote Up -0Vote Down

    Okay… So I’ve blogged many times before about the Storage Engine API in Drizzle. This API is somewhat inherited from MySQL. We have very much attempted to make it a much cleaner interface. Our goals in making changes include: make it much easier to write and maintain a storage engine, make the upper layer code obviously correct and clear in what it’s doing and being able to more easily introduce optimisations.

    I’ve recently added a Storage Engine that is only used in testing: storage_engine_api_tester. I’ve blogged on it producing call graphs (really state transition graphs) before both for Storage Engine and Cursor.

    I’ve been expanding the test. My

      [Read more...]
    Cursor states
    +2 Vote Up -0Vote Down

    Following on from my post yesterday on the various states of a Storage Engine, I said I’d have a go with the Cursor object too. A Cursor is used by the Drizzle kernel to get and set data in a table. There can be more than one cursor open at once, and more than one per thread. If your engine cannot cope with this, it is its responsibility to figure it out and return the appropriate errors.

    Let’s look at a really simple operation, inserting a couple of rows and then reading them back via a full table scan.

    Now, this graph is slightly incomplete as there is no doEndTableScan() call. But you can see

      [Read more...]
    Storage Engine API state graph
    +2 Vote Up -0Vote Down

    Drizzle still has a number of quirks inherited from the MySQL Storage Engine API (e.g. BLOBs, row buffer, CREATE SELECT and lack of DDL transaction boundaries, key tuple format). One of the things we fixed a long time ago was to have proper methods for StorageEngines to be called for: startTransaction, startStatement, endStatement, commit and rollback.

    If you’ve had to implement a transactional storage engine in MySQL you will be well aware of the pattern

      [Read more...]
    BLOBS in the Drizzle/MySQL Storage Engine API
    +3 Vote Up -0Vote Down

    Another (AFAIK) undocumented part of the Storage Engine API:

    We all know what a normal row looks like in Drizzle/MySQL row format (a NULL bitmap and then column data):

    Nothing that special. It’s a fixed sized buffer, Field objects reference into it, you read out of it and write the values into your engine. However, when you get to BLOBs, we can’t use a fixed sized buffer as BLOBs may be quite large. So, the format with BLOBS is the bit in the row is a length of the blob (1, 2, 3 or 4 bytes – in Drizzle it’s only 3 or 4 bytes now and soon only 4 bytes once we fix a bug that isn’t interesting to discuss here). The Second part of the

      [Read more...]
    Using the row buffer in Drizzle (and MySQL)
    +1 Vote Up -0Vote Down

    Here’s another bit of the API you may need to use in your storage engine (it also seems to be a rather unknown. I believe the only place where this has really been documented is ha_ndbcluster.cc, so here goes….

    Drizzle (through inheritance from MySQL (http://mysql.com)) has its own (in memory) row format (it could be said that it has several, but we’ll ignore that for the moment for sanity). This is used inside the server for a number of things. When writing a Storage Engine all you really need to know is that you’re expected to write these into your engine and return them from your engine.

    The row buffer format itself is kind-of documented (in that it’s mentioned in the MySQL Internals

      [Read more...]
    Storage Engine API: write_row, CREATE SELECT and DDL
    +1 Vote Up -0Vote Down

    (this probably applies exactly the same for MySQL (http://mysql.com) and Drizzle… but I’m just speaking about current Drizzle here)

    In my current merge request for the embedded-innodb-create-select-transaction-arrgh branch (also see this specific revision), you’ll notice an odd hoop that we have to jump through to make CREATE SELECT statements work with an engine such as InnoDB.

    Basically, this is what happens:

    • start transaction
    • start executing SELECT QUERY
      [Read more...]
    The Drizzle (and MySQL) Key tuple format
    +1 Vote Up -0Vote Down

    Here’s something that’s not really documented anywhere (unless you count ha_innodb.cc as a source of server documentation). You may have some idea about the MySQL/Drizzle row buffer format. This is passed around the storage engine interface: in for write_row and update_row and out for the various scan and index read methods.

    If you want to see the docs for it that exist in the code, check out store_key_val_for_row in ha_innodb.cc.

    However, there is another format that is passed to your engine (and that your engine is expected to understand) and for lack of a better name, I’m going to call it the key tuple format. The first place you’ll probably see this is when implementing the index_read function for a Cursor (or handler in MySQL speak).

    You get two things: a pointer to the buffer and the length of the buffer. Since a

      [Read more...]
    Recent Work on Improving Drizzle’s Storage Engine API
    +5 Vote Up -1Vote Down

    Over the past six weeks or so, I have been working on cleaning up the pluggable storage engine API in Drizzle.  I’d like to describe some of this work and talk a bit about the next steps I’m taking in the coming months as we roll towards implementing Log Shipping in Drizzle.

    First, how did it come about that I started working on the storage engine API?

    From Commands to Transactions

    Well, it really goes back to my work on Drizzle’s replication system.  I had implemented a simple, fast, and extensible log which stored records of the data changes made to a server.  Originally, the log was called the Command Log, because the Google Protobuffer messages it contained were called

      [Read more...]
    Improving the Storage Engine “API”
    +0 Vote Up -0Vote Down

    I increasingly enclose the API part of “Storage Engine API” in quotes as it does score a rather large number on the API Design Rusty levels (Coined by Rusty Russell). I give it a 15 (out of 18. lower is better) in this case “The obvious use is wrong”.

    The ideas is that your handler gets called to write a row (the amazingly named handler::write_row()). It’s passed a buffer which is the row to be stored. An engine that uses the MySQL row format (lets say, ARCHIVE) will simply pack the row and write it out.

    Unless there is a TIMESTAMP field with auto set on insert. Up until now (and still now in MySQL) the engine had to check if this was the case and make sure the timestamp field was updated.

    To remove

      [Read more...]
    Showing entries 1 to 15

    Planet MySQL © 1995, 2014, Oracle Corporation and/or its affiliates   Legal Policies | Your Privacy Rights | Terms of Use

    Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.