I'm pleased to announce the release of Spider storage engine version 2.5(beta) and Vertical Partitioning storage engine version 0.5(alpha).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
Vertical Partitioning is a Storage Engine for vertical partitioning for a table.
http://launchpad.net/vpformysql

The main changes in this version are following.
- The standalone compilation with Vertical Partitioning storage engine is now available.

- Vertical Partitioning can be used for multi-step partitioning.
  Example:
    create table a(a int primary key)engine=myisam;
    create table b(a int primary key)engine=vp comment 'tnl "a"';











  [Read more...]

I'm pleased to announce the release of Spider storage engine version 2.4(beta) and Vertical Partitioning storage engine version 0.4(alpha).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
Vertical Partitioning is a Storage Engine for vertical partitioning for a table.
http://launchpad.net/vpformysql

The main changes in this version are following.
- "Engine-condition-pushdown" is available on Vertical Partitioning with Spider.

Please see "99_change_logs.txt" in the download documents for checking other changes.
Thanks to Gennady for bug report.

Enjoy!

I'm pleased to announce the release of Spider storage engine version 2.3(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/

The main changes in this version are following.
- Add UDF parameter "connection_channel".
  This parameter is used for increasing background parallelism to same remote server by using multiple connections.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

I'm pleased to announce the release of Spider storage engine version 2.2(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/

The main changes in this version are following.
- Add UDF parameter "iop000".
  You can use "ignore" and "replace" at inserting records to temporary table list's tables using this parameter.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

I'm pleased to announce the release of Spider storage engine version 2.1(beta).
http://spiderformysql.com/

The main changes in this version are following.
- Add UDFs "spider_direct_sql" and "spider_bg_direct_sql".
  These UDFs execute hand writing SQLs on remote server by using Spider's table link and store result sets in temporary tables.
  Main usages are following.
  1.Maintenance remote servers.
    (Create table on remote servers for Spider's link tables, etc.)
  2.Use every resources (excluding table) on remote servers.
  3.Mass update of the bulk data. (Update all rows in tables, etc.
    There are next advantages. It is possible to update it in parallel










  [Read more...]

This is the Sharding for the masses: Introducing the SPIDER storage engine by Giuseppe Maxia, given at OpenSQLCamp, at FrOSCon, in August 2009. These are somewhat live notes, and the slides are available too.

Sharding for the masses View more documents from Giuseppe Maxia.

Why sharding? Scaling, of course. The MySQL way to solve this, is replication (even Yahoo! and Google use this).

When the master doesn’t have enough resources to cope with what you do (i.e. large data sets), replication

  [Read more...]

I'm pleased to announce the release of Spider storage engine version 2.0(beta).
http://spiderformysql.com/
From this time, version 1.x is stable releases, version 2.x is development releases for new features.

The main changes in this version are following.
- Add table parameter "semi_table_lock_connection".
- Add server parameter "spider_semi_table_lock_connection".
  Spider has "spider_semi_trx", "spider_semi_trx_isolation", (for transactional tables like InnoDB) "spider_semi_table_lock" and "semi_table_lock" (for non transactional tables like MyISAM) options for consistent reading at remote servers during executing 1 SQL at local server. But you must use different connections for transactional tables and non transactional tables because "semi_table_lock" causes implicit transaction






  [Read more...]

FrOSCon and the OpenSQLCamp are about to start.
I am packing for Sankt Augustin, where I will attend the fourth edition of FrOSCon and the second OpenSQLCamp. I will have two sessions, Sharding for the masses, about the Spider storage engine and MySQL Sandbox 3, about one of my favorite tools.

The program is very rich. There will be several tracks in the main event and in the associated conferences. If you have any involvement or simply some





  [Read more...]

I'm pleased to announce the release of Spider storage engine version 1.0.
Development status is GA from this version.
http://spiderformysql.com/

From this version, you can download shared library for Linux x86_64.
Thanks to Kazuho for information.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

I'm pleased to announce the release of Spider storage engine version 0.18.(RC)
http://spiderformysql.com/

The main changes in this version are following.
- Add table parameter "use_pushdown_udf".
- Add server parameter "spider_use_pushdown_udf".
  "use_pushdown_udf" and "spider_use_pushdown_udf" are used for sending UDF string to remote server if engine_condition_pushdown is used.

From this version, you can use '\r' and '\n' like a space between table parameter name and table parameter value.
Thanks to Akihiro for report.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

I'm pleased to announce the release of Spider storage engine version 0.17.(RC)
Development status is RC from this version.
http://spiderformysql.com/

The main changes in this version are following.
- Add table parameter "use_table_charset".
- Add server parameter "spider_use_table_charset" and "spider_local_lock_table".
  "use_table_charset" and "spider_use_table_charset" are used for choosing the communication charset (UTF8 or table charset) at connections from Spider to remote server. Basically, you don't need changing this option but you can change this option for performance improvement if the SQL has very longer table and column name string than column data string.
  "spider_local_lock_table" is used for suppressing sending "lock table" statement to remote server







  [Read more...]

I'm pleased to announce the release of Spider storage engine version 0.16.(beta)
http://spiderformysql.com/

The main changes in this version are following.
- Add server parameter "spider_same_server_link" and "spider_table_init_error_interval".
  "spider_same_server_link" is used for creating table link to same MySQL server's table.
  "spider_table_init_error_interval" is used for protection from connection consumption of eternal loop of table link. Please increase this parameter using "set global" command if eternal loop of table link causes connection consumption.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

I'm pleased to announce the release of Spider storage engine version 0.15.(beta)
http://spiderformysql.com/

The main changes in this version are following.
- Add table parameter "auto_increment_mode".
- Add server parameter "spider_auto_increment_mode".
  Spider supports 3 modes.
  0:Use a count that Spider get from the remote server with exclusive lock for auto increment value.
  1:Use a count that Spider has internal for auto increment value.
  2:Auto increment value is given by the remote server.

- Add server parameter "spider_ping_interval_at_trx_start".
  Performance improvement by frequency decrease of checking broken connections.

Please see "99_change_logs.txt" in the download documents for checking













  [Read more...]

I'm pleased to announce the release of Spider storage engine version 0.14.(beta)
http://spiderformysql.com/

The main changes in this version are following.
- The standalone compilation with Spider storage engine is now available.
Please see "03_install_spider_storage_engine.txt" for more detail.
Tatsuhiko and Tetsuro,
Thank you for your advice!

Please see "99_change_logs.txt" in the download documents for checking other changes.


Spider storage engine was introduced at following with the usage.

The Data Charmer(English)
http://datacharmer.blogspot.com/2009/04/test-driving-spider-storage-engine.html
















  [Read more...]

I'm pleased to announce the release of Spider storage engine version 0.13.(beta)
http://spiderformysql.com/

The main changes in this version are following.
- Add table parameter "crd_bg_mode" and "sts_bg_mode".
- Add server parameter "spider_crd_bg_mode" and "spider_sts_bg_mode".
  MySQL needs table's analysis informations when select statement is executed. Spider storage engine gets this informations from remote table but this action often generates over-heads.
  This options remove generated over-heads by getting analysis informations at background but you must set this option to 0 if the table's analysis informations change frequently.
  * There are other options for the over-heads reduction of the analysis informations acquisition. I will introduce this at







  [Read more...]

In my previous article about the Spider storage engine, I made some tests and I saw that the engine has potential. I did also identify some problems, which were promptly addressed by the author. I have looked at the latest version (0.12), and the results are very much encouraging.

Installing the Spider storage engine is not trivial but not extremely difficult either. My previous article has a



  [Read more...]

I released Spider storage engine version 0.12.(beta)
http://spiderformysql.com/

Spider storage engine can be used for database sharding by using table partitioning and table link.
Spider storage engine can synchronize an update for remote MySQL servers by XA transaction at Spider-internal.
Introduction document:
http://www.mysqlconf.com/mysql2009/public/schedule/detail/6837

The main changes in this version are following.
("Table parameter" that is used following is parameters for each table. Please see "04_table_create.txt" and "06_table_parameters.txt" in the download documents for more details.)
- Add table parameter "table_count_mode".
  If storage engine can return correct full record










  [Read more...]

At the MySQL Conference 2009 I attended a session about the Spider storage engine, an engine with built-in sharding features.
The talk was notable for the speaker wearing a spiderman costume, and for some language barrier that made the talk less enjoyable than it should be. That's a pity, because the engine is very intriguing, and deserves some exploration.

What is the Spider engine, then? In short, it's an extension to the partitioning engine with the ability of connecting to remote servers. Basically, partitions + federated, except that Federated is explicitly removed during the compilation. Additionally, the spider




  [Read more...]