Sep
20
2009
Having fun with Tokyo Tyrant
Posted by Patrick Galbraith on Sun 20 Sep 2009 16:01 UTC
Tags:

I decided to thee other day to investigate using Tokyo Tyrant because I was interested in the fact it has a memcached protocol and I wanted to get a feel for how it works with my memcached functions for MySQL (UDF). Matt Ingenthron came up with a good term recently: Mem-capable, which Tokyo Tyrant is. I find any key/value storage to be of great interest, particularly those that you simply change the port your memcached client is connecting and simply use it the same way you would memcached.

So, just what is Tokyo Tyrant?

Tokyo Tyrant is a database server, written by Mikio Hirabayashi, for Tokyo Cabinet. It provides for concurrent remote connections to Tokyo Cabinet . It provides its own binary protocol as well as a memcached and HTTP compatibility protocols. It also has C, Perl, PHP, Java, Erlang, Python and Ruby bindings. Other features Tokyo Tyrant offers:

* Hot backup and update log
* Async Replication
* Lua extension

What then is Tokyo Cabinet? Tokyo Cabinet is a database library, also written by Mikio Hirabayashi, key/value database library at that and a modern implementation of DBM. Tokyo Cabinet stores data in a simple database file, each record being a key/value pair. When I say database, this isnt' the type of database where you have tables-- just key/value pairs. For on-disk storage, Tokyo Cabinet also provides you the ability to store records organized into hash table, B+ tree or fixed-length array. Tokyo Cabinet also has an in-memory storage ability with list,tree and hash (with compression using bzip, gzip or custom). Take your pick! Also of interest is that Tokyo Cabinet has various language bindings for scripting languages.

Installing Tokyo Cabinet and Tokyo Tyrant

Installation of Tokyo Cabinet and Tokyo Tyrant was extremely simple. The links for the latest:

Tyrant: http://1978th.net/tokyotyrant/tokyotyrant-1.1.34.tar.gz
Cabinet: http://1978th.net/tokyocabinet/tokyocabinet-1.4.33.tar.gz

The only Ubuntu package I had to install on my server was libbz2-dev. The rest of the installation process is simple:

tar xvzf package
cd package-dir
./configure
make
sudo make install

On my server, I created a user called "tokyo" that I intended to run the Tokyo Tyrant server, ttserver, as. I also needed to create a directory, owned by the tokyo user, /var/ttserver.

I then change the permissions of the Tokyo Tyrant server to be owned by the tokyo user and set the user bit so that it runs as the tokyo user. I did this because Tokyo Tyrant will allow you to run the server as root, whereas memcached will only let you run it as a non-root user. I didn't to run it as root for security reasons.

chown tokyo /usr/local/bin/ttserver
chmod u+s /usr/local/bin/ttserver

There's a utility script that comes with Tokyo Tyrant, ttservctl, which I used to start the server:

./ttservctl start

You can copy this into your /etc/init.d directory and set up your run levels so it starts automatically.

Testing

My tests were pretty simple, and I started out only used on-file hash database, TCHDB, which is what it starts up as using ttservctl, which specifies the database name as /var/ttserver/casket.tch. Interestingly, the database name you specify is the type of storage that will be used. For instance, if you start up ttserver with:

ttserver "*" -- in-memory hash
ttserver "+" -- in-memory tree
ttserver /var/ttserver/casket.tch -- on-disk hash
ttserver /var/ttserver/casket.tcb -- on-disk b-tree
ttserver /var/ttserver/casket.tct -- on-disk fixed-length

I set up an Amazon AMI for my testing needs. I first ran only one Tokyo Tyrant instance, and started off with using memslap:

Tokyo, on-disk hash, 100 threads:

patg@domU-12-31-39-02-C8-92:~/libmemcached$ memslap --concurrency=100 --servers=127.0.0.1:1978
Threads connecting to servers 100
Took 121.777 seconds to load data

Tokyo, in-memory hash, 100 threads:

root@domU-12-31-39-07-62-02:/home/patg/libmemcached# memslap --concurrency=100 --servers=127.0.0.1:1978
Threads connecting to servers 100
Took 43.827 seconds to load data

memcached, 100 threads:

patg@domU-12-31-39-02-C8-92:~/libmemcached$ memslap --concurrency=100 --servers=127.0.0.1:11211
Threads connecting to servers 100
Took 31.511 seconds to load data

Interesting. For a little more information, I then decided to use Matt Ingenthron's test tool, memcachedtest, which I'll summarize there output results here:

Tokyo, on-disk hash, one server, one thread:

Avg set: 111 us (3260) min: 60 us (0) max: 20 ms (0)
Avg get: 128 us (6740) min: 56 us (0) max: 21 ms (0)
Usr: 0.051992
Sys: 0.111982
Tot: 3.18446744073709007217

Tokyo, in-memory hash, one server, one thread:

Average with 1 threads:
Avg set: 58 us (3260) min: 35 us (0) max: 20 ms (0)
Avg get: 64 us (6740) min: 37 us (0) max: 21 ms (0)
Usr: 0.075988
Sys: 0.199969
Tot: 1.307640


memcached, one server, 1 thread:

Avg set: 74 us (3260) min: 46 us (0) max: 20 ms (0)
Avg get: 88 us (6740) min: 41 us (0) max: 21 ms (0)
Usr: 0.056991
Sys: 0.064990
Tot: 2.18446744073709342982


Tokyo, on-disk hash, one server, 10 threads:


Avg set: 451 us (3316) min: 60 us (2) max: 22 ms (3)
Avg get: 439 us (6684) min: 54 us (2) max: 22 ms (4)
Usr: 0.126980
Sys: 0.291955
Tot: 5.591224

Tokyo, in-memory hash, one server, 10 threads:

Average with 10 threads:
Avg set: 139 us (3412) min: 32 us (4) max: 13 ms (0)
Avg get: 194 us (6588) min: 33 us (8) max: 30 ms (0)
Usr: 0.108983
Sys: 0.233964
Tot: 6.18446744073708891863


memcached, one server, 10 threads:

Avg set: 214 us (3300) min: 43 us (7) max: 28 ms (4)
Avg get: 241 us (6700) min: 35 us (0) max: 63 ms (6)
Usr: 0.220966
Sys: 0.319951
Tot: 5.420549

Tokyo, on disk hash, one server, 25 threads:

Avg set: 903 us (3354) min: 62 us (24) max: 129 ms (18)
Avg get: 793 us (6646) min: 55 us (11) max: 99 ms (15)
Usr: 0.158975
Sys: 0.321951
Tot: 6.18446744073709267538

Tokyo, in-memory hash, one server, 25 threads:

Average with 25 threads:
Avg set: 347 us (3320) min: 39 us (15) max: 71 ms (12)
Avg get: 364 us (6680) min: 43 us (0) max: 105 ms (3)
Usr: 0.111982
Sys: 0.242963
Tot: 5.433262

memcached, one server, 25 threads:

Avg set: 316 us (3284) min: 44 us (21) max: 53 ms (21)
Avg get: 322 us (6716) min: 36 us (4) max: 58 ms (22)
Usr: 0.199969
Sys: 0.238963
Tot: 6.18446744073709020772

Then I decided to try with 4 servers.

Tokyo, on disk hash, 10 threads, 4 servers

Avg set: 573 us (3265) min: 57 us (7) max: 24 ms (0)
Avg get: 512 us (6735) min: 53 us (7) max: 20 ms (3)
Usr: 0.132979
Sys: 0.276957
Tot: 5.600615

Tokyo, in-memory hash, 10 threads, 4 servers:

Avg set: 205 us (3352) min: 33 us (6) max: 26 ms (1)
Avg get: 196 us (6648) min: 36 us (0) max: 24 ms (9)
Usr: 0.089986
Sys: 0.229965
Tot: 6.18446744073708876934

memcached, 10 threads, 4 servers

Avg set: 149 us (3304) min: 33 us (0) max: 19 ms (7)
Avg get: 154 us (6696) min: 27 us (6) max: 40 ms (7)
Usr: 0.115982
Sys: 0.263959
Tot: 6.18446744073708852052

Tokyo, on-disk hash, 25 threads, 4 servers

Avg set: 777 us (3262) min: 62 us (13) max: 43 ms (20)
Avg get: 742 us (6738) min: 49 us (14) max: 45 ms (16)
Usr: 0.141978
Sys: 0.354946
Tot: 5.638158
Server time:

Tokyo, in-memory hash, 25 threads, 4 servers

Avg set: 238 us (3326) min: 33 us (10) max: 25 ms (6)
Avg get: 215 us (6674) min: 34 us (1) max: 28 ms (18)
Usr: 0.096985
Sys: 0.289955
Tot: 5.461074

memcached, , 25 threads, 4 servers

Avg set: 195 us (3290) min: 32 us (24) max: 13 ms (21)
Avg get: 299 us (6710) min: 29 us (5) max: 115 ms (0)
Usr: 0.093985
Sys: 0.352946
Tot: 6.18446744073709011876

So, I decided to throw moxi (http://labs.northscale.com/moxi) into the mix! To start moxi against Tokyo Tyrant, I just ran it with a different port:

moxi -u patg -z 22122=127.0.0.1:1978,domU-12-31-39-07-85-01.compute-1.internal:1978,domU-12-31-39-07-7D-62.compute-1.internal:1978,domU-12-31-39-07-7A-62.compute-1.internal:1978 &

against moxi w/tokyo:

Moxi forwarded to 4 tokyo tyrant servers using on-disk hash, 10 threads
Avg set: 9087 us (3284) min: 137 us (0) max: 2343 ms (7)
Avg get: 2088 us (6716) min: 135 us (0) max: 1283 ms (9)
Usr: 0.106983
Sys: 0.282956
Tot: 6.466951

moxi forwarded to 4 tokyo tyrant servers using in-memory hash, 10 threads:

Avg set: 1059 us (3341) min: 101 us (0) max: 33 ms (0)
Avg get: 1056 us (6659) min: 95 us (0) max: 34 ms (2)
Usr: 0.085986
Sys: 0.216967
Tot: 6.041956

moxi forwarded to 4 memcached servers, 10 threads

Avg set: 1499 us (3254) min: 118 us (9) max: 31 ms (7)
Avg get: 1492 us (6746) min: 109 us (3) max: 31 ms (4)
Usr: 0.170974
Sys: 0.202969
Tot: 7.18446744073709056958

moxi fowarded to 4 tokyo tyrant sertvers, using on-disk hash, 25 threads:

Avg set: 93 ms (3316) min: 255 us (1) max: 1103 ms (0)
Avg get: 97 ms (6684) min: 261 us (1) max: 1105 ms (17)
Usr: 0.182972
Sys: 0.241963
Tot: 68.448877

moxi forwarded to 4 tokyo, using in-memory hash, 25 threads

Avg set: 2259 us (3244) min: 100 us (4) max: 34 ms (24)
Avg get: 2369 us (6756) min: 99 us (20) max: 37 ms (22)
Usr: 0.168974
Sys: 0.290955
Tot: 6.18446744073709448801

moxi fowarded to 4 memcached servers, 25 threads

Avg set: 3093 us (3326) min: 123 us (23) max: 30 ms (23)
Avg get: 3074 us (6674) min: 114 us (0) max: 30 ms (24)
Usr: 0.174973
Sys: 0.302953
Tot: 6.356237

So, as you can see from numerous tests, Tokyo Tyrant using on-disk hash store (TCHDB) is slower than either memcached or Tokyo Tyrant using in-memory. That stands to reason that memory is much faster than disk! However, depending on weather you want durability or not, that might be your deciding factor. The other interesting data here is that Tokyo in-memory hash is on par with memcached in terms of set and get average times. You would want to conduct a much more thorough test to really know one way or the other which might be faster. My goal here was not to compare necessarily but to just become acquainted with both Tokyo Tyrant and Tokyo Cabinet. What I'd like to test would be using replication to allow for the server your retrieve most of your data be in-memory and make the other node on-disk.

Overall, I think it's great that there's another Mem-capable KV storage to use! More choices are always beneficial to the developer!

Lastly, for fun, I decided to try out the memcached functions for MySQL with Tokyo Tyrant. They work just as you would hope they do. Just specify the connection:


mysql> select memc_stat_get_value('127.0.0.1:1978','version');
+-------------------------------------------------+
| memc_stat_get_value('127.0.0.1:1978','version') |
+-------------------------------------------------+
| 1.1.34                                          |
+-------------------------------------------------+
1 row in set (0.00 sec)

mysql> select memc_servers_set('127.0.0.1:1978');
+------------------------------------+
| memc_servers_set('127.0.0.1:1978') |
+------------------------------------+
|                                  0 |
+------------------------------------+
1 row in set (0.00 sec)

mysql> select memc_set('tokyo', 'hello Tokyo Tyrant!');
+------------------------------------------+
| memc_set('tokyo', 'hello Tokyo Tyrant!') |
+------------------------------------------+
|                                        1 |
+------------------------------------------+
1 row in set (0.00 sec)

mysql> select memc_get('tokyo');
+---------------------+
| memc_get('tokyo')   |
+---------------------+
| hello Tokyo Tyrant! |
+---------------------+
1 row in set (0.00 sec)