Working at Northscale has been a lot of great fun lately! I have
finally figured out how to get puppet to set up a stock Amazon
instance with everything we need and have been impressed with how
you can automate system setup with puppet. I remember when I
worked at Grazr how much of a hassle it was for us to have to
rebuild systems. Something like Puppet would have been a
godsend.
Today I though I would post about how cool moxi is. If you don't
know, moxi is a memcached proxy (http://labs.northscale.com/moxi/) which allows you
to move any complexity of having to set up the list of memcached
servers you are using. Also, moxi has some great features such
as:
* De-duplication of requests for popular keys
* A front cache, L1 cache to avoid network hops
* Fire and forget SET (Like an async SET) this means "set a
value, but don't wait to know if it was successful"
* Time-outs for setting the maximum time for operations
This post will show you how moxi can be used with the memcached
UDFs for MySQL (https://launchpad.net/memcached-udfs). It's
really quite amazingly easy. I'll even thrown in compilation of
moxi and setup of moxi.
First, obtain the source for moxi which can be found at:
git clone git://github.com/northscale/moxi.git
To build moxi, you will need both libevent and libmemcached. I'll
leave it to you to obtain both of those. For libevent, just use
the package for either centos or ubuntu. For libmemcached, use
the source because many packages available for it are somewhat
stale, and libmemcached is actively developed by Trond et
al!
Now enter the moxi source directory and do the following:
patrick-galbraiths-macbook-pro:moxi patg$ sh autogen.sh libtoolize... aclocal... autoheader... automake... autoconf... ./configure make sudo make install
At this point, moxi is ready to use. Now, depending where you
have memcached running, this will determine which servers you
proxy to. In this demonstration, I'm using 4 different memcached
servers:
* Two locally within my network
* Two Northscale memcached Amazon EC2 images (http://labs.northscale.com/memcached-ami/, ami
ami-bf4cadd6). These are great-- you just launch them and then
*use* them!
So, to launch moxi (I changed the hostnames to protect the
innocent)
moxi -z 33133=127.0.0.1:22122,192.168.1.118:11211,ec2-00-00-00-yy.compute1.amazonaws.com:11211,ec2-00-00-00-xx.compute-1.amazonaws.com:11211 -vvv
So, the syntax is quite simple:
moxi -z "port I want to answer to"="host:port server list I am proxying for"
In this example, just as you can also specify with memcached
itself, I'm running with the -vvv flag for verbosity because I'm
a geek and this is a great way to see what moxi is doing. Also,
I'm running so I'm using port 33133 to answer to.
So, I installed my UDFs on the latest version of MariaDB, which I
have installed in /usr/local/maria:
bzr branch lp:memcached-udfs cd memcached-udfs sh config/bootstrap ./configure --with-mysql=/usr/local/maria/bin/mysql_config --libdir=/usr/local/maria/lib/mysql/plugin/ make sudo make install mysql -u root test < sql/install_functions.sql
At this point, the UDFs are ready to use!
The convenient thing here is that moxi makes it so you don't have
to specify a long server list in the memc_servers_set()
UDF:
mysql> select memc_servers_set('127.0.0.1:33133');
+-------------------------------------+
| memc_servers_set('127.0.0.1:33133') |
+-------------------------------------+
| 0 |
+-------------------------------------+
Now, I was able to start storing and retrieving data via moxi to
my 4 memcached instances:
mysql> select memc_set('key1', 'test value');
+--------------------------------+
| memc_set('key1', 'test value') |
+--------------------------------+
| 1 |
+--------------------------------+
1 row in set (0.07 sec)
mysql> select memc_get('key1');
+------------------+
| memc_get('key1') |
+------------------+
| test value |
+------------------+
1 row in set (0.08 sec)
And of interest, observe the output of moxi:
cproxy_create on port 33133, downstream 127.0.0.1:22122,192.168.1.118:11211,ec2-72-44-62-67.compute-1.amazonaws.com:11211,ec2-67-202-38-195.compute-1.amazonaws.com:11211 cproxy_listen on port 33133, downstream 127.0.0.1:22122,192.168.1.118:11211,ec2-72-44-62-67.compute-1.amazonaws.com:11211,ec2-67-202-38-195.compute-1.amazonaws.com:11211 <40 server listening (proxy-upstream-ascii) <40 initialized conn_funcs to default <41 server listening (proxy-upstream-ascii) <41 initialized conn_funcs to default <41 cproxy listening on port 33133 <40 cproxy listening on port 33133 moxi listening on 33133 with 2 conns <42 new proxy-upstream-ascii client connection 42: going from conn_new_cmd to conn_waiting 42: going from conn_waiting to conn_read 42: going from conn_read to conn_parse_cmd <42 cproxy_process_upstream_ascii version >42 VERSION 0.9.6 42: going from conn_parse_cmd to conn_write 42: going from conn_write to conn_new_cmd 42: going from conn_new_cmd to conn_waiting 42: going from conn_waiting to conn_read <43 new proxy-upstream-ascii client connection 43: going from conn_new_cmd to conn_waiting 43: going from conn_waiting to conn_read 43: going from conn_read to conn_parse_cmd <43 cproxy_process_upstream_ascii set key1 0 0 10 43: going from conn_parse_cmd to conn_nread 43 pause_upstream_for_downstream 43: going from conn_nread to conn_pause assign_downstream cproxy_add_downstream 0 4 cproxy_create_downstream: 127.0.0.1:22122,192.168.1.118:11211,ec2-72-44-62-67.compute-1.amazonaws.com:11211,ec2-67-202-38-195.compute-1.amazonaws.com:11211, 0, 7 release_downstream check_downstream_config 1 check_downstream_config 1 assign_downstream, matched to upstream 43 <44 new proxy-downstream-ascii client connection <45 new proxy-downstream-ascii client connection <46 new proxy-downstream-ascii client connection <47 new proxy-downstream-ascii client connection 46: going from conn_pause to conn_mwrite assign_downstream, done 46: going from conn_mwrite to conn_new_cmd 46: going from conn_new_cmd to conn_waiting 46: going from conn_waiting to conn_read 46: going from conn_read to conn_parse_cmd <46 cproxy_process_a2a_downstream STORED 46: going from conn_parse_cmd to conn_pause >43 STORED 43: going from conn_pause to conn_write <46 cproxy_on_pause_downstream_conn 46 release_downstream_conn, downstream_used 1 1 release_downstream check_downstream_config 1 assign_downstream assign_downstream, done 43: going from conn_write to conn_new_cmd 43: going from conn_new_cmd to conn_waiting 43: going from conn_waiting to conn_read 43: going from conn_read to conn_parse_cmd <43 cproxy_process_upstream_ascii quit 43: going from conn_parse_cmd to conn_closing <43 cproxy_on_close_upstream_conn <43 connection closed. <43 new proxy-upstream-ascii client connection 43: going from conn_new_cmd to conn_waiting 43: going from conn_waiting to conn_read 43: going from conn_read to conn_parse_cmd <43 cproxy_process_upstream_ascii get key1 43 pause_upstream_for_downstream 43: going from conn_parse_cmd to conn_pause assign_downstream cproxy_add_downstream 0 4 cproxy_create_downstream: 127.0.0.1:22122,192.168.1.118:11211,ec2-72-44-62-67.compute-1.amazonaws.com:11211,ec2-67-202-38-195.compute-1.amazonaws.com:11211, 0, 7 release_downstream check_downstream_config 1 check_downstream_config 1 assign_downstream, matched to upstream 43 <48 new proxy-downstream-ascii client connection <49 new proxy-downstream-ascii client connection <50 new proxy-downstream-ascii client connection <51 new proxy-downstream-ascii client connection forward multiget get key1 (3 0) 50: going from conn_pause to conn_mwrite forward multiget nwrite 1 out of 4 assign_downstream, done 50: going from conn_mwrite to conn_new_cmd 50: going from conn_new_cmd to conn_waiting 50: going from conn_waiting to conn_read 50: going from conn_read to conn_parse_cmd <50 cproxy_process_a2a_downstream VALUE key1 0 10 50: going from conn_parse_cmd to conn_nread <50 cproxy_process_a2a_downstream_nread 0 200 50: going from conn_nread to conn_new_cmd <43 cproxy ascii item response success 50: going from conn_new_cmd to conn_parse_cmd <50 cproxy_process_a2a_downstream END 50: going from conn_parse_cmd to conn_pause <50 cproxy_on_pause_downstream_conn 50 release_downstream_conn, downstream_used 1 1 release_downstream 43: going from conn_pause to conn_mwrite check_downstream_config 1 assign_downstream
You can see (yes, this a lot of output) how moxi is talking to
the 4 different servers, a simple set and get of the key "key1"..
Also you see this concept of "downstream" and "upstream" -- to
where the data is actually being stored versus where it is passed
to moxi from, for instance a PHP or Perl client, or even another
moxi.
So, as you can see, using moxi with the memcached Functions for
MySQL (memcached UDFs), you gain even more convenience and
ability to store data to memcached. Also, you've seen mention of
Northscale's Amazon EC2 Machine Images (AMIs) and how you can
easily add more memcached instances for your application's needs.