My technology stack services more then 2 Million Daily Active
users. Its very basic. PHP talks to mySQL, Memcache,
RabbitMQ, Gearman and Facebook. Now that we have more Java
specifically to support our SmartFox Server and other
services, blurring the lines between what data is set
in PHP and what data is read in Java is very necessary.
Java J-Connect makes reading mySQL Data as simple IMHO as PHP's
PDO. What is hard is reading PHP's serialized format from
PHP's Memcache library.
In PHP there are two main C backed Libraries. There is
Memcache the original PHP library which
I happen to use, and Memcached which is the library I wanted to use but
didn't deploy since EC2 package system conflicted and cause
issues (I fixed them but to late to deploy). Memcache stores data
in PHP's serialized format and compresses it via ZLIB, while
Memcached can store data as PHP's serialized format, JSON,
Binary Serialized (which is rather awesome), JSON
Array Notation and has a multitude of compressing formats none of
which are pure ZLIB that I noticed.
Here is the problem. Spymemcached is a lib for talking to
memcache but can't unserialized PHP serialized format (or read
it natively and return a string) and cannot Decompress
ZLIB but can Decompress GZIP. Now a great speed up would be to
use PHP's serialized data set from PHP and share memcache
resources from PHP and Java just like what is done for the mySQL
resources.
What needs to be done? Well, build your own Transcoder for
Spymemcached. Fortunately Spymemcached documented an
interface to do just that.
What is needed. Implement the spymemcached Interface defined
here. Use org.lorecraft.phparser to
unserialize PHP data defined here. Return the Object.
Below is the code.
package com.schoolfeed.spymemcached;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.util.zip.InflaterInputStream;
import net.spy.memcached.CachedData;
import net.spy.memcached.compat.CloseUtil;
import net.spy.memcached.transcoders.BaseSerializingTranscoder;
import net.spy.memcached.transcoders.Transcoder;
import org.lorecraft.phparser.*;
public class PHPSerializedTranscoder extends BaseSerializingTranscoder implements Transcoder <Object> {
static final int COMPRESSED=2;
/**
* Get a serializing transcoder with the default max data size.
*/
public PHPSerializedTranscoder() {
this(CachedData.MAX_SIZE);
}
/**
* Get a serializing transcoder that specifies the max data size.
*/
public PHPSerializedTranscoder(int max) {
super(max);
}
/**
* decode the byte data from Memcache decompress it if necessary and return the Object
* @param CacheData - the byte data is turned into a object
* @returns Object
*/
public Object decode(CachedData d){
byte[] data=d.getData();
Object rv=null;
String ds="N;";
if((d.getFlags() & COMPRESSED) != 0) {
getLogger().debug("Looks like d is compressed");
data=decompress(d.getData());
}
ds=decodeString(data);
getLogger().debug("DECODED: [" + ds + "] about to SerializedPhpParser");
SerializedPhpParser sp = new SerializedPhpParser(ds);
try {
rv = sp.parse();
getLogger().debug("Parse was cool!!");
} catch(Exception e){
getLogger().debug("Not a PHP Object? : " + ds);
rv = ds;
}
return rv;
}
/**
* PHP Memcache stores compress data in ZLIB format override the base class decompress method to handle ZLIB
*
* @param byte array - raw data from Memcache
* @returns byte array
*/
protected byte[] decompress(byte[] in) {
ByteArrayOutputStream bos=null;
final int BUFFER = 2048;
if(in != null) {
ByteArrayInputStream bis=new ByteArrayInputStream(in);
bos=new ByteArrayOutputStream();
InflaterInputStream iis = null;
try {
iis = new InflaterInputStream(bis);
byte[] buf=new byte[BUFFER];
int r=-1;
while((r=iis.read(buf, 0, BUFFER)) > 0) {
bos.write(buf, 0, r);
}
} catch (IOException e) {
getLogger().warn("Failed to decompress data", e);
bos = null;
} finally {
CloseUtil.close(iis);
CloseUtil.close(bis);
CloseUtil.close(bos);
}
}
return bos == null ? null : bos.toByteArray();
}
/**
* encode -- not implemented yet
*
*/
public CachedData encode(Object o){
int flags = 0;
byte[] b=null;
return new CachedData(flags, b, getMaxSize());
}
/**
* no need to async Decode let's do it realtime
*/
public boolean asyncDecode(CachedData d) {
return false;
}
}
This is a stop-gap solution until we make the transition to
Memcached with JSON encoding. Then I can use Jackson-JSON - which is a fast JSON
encoder/decoder for Java enabling a great portable message
protocol between the two stacks and nearly any other language we
might add to the system (like Python).