I started playing around with protobuf when doing some stuff in Drizzle (more about that later), and since the examples where using IOStream, the table reader and writer that Brian wrote is using IOStreams. Now, IOStreams is pretty powerful, but it can be a pain to use, so of course I start tossing together some utilities to make it easier to work with.

Being a serious Perl addict since 20 years, I of course start missing a lot of nice functions for manipulating strings, and the most immediate one is join, so I wrote a C++ IOStream manipulator to join the elements of an arbitrary sequence and output them to an std::ostream.

In this case, since the I/O Manipulator takes arguments, it has to be written as a class. Recall that …

So, I am working on MemProxy some.  Mainly, I am trying to implement more of the Cache-Control header's many options.  The one that has me a bit perplexed s-maxage.  Particularly when combined with max-age.

s-maxage is the maximum time in seconds an item should remain in a shared cache.  So, if s-maxage is set by the application server, my proxy should keep it for that amount of time at the most.  Up until now, I have just been looking at max-age.  But, s-maxage is the proper one for a proxy to use if it is present.  I do not send the s-maxage through because this is a reverse proxy and, IMO, that is proper behavior for an application accelerating proxy.  However, I do send forward the max-age value that is set by the application servers.  If no max-age is set, I send a default as defined in the script.  Also, if no-cache …

As I said before, I was invited to be on a panel at Velocity Conference.  I was delighted to go.  I had never been to San Francisco.  I have been to Portland and Santa Clara several times.  The panel was great.  It was the Brian and photo sharing sites show.  Seriously, it was me (dealnews.com), John Allspaw of Flickr, Don MacAskill of SmugMug and Farhan Mashraqi of Fotolog.  Oh, there was also Shayan Zadeh of Zoosk, a social dating network and Michael Halligan, a consultant from BitPusher.  We all had similar ideas.  I told my …

I was sitting in a train in the middle of rainy Ireland when I received a mail that Nokia has bought Symbian and is releasing it as Open Source. I didn't believe a word of it. But the web was full of news about it, so it was true. This is an amazing turn of events that I didn't anticipate at all. (You may or may not know that in my previous job I was heavily involved with Symbian programming. Ironically, one reason I left just 6 months ago is that I wanted to work in an Open Source environment :-)

read more

I have been working on a set of efficient C++ classes for storing N-bit (N<=64) values at arbitrary bit offsets into a buffer. Essentially a way to address memory bit-wise, rather than the usual byte-wise or word-wise. The classes support either fixed-sized bitfield storage (eg. say 27-bit values), or compressed values where small numbers are stored in fewer bits than large numbers (for example 0-15 are stored in 6 bits, 16-2**24 are stored in 26 bits, and so on).

The basic idea is of course to save space when storing large amounts of data eg. for a relational database. Today most systems tend to pad much of their numbers to 32 or 64 or whatever bits, wasting space. This is done to allow efficient access, as it is much faster to access data when it is sized and aligned to one of the word sizes supported by the machine.

Or is it really?

Todays CPUs are so fast compared to I/O (disk, network) and even to main …

Well, neither did I until today. HA!

Velocity is a new O'Reilly conference dedicated to "Optimizing Web Performance and Scalability".  It starts next Monday.  Yesterday I was contacted by Adam Jacobs of HJK Solutions about taking part in a panel discussion about what happens when success comes suddenly to a web site.  I think he thought I was in the bay area.  Little did he know I am in Alabama.  But, amazingly, I was able to work it all out so I can be there.  I wish I had known about this conference ahead of time.  It sounds really awesome.  Performance has always been something I focus on.  I hope to share some and learn …

I am giving a talk titled "An Introduction to MySQL" here in Birmingham, AL on June 21, 2008 at 3PM.

I love living in Alabama.  I was born and raised in Huntsville.  However, Birmingham has always seemed a bit behind in technology compared to what I do for a living.  There is good reason.  The industry here is medical, banking, industrial and utilities.  I don't really want my doctors keeping my medical records in an alpha release of anything.  Same goes for my banking and utilities.  But, as this page shows, the companies here are catching up.  So, I am happy to present MySQL to as many people as I can in this town.  Hopefully I will help some folks that have not been exposed to MySQL or any open source for that matter.
…

MemProxy 0.1 is out!  It has taken me a while, but I have finally gotten around to releasing the code that I credited with saving us during a Yahoo! mention.  It is a caching proxy "server" that uses memcached for storing the cache.  I put server in quotes because it is really just a PHP script that handles the caching and talking to the application servers.  Apache and other HTTP servers already do a good job talking HTTP to a vast myriad of clients.  I did not see any reason to reinvent the wheel.  Here are some of the features that make it different from anything I could find:

• 
  
• Uses memcached for storage
  
• Serves cache headers to clients based on TTL of cached data
  
…

So, we had a cron job hanging for hours.  No idea why.  So, I started debugging.  It all came down to a call to in_array().  See, this job is importing data from a huge XML file into MySQL.  After it is done, we want to compare the data we just added/updated to the data in the table so we can deactivate any data we did not update.  We were using a mod_time field in mysql in the past.  But, that proved to be an issue when we wanted to start skipping rows from the XML that were present but unchanged.  Doing that saved a lot of MySQL writes and sped up the process.

So, anyhow, we have this huge array of ids accumulated during the import.  So, an in clause with 2 million parts would suck.  So, we suck back all the ids in the database that exist and stick that into an array.  We then compared the two arrays by looping one array and using in_array() to check if the value was in the …

SimpleXML is neat.  Some people don't think it is so simple.  Boy, use the old stuff.  The DOM-XML stuff.

Anyhow, one annoying thing about SimpleXML has to do with caching.  When using web services, we often cache the contents we get back.  We were having a problem where we would get an error about a SimpleXML node not existing.  We were caching the data in memcached which serializes the variable.  So, when it unserialized the variable, there were references in there to some SimpleXML nodes that we did not take care of.  Basically, a tag like:

<foo>bar</foo>

is a string.  But a tag like:

<foo></foo>

is an empty SimpleXML …