Observation on web page design during the Hacker Holiday:

It crossed my mind the other day that the web has moved to a design where you have a "so-so" intro page to the site, and a very customized personal page. This is not a new thought, I recognized this trend years ago, but thinking about it again this week got me to thinking more about cache design.

Slashdot uses a customizable front page, but most objects and design elements are reused. A few recent sites like Digg emulate this site design, but its just not as common any more for large sites. More often now sites use the homepage as a jumping off point and expect the user to spend more time in a "personalized page" where components will be less reusable (from a site wide viewpoint). Sites like Livejournal, FaceBook, MySpace, Tribe, represent the "personalized page" design.

This means? Sites like Slashdot optimize for a handful of pages, while the …

If you've followed this blog, you know that I feel Microsoft's Sharepoint is the future of Microsoft's enterprise lock-in. Intriguingly, an open source company, SocialText, has joined forces with Microsoft to help them achieve that goal even faster. Ross and company are even providing an easy migration from JotSpot (acquired by Google), just in case you wanted to expedite the lock-in. :-)

Ross is a smart guy and understands that open source is still too young to be linking up solely with open source companies. Microsoft, for its part, recognizes the future, and so has partnered with …

(note: see below for updates)

This is an idea I've been kicking around for a while and wanted some feedback. Memcached does an amazing job as it is but there's always room for improvement.

Two areas that memcached could be improved are local acceleration and large capacity support (terabyte range)

I believe this could be done through a "hierarchy of caches" with a local in-process cache used to buffer the normal memcached and a disk-based memcached backed by berkeley DB providing large capacity.

The infrastructure would look like this:


in-process memcached -> normal memcached -> disk memcached

The in-process memcached would not be configured to access a larger memcached cluster. Clients would not use the network to get() objects and it would only take up a small amount of memory on the local machine. Objects would not serialize themselves before …

After being "on hold" briefly for a variety of reasons, the Community Doxygen Project, which aims to "doxify" the MySQL server source code comments into standardized Javadoc commenting, is coming together.

David Shrewsbury will be leading the Doxygen Project for the community, with help from myself, Nicklas Westerlund, Frank Mash, Baron Schwartz, and Ronald Bradford. David has been working …

I am happy to announce the latest creation from foundeo: fusionKit.

fusionKit is a CD full of some handy ColdFusion components and UDF's. It is a similar concept to the DRK's that Macromedia used to sell, but is it's 100% ColdFusion.

My favorite component in the kit is the Bayesian CFC. You may recall that spam filters tend to use bayesian analysis to determine if a message is spam or ham. This CFC allows you to perform the same kind of analysis on any block of text. I am using this CFC in one of my client's blogs, and it has blocked over 4,000 spam comments in one week!

There is also the tagging CFC, which makes it easy to work with tags or …

I will be attending CodeMash in January (perhaps even as a speaker, as I put in a proposal for a session. It is a semi-local event for me, as it is in Sandusky, Ohio, which is about an hour and a half away from Columbus. Should be a great event for coders, and represents all platforms, not just Linux, so is bound to be a good event for networking with a variety of enthusiasts of all development platforms. Check it out!

My company has 9 production MySQL servers.

Our company does:

over 4 billion queries a week — an average of over 450,000 per machine, though in reality 2 servers do near 1 billion themselves, 5 do about the average, and 2 do much less (about 65k and 100k queries).

receive over 1380 GB (almost 1.35 TB!!) of data per week, an average of over 153 GB per server.

send out over 1400 GB of data per week, an average of 157 GB per server.

Our hardware is only somewhat beefy — 64-bit architecture, 3.20 GHz Intel(R) Xeon(TM) CPU, 6GB of RAM in 4 of the 5 most-used servers (4GB in the others).

We make over USD $220,000 per week ($10 billion per year) in sales for our web application.

If we bought the highest level of service, Platinum, for all 9 production machines, the cost would be 0.40% of our sales. The cost would be less than the cost of a new IT person (even a junior IT person!), and …

My company has 9 production MySQL servers.

Our company does:

over 4 billion queries a week — an average of over 450,000 per machine, though in reality 2 servers do near 1 billion themselves, 5 do about the average, and 2 do much less (about 65k and 100k queries).

receive over 1380 GB (almost 1.35 TB!!) of data per week, an average of over 153 GB per server.

send out over 1400 GB of data per week, an average of 157 GB per server.

Our hardware is only somewhat beefy — 64-bit architecture, 3.20 GHz Intel(R) Xeon(TM) CPU, 6GB of RAM in 4 of the 5 most-used servers (4GB in the others).

We make over USD $220,000 per week ($10 billion per year) in sales for our web application.

If we bought the highest level of service, Platinum, for all 9 production machines, the cost would be 0.40% of our sales. The cost would be less than the cost of a new IT person (even a junior IT person!), and …

Sometimes the NDBAPI is not perhaps the most straightforward. This might be useful if you are struggling with scan filters (filtering tuples in the ndbd kernel, like engine-condition-pushdown) in combination with strings. Please have a look at my previous blog about how to use strings in the NDBAPI.

If you are using string daratypes with the NdbScanFilter::COND_EQ then the following rules applies:

• VARCHAR and VARBINARY strings must have their length prepended (1 or 2 bytes)
• CHAR must be space padded
• BINARY must be null byte padded
  

However, for NdbScanFilter::COND_LIKE the following applies:

• VARCHAR and VARBINARY must NOThave the lenght prepended!
• CHAR must NOT be space padded
• BINARY must NOT be null byte padded

Many thanks to Pekka (MySQL) and Jim Dowling (MySQL) on this subject.

The test results that I have generated before (see my previous post) did not really help me in reaching my project goal, which is to make it possible to predict SQLbusRT's performance, reliability and scalability when it is used in an application.

In my new strategy I'm gonna take one step at a time to put the different components of SQLbusRT to the test.

My first upcoming test will involve sending messages back and forth between two processes, using the ORTE communication bus. This way, I can really measure ORTE's performance and reliability alone, without influence of MySQL. I will perform this test in different scenario's which will let me determine the scalability as well.

After having processed the results of these tests, I will include MySQL. Making these separate tests will let me determine the influence of ORTE and MySQL separately.

The new test results will of course be posted …