Few months ago I’ve switched one of our internal projects from doing synchronous database saves of analytics data to an asynchronous processing using starling + a pool of workers. This was the day when I really understood the power of specialized queue servers. I was using database (mostly, MySQL) for this kind of tasks for years and sometimes (especially under a highly concurrent load) it worked not so fast… Few times I worked with some queue servers, but those were either some small tasks or I didn’t have a time to really get the idea, that specialized queue servers were created just to do these tasks quickly and efficiently.

All this time (few months now) I was using starling noticed really bad thing in how it works: if workers die (really die, or lock on something for a long time, or just start lagging) …

Karim Lakhani has put together a business case study on Wikipedia. It is worth noting that Wikipedia uses MySQL as its database engine. 

Now why would I do that, especially when my blog is syndicated through it?

Simple. I’m always trying to cut back on RSS and other “stuff.” Less is more. But don’t worry – I subscribed again. Only this time, I subscribed to my own filtered version of it! I built it via Yahoo! Pipes. And I made it public, so you can subscribe to it too.

One of the stated goals of the Drizzle project is to "reuse many eyeballs". I dislike "Not Invented Here", it breaks one of my primary rules that is "all engineers should be lazy".

By lazy I don't mean "don't do your work". Being an engineer means that you build stuff. If you aren't building stuff, then you are not an engineer.

Being lazy means that you reuse other people's work as much as possible. Skip re-inventing the wheel.

Sometime ago MySQL introduced a uuid() function into the server. It creates infinite numbers of keys for you, at the cost of creating a large footprint in your indexes. There is a trade-off in this, but I find people are willing to make it.

What was the problem?

We wrote our own UUID function instead of just inheriting the one that most systems provide. What does this lead to?

Code …

Continued from Part 2

Q: So what are some applications or prototypes you are actually working on? Which do you see as the most interesting ones? Can I do something useful with this today already?

In general, what I find most interesting are the use-cases that utilize the aspects that make a web-server on a mobile personal device unique. Use-cases that take advantage of the fact that the context - location, surrounding devices and people, etc. - constantly changes, and the fact that the web-site "administrator" is always there.

And I get all worked up when I think on the implications - even if I obviously don't know what they all might be - if all mobile phones were equipped with a globally accessible web-server (I ignore all technical challenges). For instance, we already have an implementation of …

I wrote an application that is able to send out 3-8 million messages an hour with only 10 CPU's. This application is a part of an Offline Task system that scales linearly.

How is this done, I'll go into detail hopefully at the mySQL conference if they accept my proposal.

The scope of this blog post is to go over building the "Task Queues". Currently I have 13 Queues, one queue for each Shard that I run. The data is federated by user or randomly federated with a GUID that lives as long as the job. A request came in to add 20 million jobs to the queue all at once. The problem is with this list, will I cause deadlocks in innodb as I add the jobs to the queue as one transaction? Can live traffic still write to this queue?

To verify that Deadlocks will not occur - having an understanding about how locks work in INNODB is key. I suggest reading …

Recently, I came across this short video clip by Nick Kloski where he briefly describes the advantages of running MySQL on Sun Servers (Linux, Windows, and Solaris) and details some of the best server options for MySQL and when they should be used.  There's also some details on how you can try the systems out before buying them.  Check it out here.

 If you would like more specifics than the above video, then there are some examples of customers deploying MySQL in Containers and / or LDoms on Sun systems at:

http://www.sun.com/systems/solutions/mysql/ecoscale/perspectives.jsp

Additionally, here a a few references to blogs from my colleagues that post specific details on tests that they have done.

Continued from Part 2

Q: So what are some applications or prototypes you are actually working on? Which do you see as the most interesting ones? Can I do something useful with this today already?

In general, what I find most interesting are the use-cases that utilize the aspects that make a web-server on a mobile personal device unique. Use-cases that take advantage of the fact that the context - location, surrounding devices and people, etc. - constantly changes, and the fact that the web-site "administrator" is always there.

And I get all worked up when I think on the implications - even if I obviously don't know what they all might be - if all mobile phones were equipped with a globally accessible web-server (I ignore all technical challenges). For instance, we already have an implementation of …

Continued from Part 2

Q: So what are some applications or prototypes you are actually working on? Which do you see as the most interesting ones? Can I do something useful with this today already?

In general, what I find most interesting are the use-cases that utilize the aspects that make a web-server on a mobile personal device unique. Use-cases that take advantage of the fact that the context - location, surrounding devices and people, etc. - constantly changes, and the fact that the web-site "administrator" is always there.

And I get all worked up when I think on the implications - even if I obviously don't know what they all might be - if all mobile phones were equipped with a globally accessible web-server (I ignore all technical challenges). For instance, we already have an implementation of …

Note that I’m talking about using these tools in some kind of professional way, and more specifically, I’m talking about using Excel as a database, and using VPS hosting to host “professional” web sites. By “professional”, I mean something other than your personal blog, picture gallery, or other relatively inconsequential site.

Excel is not a database

Here’s the thing: Excel isn’t a database. Most people who don’t work in IT don’t seem to understand this, and they’re deathly afraid to actually communicate with anyone in IT, so they take matters into their own hands, and create problems so big that IT is forced to get involved, because at some point this spreadsheet becomes “critical” to some business function. Then IT gets even more bitter toward the non-IT folk, validating some of the reasons the non-IT folk went that route in the first place, and virtually guaranteeing that they won’t come to the IT …