With the fast growth of virtualized data centers, and companies like Google, Amazon and Facebook, it's easy to forget how much is built on open-source components, aka commodity software.  In a very real way open-source has enabled the huge explosion of commodity hardware, the fast growth of the internet itself, and now the further acceleration through cloud services, cloud infrastructure, and virtualization of data centers.

Your typical internet stack and application now stands on the shoulders of tens of thousands of open source developers and projects.  Let's look at a few of them.

1. Operating System - Linux

The commodity hardware craze would never have happened without the help of an open-source operating system to run on it.  Linux is an old story now, nonetheless everything else stands on it's shoulders.

2. Multi-purpose Webserver - Apache

As of July …

If you're headhunting a cloud computing expert, specifically someone who knows Amazon Web Services (AWS) and EC2, you'll want to have a battery of questions to ask them to assess their knowledge.  As with any technical interview focus on concepts and big picture.  As the 37Signals folks like to say "hire for attitude, train for skill".  Absolutely!

1. Explain Elastic Block Storage?  What type of performance can you expect?  How do you back it up?  How do you improve performance?

EBS is a virtualized SAN or storage area network.  That means it is RAID storage to start with so it's redundant and fault tolerant.  If disks die in that RAID you don't lose data.  Great!  It is also virtualized, so you can provision and allocate storage, and attach it to your server with various API calls.  No calling the …

Modern internet infrastructure are complex.  Components and services are prone to failure.  Resiliency involves building redundancy, best practices and processes into your architecture to make you able to bend and not break.

• Migrating to cloud service providers
• Rearchitecting and refactoring applications to scale
• Scaling the database tier - MySQL and Oracle
• Building redundancy into every layer
• Deploying object caches - memcache
• Deploying page caches - varnish
• Migrating to Innodb - transactional storage engine
• Infrastructure design
• Infrastructure automation
• Disaster Recovery
• Business Continuity with cloud deployments

Call or Skype us in New York City +1-212-533-6828

MySQL databases are great work horses of the internet.  They back tons of modern websites, from blogs and checkout carts, to huge sites like Facebook.  But these technologies don't run themselves.  When you're faced with a system that is slowing down, you'll need the right tools to diagnose and troubleshoot the problem.  MySQL has a huge community following and that means scores of great tools for your toolbox. Here are 7 ways to troubleshoot MySQL.

1. Use innotop

Innotop is a great tool for MySQL which despite the name monitors MySQL generally as well as InnoDB usage.  It's fairly easy to install, just download the perl script. Be sure to include a [client] section to your local users .my.cnf file (you have one don't you?).  Inside that section, place one line with "user=xyz" and one line with "password=abc".

If you're concerned that installing something new is too …

1. Backup outside of the Cloud

Some of the high profile companies affected by Amazon's April 2011 outage could have recovered had they kept a backup of their entire site outside of the cloud.  With any hosting provider, managed traditional data center or cloud provider, alternate backups are always a good idea.  A MySQL logical backup and/or incremental backup can be copied regularly offsite or to an alternate cloud provider.  That's real insurance!

2. Use alternate regions and availability zones

Amazon's outage in April …

Call or SKYPE us in New York City: +1-212-533-6828

MySQL in the Cloud

• Deploying MySQL in the Cloud
• Cloud Security considerations for databases
• Deploying in multiple Availability Zones
• Deploying in multiple regions
• Deploying MySQL with multi-az
• Horizontal scalability in the cloud
• MySQL spinup scripts
• Deploying Amazon RDS
• Evaluating RDS
• Evaluating MySQL in the cloud
• EBS Performance considerations
• Tuning MySQL parameters in the cloud
• Backup types & options
• Mitigating against cloud server failure
• Vertical scalability & MySQL on EC2

MySQL Consulting and Services

• Backup & recovery
• MySQL master master replication
• MySQL replication & horizontal scaling …

Deploying in the Amazon cloud is touted as a great way to achieve high scalability while paying only for the computing power you use. How do you get the best scalability from the technology?

1. Use Auto-scaling

Auto-scaling is a unique feature of cloud computing and Amazon's EC2 offering. Setup a load balancer and a couple of webservers for your application as you normally would. Design your webserver based on a template AMI that you'll reuse over and over. Then setup auto-scaling and set thresholds based on the traffic you forecast. When a threshold is passed, AWS will spinup a new instance of your webserver, and roll it into the load balancer pool automatically. Once traffic falls below the scale back threshold, Amazon will take a server out of the pool for you.

Be sure to monitor this …

I hope everyone is enjoying summertime, at least in the northern hemisphere. I’m about to head out to the pool, but before I go, here is a summary of this week’s blog posts.

1. Introduction to Perl interface for Monitis API
Monitis announces a simple way to access its API through Perl, a high-level, general-purpose, interpreted, dynamic programming language. This post demonstrates some examples for using the API with Perl and describes some of the benefits of the programming language. The source can be found on our Github page.

2. 101 Tips to MySQL Tuning and Optimization …

I was at a conference recently talking with a Major Cloud Hosting Provider and mentioned that for database servers, I really want large instances, quite a bit larger than the largest I can get now. The lack of cloud servers with lots of memory, many fast cores, and fast I/O and network performance leads to premature sharding, which is costly. A large number of applications can currently run on a single real server, but would require sharding to run in any of the popular cloud providers’ environments. And many of those applications aren’t growing rapidly, so by the time they outgrow today’s hardware we can pretty much count on simply upgrading and staying on a single machine.

The person I was talking to actually seemed to become angry at me, and basically called me an idiot. This person’s opinion is that no one should be running on anything larger than 4GB of memory, and anyone who doesn’t build their system to be sharded …

In March I posted a series of blog posts on my paternity leave MepSQL project, which I called MepSQL. There was still one piece created in the MepSQL buildsystem that I didn't publish or blog about. Since it is generally useful, I wanted to generalize and polish it and publish it separately. I finally had that done last week, when I also found that somebody else, namely alestic.com already published a similar solution 2 years ago. So yesterday I ported my BuildBot setup to use that system instead and am happy to publish it at the Open DB Camp 2011 in Sardinia.

Ok, so let's go back a little... What is the problem we are solving?

Let's …