RDS has created some basic tooling for looking at the internal status variables within MySQL called Global Status History or "GoSH" though unfortunately the tooling is broken.

The tooling, detailed at the following link here is meant to snapshot the output of "show status" and squirt the data into a table in the MySQL schema called mysql.rds_global_status_history.

The issue is that the bootstrapped event called 'ev_rds_gsh_collector' is missing and therefore the procedures to enable the snapshotting fail.

It appears that whenever an RDS instance is started (or bounced) the contents of the 'mysql' schema is re-created - therefore the injection of the row into the event table is missing from whatever init script AWS has.





  [Read more...]

Read the original article at Cloud Operations Interview

What does a cloud computing expert need to know? How do you hire a cloud computing expert? Competition for operations & DBAs is fierce, so you’ll want to know how to find the best.

If you’re a systems administrator or ops guy, you may want to prepare for an interview for such a position. Meanwhile, if you’re a director of it or operations, a recruiter or manager in HR, you’ll want to have some idea how to find the right candidate.

Here’s my guide to do just that. You may

  [Read more...]

Read the original article at AirBNB didn’t have to fail

Today part of Amazon Web Services failed, taking down with it a slew of startups that all run on Amazon’s Cloud infrastructure. AirBNB was one of the biggest, but also Heroku, Reddit, Minecraft, Flipboard & Coursera down with it. Its not the first time. What the heck happened, and why should we care?

1. Root Cause

The AWS service allows companies like AirBNB to build web applications, and host them on servers owned and managed by Amazon.

  [Read more...]

I have seen a few posts on DBA.SE (where I answer a lot of questions) recommending the use of semi-synchronous replication in MySQL 5.5 over a WAN as a way to improve the reliability of replication. My gut reaction was that this is a very bad idea with even a tiny write load, but I wanted to test it out to confirm. Please note that I do not mean to disparage the author of those posts, a user whom I have great respect for.

What is semi-synchronous replication?

The short version is that one slave has to acknowledge receipt of the binary log event before the query returns. The slave doesn’t have

  [Read more...]

Read the original article at A CTO Must Never Do This…

A couple years back I was contacted to look at a very strange problem.

The firm ran flash sales. An email goes out at noon, the website traffic explodes for a couple of hours, then settles back down to a trickle.

Of course you might imagine where this is going. During that peak, the MySQL database was brought to its knees. I was asked to do analysis during this peak load, and identify and fix problems. Make it go faster, please!

First day on the job I’m working with a team of outsourced DBAs. I was also working

  [Read more...]

Yesterday (4/19) I attended the AWS Summit in NYC (http://aws.amazon.com/aws-summit-2012/nyc).

I'm a big fan and also a heavy user of AWS especially S3, EC2, and naturally, RDS. In every point in time I have several dozens of AWS machines running for me out there in the East region, and in some cases when we do some special benchmarks and tests, number of EC2 and RDS machines can easily reach 3-digit. As I said, I'm a fan...

A few quotes I was able to catch and document on my laptop, on my laps...:

"When you develop an app for facebook, you must be prepared (and be afraid) that to your party, not noone will show up, but everybody will show up!"

So true! Simple and true. We all want to succeed, to have success with our app. We have to think about scaling




  [Read more...]

Modern internet infrastructure are complex.  Components and services are prone to failure.  Resiliency involves building redundancy, best practices and processes into your architecture to make you able to bend and not break.

• Migrating to cloud service providers
• Rearchitecting and refactoring applications to scale
• Scaling the database tier - MySQL and Oracle
• Building redundancy into every layer
• Deploying object caches - memcache
• Deploying page caches - varnish
• Migrating to Innodb - transactional storage engine
• Infrastructure design
• Infrastructure automation
• Disaster Recovery
• Business Continuity with cloud deployments

Call or Skype us in New York City +1-212-533-6828

Deploying in the Amazon cloud is touted as a great way to achieve high scalability while paying only for the computing power you use. How do you get the best scalability from the technology?

1. Use Auto-scaling

Auto-scaling is a unique feature of cloud computing and Amazon's EC2 offering. Setup a load balancer and a couple of webservers for your application as you normally would. Design your webserver based on a template AMI that you'll reuse over and over. Then setup auto-scaling and set thresholds based on the traffic you forecast. When a threshold is passed, AWS will spinup a new instance of your webserver, and roll it into the load balancer pool

  [Read more...]

When I spoke at Percona Live (video here) on running an E-commerce database in Amazon EC2, I briefly talked about using RAID 10 for additional performance and fault tolerance when using EBS volumes. At first, this seems counter intuitive. Amazon has a robust infrastructure, EBS volumes run on RAIDed hardware, and are mirrored in multiple availability zones. So, why bother? Today, I was reminded of just how important it is. Please note that all my performance statistics are based on direct experience running a MySQL database on a m2.4xlarge instance and not on some random bonnie or orion benchmark. I have those graphs floating around on my hard drive in glorious 3D and, while interesting, they do not necessarily reflect real-life

  [Read more...]

In the press in the last two days has been the reported outage of Amazon Web Services Elastic Compute Cloud (EC2) in just one North Virginia data center. This has affected many large website includes FourSquare, Hootsuite, Reddit and Quora. A detailed list can be found at ec2disabled.com.

For these popular websites was this avoidable? Absolutely.

Basic scalability principles if deployed in these systems architecture would have averted the significant downtime regardless of your development stack. While I work primarily in MySQL these principles are not new, nor are they complicated, however they are fundamental concepts in scalability that apply to any technology

  [Read more...]

Found this video regarding Netflix's migration to Amazon's AWS cloud very informative. Enjoy!

Amazon's AWS team has published a series of whitepapers covering various scenarios for migrating into AWS cloud infrastructure. Links to these whitepapers are provided below for your convenience:

- Migrating applications to the AWS cloud
- Migrating web application
- Migrating batch processing applications
- Migrating backend processing pipelines

Cloud Application Architectures

Cloud Application Architectures. By George Reese, O’Reilly 2009. (Here’s a link to the publisher’s site).

This is a great book on how to build apps in the cloud! I was happy to see how much depth it went into. It’s short — 150 pages plus some appendixes — so I was expecting it to be a superficial overview. But it isn’t. It is thorough. And it is also obviously built on his own experience building very specific applications that he uses to run his business — he

  [Read more...]

Tomorrow, I'll be attending the Executive Seminar on Cloud Computing at NASDAQ MarketSite (NY). Speakers include Dr. Werner Vogels and Mårten Mickos (ex-CEO of MySQL). Big thanks to Amazon and RightScale who were able to accommmodate my RSVP even when the registration had formally closed.

I hope to be able to catch up with Mårten Mickos during the event. In case I do succeed in catching up, is there any question you want me to ask him? You can email me or post a comment.

It's funny that the event site still shows Mårten's title as "SVP of Sun Microsystems’ Database Group."

A lot of people are into the whole cloud computing scenario these days. However, no one has talked about offering DBA-like services in the cloud, all automated, so that startups don’t have to get their own DBAs.

Enter FathomDB. They are poised to offer databases as a service (maybe they’ll charge per database - so you can in theory run both WordPress and Mediawiki, if you prefix wp_ and mw_ in your table creation, for example). They are using MySQL. They’ve also taken the worry of running a database out - they will backup, they will setup (so you don’t have to issue GRANT commands :P), and they will also monitor your databases for you.

But what really takes the cake? The fact that they will also offer performance advisors. This totally reminds me of the MySQL

  [Read more...]

In high school, I had a great programmable calculator. I’d program it to solve complicated math and science problems “automatically” for me. Most of my teachers got upset if they found out, but I’ll always remember one especially enlightened teacher who didn’t. He said something to the effect of “Hey, if you managed to write software to solve the equation, you must thoroughly understand the problem. Way to go!”.

George Reese wrote up a blog post over at O’Reilly the other day called On Why I Don’t Like Auto-Scaling in the Cloud. His main argument seems to be that auto-scaling is bad and reflects poor capacity planning. In the comments, he specifically calls

  [Read more...]

For those of you who have been under a rock for the last several years, there is a buzz-phrase floating around—cloud computing. If you haven’t been paying attention, it is time to wake up.

While I could spend an entire blog post—if not several—on a definition of cloud computing, I will be talking only about cloud computing in the sense of companies moving servers from their building or network operations center to running virtual servers in this computing cloud.

While there are a number of companies providing virtual servers, the most visible is Amazon, with their Amazon Web Services (AWS). I will be talking about AWS in this post as it is the service with which I am most familiar. It seems like every month, AWS rolls out new options and services. Just recently Amazon announced that you can now run on AWS the Windows operating system along with

  [Read more...]

I've ported my AWS S3 storage engine to Drizzle.

The source is at bzr branch lp:~fallenpegasus/drizzle/awss3. Pull it and build it like you would the main Drizzle. The engine is built-in, no need to plugin load it.

Example use:

   CREATE TABLE colors
     (nam VARCHAR(32) NOT NULL PRIMARY KEY, val BLOB)
        ENGINE='AWSS3'
        CONNECTION='awss3 bucket_name aws_id aws_secret';
   SELECT val FROM colors WHERE nam='BlueViolet';

I will try to keep it tracking the main Drizzle dev tree.