I'll be going into detail what is Sharding, how to Shard, pitfalls of Sharding, performance/throughput gains, shard roles, and performance scaling in general. I hope to make this the most comprehensive talk to date on the subject in 45 min.

The topic is called Scaling a Widget Company. I'll detail how I setup the data layer for Rockyou. How many transactions per second Rockyou is at, what the infrastructure is comprised of, how 99.999% uptime is achieved and hopefully get into BCP which I probably will not have time to go over.

If you want me to focus on specific aspects on the subject of shard'ing let me know and I will :).

This is #3 in a series of blog postings about MySQL Performance Schema.

Today when I list the tables in PERFORMANCE_SCHEMA I see:

mysql> select table_name from information_schema.tables where table_schema = 'performance_schema';
+----------------------------------------------+
| table_name                                   |
+----------------------------------------------+
| cond_instances                               |
| events_waits_current                         |
| events_waits_history                         |
| events_waits_history_long                    |
| events_waits_summary_by_event_name           |
| events_waits_summary_by_instance             |
| events_waits_summary_by_thread_by_event_name |
| file_instances                               |
| file_summary_by_event_name                   |
| file_summary_by_instance                     |
| mutex_instances                              |
| performance_timers                           |
| …

Every BI installation has power users that just want “data dumps.” They may need the dumps for a variety of reasons:

• You’ve built crappy reports. They can’t get the information they need in *YOUR* reports.
• They need to feed the data into another system. They want to select all customers who bought product X in time period Y to send them a recall notice. Need a dump of email / addresses to send them the notice.
• They are addicted to Excel; they feel like a super hero whizzing through the data making fancy graphs and doing a few of their own ratios/calculations.
• They want to munge the numbers. They will export it to Excel, throw out the data that makes them look bad, and then present it to their boss with shiny positive results.

I had a customer who needed something to “feed the data to another system.” Their original approach was to write a Pentaho Report that formatted …

Sun's Software Stack - At the right place at the right timeThe technical industry has always had tremendous ebbs and flows. Just ask powerful or up and coming companies like Digital Equipment Corporation, Silicon Graphics, Informix, Netscape, Atari, and even Sun. Yet sometimes, "everything is in the timing".With the current economic environment, it is safe to say that all organizations are

In case you have not noticed, while I remain the Original Big DBA Head…  Yves Trudeau Has earn the title of Big DBA Head as well.  I have noticed that I was getting credit for a couple of his posts, so I wanted to set the record straight.   While Yves and I actually both started on the same day in the professional services group for MySQL, we are not the same person:)

In working with customers every week I am seeing some trends that I would like to share with everyone.There is a tremendous upswing in Oracle customers looking at MySQL. A year ago about half the customers were evaluating MySQL, now I am seeing about 90% saying they have been mandated to look at using MySQL for future DBA projects.Must stronger acceptance by the traditional Oracle DBA as to the

UFS has been the main filesystem on Solaris until the arrival of ZFS in 2004. UFS has been around since the SunOS 4.x days and is still quite widely used on Solaris. Although OpenSolaris uses ZFS by default, many database users have mostly used UFS.

UFS is a buffered filesystem because by default it uses free memory to cache data. While buffering is useful for a large number of applications, database users have generally stayed away from it by using DirectIO. If you are using UFS with MySQL, you have the option of using UFS buffered or UFS DirectIO. In this blog I will try to describe some of the reasoning behind why UFS DirectIO should be used with MySQL.

Buffered UFS Limitations (with MySQL/InnoDB)

UFS buffers filesystem data using 8K pages (4K on x86) in the buffer cache. Innodb stores records in 16K size pages and issues reads of size 16k. This corresponds …

UFS has been the main filesystem on Solaris until the arrival of ZFS in 2004. UFS has been around since the SunOS 4.x days and is still quite widely used on Solaris. Although OpenSolaris uses ZFS by default, many database users have mostly used UFS.

UFS is a buffered filesystem because by default it uses free memory to cache data. While buffering is useful for a large number of applications, database users have generally stayed away from it by using DirectIO. If you are using UFS with MySQL, you have the option of using UFS buffered or UFS DirectIO. In this blog I will try to describe some of the reasoning behind why UFS DirectIO should be used with MySQL.

Buffered UFS Limitations (with MySQL/InnoDB)

UFS buffers filesystem data using 8K pages (4K on x86) in the buffer cache. Innodb stores records in 16K size pages and issues reads of size 16k. This corresponds …

It’s been a while since I’ve sent any news about the MySQL Cacti Templates project I started a while ago. Here’s what’s new:

First of all, it’s not just MySQL templates anymore. It’s a generic framework and tools for creating templates, and I’ve written templates for Apache, Memcached, etc. I just haven’t publicized [...]

I was recently involved in a project where the main requirement was the smallest possible latency for queries. The queries were simple insert and update statements, the update being by primary key. The cluster was using regular Gigabits Ethernet and formed by 3 servers with 8 cores each, one Management node and 2 data nodes. Since throughput was not a concern, a design decision has been made to locate the MySQL daemons on the same servers as the data nodes, in order to save network hops. MySQL chooses the closest NDB data node as its transaction coordinators and the closest on will be reachable on localhost.

Apart from the architectural decision to host MySQL and the NDB nodes on the same boxes, we decided to benchmark the new real time options of NDB. The options are the following: