Had an interesting thought today, I wonder how much faster ( if at all ) updating via insert on duplicate key in bulk was compared to individual update statements.  The client  app I was looking at receives updates to a counter from various severs around the world and they are updating a row that contains time metrics.  For instance update traffic set visistcount = visistcount + 1000 where customer = 123 and hour = ‘12′ and date=’7/15/2008′.  These statements are driven by feeds from the various servers, so it lends itself to bulk operations very easily.   It seemed like batching these up will minimally save the parse and network travel time.

Basically instead of :

update bulk_update set val=10+val where val1 = 20;
update bulk_update set val=10+val where val1 = 21;
update bulk_update set val=10+val where val1 = 22;
update bulk_update set val=10+val where val1 = 23;

use …

Properties:

Description:

This cache is used by MyISAM to optimize bulk inserts. This cache is a special tree-like structure. Bulk inserts include statements like LOAD DATA INFILE..., …

I was at a client site with Yves Trudeau ( another MySQL consultant ) and the client had purchased a brand new top of the line 16 core server.  It is well documented in many places that scalability beyond 4-8 cores with innodb is less then optimal.  We were asked for a ballpark on the performance of a 16 cores vs an 8 cores, and specifically if their were any options to reduce the number of cores the mysqld process could use.   We decided to benchmark this using DBT2.   To do the test we ended up setting the CPU affinity of the mysqld process.  You can set this with the following command:  taskset.   Yves ended up trying the same DBT2 tests for 1-16 cores.    I won’t write too much about the scalability here, as I said their are way better resources out their that can explain it better, but what I wanted to do is post the results of …

While looking at the performance of multiple datafiles I noticed something kind of interesting.  A query searching for a integer based PK value that had the value escaped with single quotes was faster then another lookup on a PK without the single quotes.  That seemed a little counterintuitive… if anything I would expect escaping an integer to take a little longer.  But because they were searching for different keys, its not really all that meaningful, but i thought I would try and see just how much of an impact quoting your integer will cause.  So I used the same query tool I put together in perl to do the multi-datafile tests to run 2 queries searching for an integer PK.

See here:

mysql> select qryid, note, avg(averagequeryruntime),  count(*), querytext from Logger_single group by qryid, note; …

Over the past couple of months I have seen a few different clients with innodb setup to have multiple datafiles opposed to using file per table or a single datafile. While I generally do not think twice about a handful of datafiles, I started seeing clients with 50-100 or more 2-4GB datafiles on a single RAID set ( or in some cases a single disk ), and this started to raise the little alarms in my head. Asking these various clients the key question, why? The answers were surprisingly similar. Things like: “we always did it this way” , “their used to be a 2GB file limit”, “we did this in oracle”, etc. Their was not a ton of compelling answers ( like manageability and performance ).

Looking at these systems it seemed like they are really causing “self-induced” fragmentation. They all have large tables, and are doing scans over large amounts of the data… for instance a 40GB table in a database with 2GB datafiles. In the best …

After stepping off of the GROUP_CONCAT() solution for calculating quantiles I figured it would be nice to find a better way to calculate the median too.
Solution
I previously wrote on how to calculate the median using GROUP_CONCAT(), but I think that this is a better way:

SELECT     AVG(length) AS median                 -- take the average of left and right median
,          MIN(length) AS left_median            -- 
,          MAX(length) AS right_median           --
,          @l          AS left_median_position   --
,          @r          AS right_median_position  --
FROM (
     SELECT     @n, length                       -- @n is just here to …

My previous post on calculating percentiles with MySQL generated some comments and good discussion. In particular, I got some very interesting comments from Vladimir.

Basically, Vladimir was doubtful whether the GROUP_CONCAT() solution would be optimal in comparison to a JOIN. His proposal is to solve it like this:

SELECT   SUM(g1.r) sr
,        g2.length l
,        SUM(g1.r)/(SELECT COUNT(*) FROM film) p
FROM    (SELECT COUNT(*) r, length FROM film GROUP BY length) g1
JOIN    (SELECT COUNT(*) r, length FROM film GROUP BY length) g2
ON       g1.length < g2.length
GROUP BY g2.length
HAVING p > 0.9
ORDER BY p
LIMIT 1

First, this query sets up two identical subqueries in …

The mysql proxy project has tremendous potential to make mysql administration and usage easier. I decided to throw some load at it to get a feel for how stable and performant it is.

On EC2, I set up 6 “small” images in an example proxy setup:

- One client machine to run sysbench
- One machine to act as a mysql proxy machine, running 0.6.1 (FC4 binary)
- Four identical database servers, running mysql 5.0.45

The database configuration was largely default, with InnoDB configured for 64MB buffer pool (just enough to ensure the sysbench table was entirely in memory), 512MB log files, and 1024 max connections.

mysql-proxy was run with the following command:

mysql-proxy –proxy-backend-addresses=ip-10-251-66-63.ec2.internal:3306 –proxy-backend-addresses=ip-10-251-71-21.ec2.internal:3306 …

Testing MySQL sequential IO performance with different IO schedulers – Part 1

I stumbled upon an interesting article discussing the different Linux I/O schedulers that are available in the 2.6 kernel, and also discussing some of the deficiencies of the 2.4 “Linus elevator” I/O scheduler.

The default scheduler as of Kernel 2.6.18 is the “Completely fair queuing” scheduler, or CFQ. The previous default was the anticipatory scheduler between 2.6.0 and 2.6.17. Also available is a buffed-up version of the 2.4 scheduler called the deadline scheduler, and a “noop” scheduler. This article has a great discussion on what these schedulers mean for databases at a higher level.

Although old, …