At OpenSQL Camp in November we presented a challenge to insert one billion rows, maintaining indexes, into a MySQL table. The best results we have seen are:
| Who | Engine | Total Time | Terminal Rate | Notes |
|---|---|---|---|---|
| Mark Callaghan | InnoDB | 106.2 hours | ~1,800 |
Google Patches Tuned my.cnf parameters 10 disk SW RAID 0 |
| … |
In the last few months Sun has expanded significantly our online outreach efforts and we currently have (at least :-)) three Webinar series that cover the GlassFish products. Going through them:
|
The MySQL webinars are polished presentations that are broadcasted in high-quality and targeted at specific busines needs. Future presentations include Binod on MySQL and SailFin and Arun on … |
After suggestions by Sinisa on ways to improve my mysqldump progress patch I finally got around to working on these improvements.
This new patch has an extra parameter --show-progress-size which by default is set to 10,000. So when --verbose is used, every 10,000 lines you will get a status output of the number of rows for a particular table dumped. So what you should see is something along these lines:
shell> mysqldump -A --verbose >out.sql -- Connecting to localhost... -- Retrieving table structure for table testing... -- Sending SELECT query... -- Retrieving rows... -- 10000 of ~94347 rows dumped for table `testing` -- 20000 of ~94347 rows dumped for table `testing` -- 30000 of ~94347 rows dumped for table `testing` -- 40000 of ~94347 rows dumped for table `testing` ... shell> mysqldump -A --show-progress-size=1500 --verbose >out.sql -- Connecting to localhost... -- Retrieving table structure for table testing... -- …[Read more]
The Sun Tech
Days Singapore started earlier this morning - over 1100
developers, an outstanding audience!!!
The kick off had a good local flare when the Gods of
Longevity, Fortune, and Prosperity (Fu Lu Shou) showed up to
start the event ;-) The build up to their appearance was really
exciting as evident from the video below:
This particular event will also be recorded in Singapore Book of Records for the largest numbers
of Sun developers playing a rattle together :) Here are some
pictures from the event:
A Toshiba laptop and an iPod was …
Just before Christmas, I was working with a client that needed to
insert more than a billion rows into a NDB table. The cluster was
big,
14 nodes, and after a few hundred millions rows, we got the
following error:
2008-12-09 18:28:52 [ndbd] INFO --
dbacc/DbaccMain.cpp
2008-12-09 18:28:52 [ndbd] INFO -- DBACC
(Line: 5274) 0x0000000e
2008-12-09 18:28:52 [ndbd] INFO -- Error
handler shutting down system
2008-12-09 18:28:53 [ndbd] INFO -- Error
handler shutdown completed - exiting
2008-12-09 18:28:56 [ndbd] ALERT -- Node 10:
Forced node shutdown completed. Caused by error 2304: 'Array
index out of range(Internal error, programming error or missing
error message,
which looks like a bug in the NDB kernel. In fact, it is
not really a bug. This error is the …
A year ago this past week (on January 16, 2008), Sun Microsystems announced a historic deal to acquire MySQL AB for a billion dollars. Since then, MySQL-ers (aka Sun Dolphins) have settled in with Sun folks. While Sun has had an influence on MySQL, MySQL has also had an influence on several projects within "Sun Classic". Over this week, we hope to explore some of the projects around Sun which use MySQL and how the acquisition has helped bring us all closer together.
Today we talk to Detlef Ulherr and Thorsten Fruehauf, engineers working on Solaris Cluster and its Open Source sibling, Open HA Cluster.
I was just talking to someone who backs up their MySQL servers once a day with mysqldump, and I said in a catastrophe, you’re going to have to reload from a backup; that’s some amount of downtime, plus up to a day of lost data. And they said “We can just promote a replica, we’ve done it before. It works fine.” Granted, in some/many cases, this is fine. There are all sorts of caveats – for example, you either know that your slave has the same data as the master or you don’t care.
I wish I could say
printf("%*.16xs", n, p);
where n is a size_t and p is a void*,
and it would output n bytes as hex.
And I want to be able to say
printf("%*qs", n, p);
and it would output n bytes as a C-style backslash quoted
string.
Yes, I can write functions that do all this, but I have to worry
about buffers, allocate buffer space, and so forth. Plus its a
lot of noise and clutter, for just outputing potentially
unprintable strings, often just for logging and debugging.
Mark has found Double sysbench throughput with TCMalloc with is quite amazing.
In Drizzle, if TCMalloc is present, we’ve linked it in (and on Solaris, mtmalloc) for years now. well…. Drizzle years, which are rather like dog years.
Like a few other folks, I decided to play with iiBench and see
how fast I could insert 1B rows into an indexed InnoDB
table. Mark Callaghan published an excellent writeup of the theoretical
and practical limits when index size >> buffer
pool... but I bent the
rules a bit ;)
The platform I ran iiBench on has four dual-core Xeon 3.2Ghz,
internal 10k RPM 8 disk RAID 10, and 128GB of RAM. That's a lot,
I know, but it was built precisely to avoid InnoDB's
performance drop when an index does not fit in memory. It took
just 22 hours to run a single iiBench process and 10 hours to run
four processes in parallel (each doing 250m rows)! This was
achieved without modifications to the iiBench code, using
standard MySQL 5.1.30 binaries, the following …