First feature to cover that’s dead easy, is the built in ability to generate a time dimension, including a Fiscal Calendar attributes. If you’re using Mondrian (or come to that, your own custom SQL on a star schema) you need to have a time dimension. Time is the most important dimension! Every OLAP model I’ve ever built uses one! It something that you, as a datasmith will need to do with every project; that’s why we’ve built it right into our database.

Here’s a dead simple way to create a fully baked, ready to use Time Dimension to use with Mondrian.

-- Create a view that is our time dimension for 10 years, with our
-- Fiscal calendar starting in March (3)
create view dim_time as select * from
table(applib.fiscal_time_dimension (2000, 1, 1, 2009, 12, 31, 3));

OK, that’s it. You’ve created a Time Dimension! * see NOTE at end of post.

So, we’ve created our time dimension, complete with a Fiscal calendar for 10 years in a single statement! Awesome - but what does it contain?

-- Structure of new time dimension
select "TABLE_NAME", "COLUMN_NAME", "DATATYPE" from sys_root.dba_columns
where table_name = 'DIM_TIME';
+-------------+---------------------------------+-----------+
| TABLE_NAME  |           COLUMN_NAME           | DATATYPE  |
+-------------+---------------------------------+-----------+
| DIM_TIME    | FISCAL_YEAR_END_DATE            | DATE      |
| DIM_TIME    | FISCAL_YEAR_START_DATE          | DATE      |
| DIM_TIME    | FISCAL_QUARTER_NUMBER_IN_YEAR   | INTEGER   |
| DIM_TIME    | FISCAL_QUARTER_END_DATE         | DATE      |
| DIM_TIME    | FISCAL_QUARTER_START_DATE       | DATE      |
| DIM_TIME    | FISCAL_MONTH_NUMBER_IN_YEAR     | INTEGER   |
| DIM_TIME    | FISCAL_MONTH_NUMBER_IN_QUARTER  | INTEGER   |
| DIM_TIME    | FISCAL_MONTH_END_DATE           | DATE      |
| DIM_TIME    | FISCAL_MONTH_START_DATE         | DATE      |
| DIM_TIME    | FISCAL_WEEK_NUMBER_IN_YEAR      | INTEGER   |
| DIM_TIME    | FISCAL_WEEK_NUMBER_IN_QUARTER   | INTEGER   |
| DIM_TIME    | FISCAL_WEEK_NUMBER_IN_MONTH     | INTEGER   |
| DIM_TIME    | FISCAL_WEEK_END_DATE            | DATE      |
| DIM_TIME    | FISCAL_WEEK_START_DATE          | DATE      |
| DIM_TIME    | FISCAL_DAY_NUMBER_IN_YEAR       | INTEGER   |
| DIM_TIME    | FISCAL_DAY_NUMBER_IN_QUARTER    | INTEGER   |
| DIM_TIME    | FISCAL_YEAR                     | INTEGER   |
| DIM_TIME    | YEAR_END_DATE                   | DATE      |
| DIM_TIME    | YEAR_START_DATE                 | DATE      |
| DIM_TIME    | QUARTER_END_DATE                | DATE      |
| DIM_TIME    | QUARTER_START_DATE              | DATE      |
| DIM_TIME    | MONTH_END_DATE                  | DATE      |
| DIM_TIME    | MONTH_START_DATE                | DATE      |
| DIM_TIME    | WEEK_END_DATE                   | DATE      |
| DIM_TIME    | WEEK_START_DATE                 | DATE      |
| DIM_TIME    | CALENDAR_QUARTER                | VARCHAR   |
| DIM_TIME    | YR                              | INTEGER   |
| DIM_TIME    | QUARTER                         | INTEGER   |
| DIM_TIME    | MONTH_NUMBER_OVERALL            | INTEGER   |
| DIM_TIME    | MONTH_NUMBER_IN_YEAR            | INTEGER   |
| DIM_TIME    | MONTH_NUMBER_IN_QUARTER         | INTEGER   |
| DIM_TIME    | MONTH_NAME                      | VARCHAR   |
| DIM_TIME    | WEEK_NUMBER_OVERALL             | INTEGER   |
| DIM_TIME    | WEEK_NUMBER_IN_YEAR             | INTEGER   |
| DIM_TIME    | WEEK_NUMBER_IN_QUARTER          | INTEGER   |
| DIM_TIME    | WEEK_NUMBER_IN_MONTH            | INTEGER   |
| DIM_TIME    | DAY_FROM_JULIAN                 | INTEGER   |
| DIM_TIME    | DAY_NUMBER_OVERALL              | INTEGER   |
| DIM_TIME    | DAY_NUMBER_IN_YEAR              | INTEGER   |
| DIM_TIME    | DAY_NUMBER_IN_QUARTER           | INTEGER   |
| DIM_TIME    | DAY_NUMBER_IN_MONTH             | INTEGER   |
| DIM_TIME    | DAY_NUMBER_IN_WEEK              | INTEGER   |
| DIM_TIME    | WEEKEND                         | VARCHAR   |
| DIM_TIME    | DAY_OF_WEEK                     | VARCHAR   |
| DIM_TIME    | TIME_KEY                        | DATE      |
| DIM_TIME    | TIME_KEY_SEQ                    | INTEGER   |
+-------------+---------------------------------+-----------+

-- Let's look at a few rows
select time_key_seq, time_key, yr, month_number_in_year, fiscal_year
, fiscal_month_number_in_year from dim_time;
+---------------+-------------+-------+-----------------------+--------------+------------------------------+
| TIME_KEY_SEQ  |  TIME_KEY   |  YR   | MONTH_NUMBER_IN_YEAR  | FISCAL_YEAR  | FISCAL_MONTH_NUMBER_IN_YEAR  |
+---------------+-------------+-------+-----------------------+--------------+------------------------------+
| 1             | 2000-01-01  | 2000  | 1                     | 2000         | 11                           |
| 2             | 2000-01-02  | 2000  | 1                     | 2000         | 11                           |
| 3             | 2000-01-03  | 2000  | 1                     | 2000         | 11                           |
| 4             | 2000-01-04  | 2000  | 1                     | 2000         | 11                           |
| 5             | 2000-01-05  | 2000  | 1                     | 2000         | 11                           |
| 6             | 2000-01-06  | 2000  | 1                     | 2000         | 11                           |
| 7             | 2000-01-07  | 2000  | 1                     | 2000         | 11                           |
| 8             | 2000-01-08  | 2000  | 1                     | 2000         | 11                           |
| 9             | 2000-01-09  | 2000  | 1                     | 2000         | 11                           |
| 10            | 2000-01-10  | 2000  | 1                     | 2000         | 11                           |
+---------------+-------------+-------+-----------------------+--------------+------------------------------+

Generating the Time Dimension is accomplished using DynamoDBs ability to include Java based UDF Table Functions. Table functions are really powerful - they allow a BI developer to write custom functions that output a “table” that can be queried like ANY OTHER TABLE (mostly). Check out the wiki page FarragoUdx if your interested.

And of course: download LucidDB and give it a whirl!

NOTE: To be candid, doing it as a view isn’t the best approach. For anything beyond tiny (5 million +) we should actually create the table, and do an INSERT INTO SELECT * FROM TABLE(fiscal_time_dimension).

SET @sourceString = 'a,b,c,d,e';
SET @sql = CONCAT('INSERT INTO t VALUES (\'', REPLACE(@sourceString, ',', '\'),(\''), '\')');
PREPARE myStmt FROM @sql;
EXECUTE myStmt;

Just to show what’s going on:

mysql> SELECT @sql;
+----------------------------------------------------+
| @sql                                               |
+----------------------------------------------------+
| INSERT INTO t VALUES ('a'),('b'),('c'),('d'),('e') |
+----------------------------------------------------+

What’s the “ideal” use case for the PBXT engine, and how does it compare in performance?  When would I use PBXT instead of a storage engine like MyISAM, InnoDB or XtraDB?

Unfortunately it is not possible to point to a specific category of applications and say, “PBXT will be better here, so try it”.  PBXT is a general purpose transactional storage engine, designed to perform well on a broad range of tasks, much like InnoDB.  However, PBXT’s log-based architecture makes performance characteristics different to both MyISAM and InnoDB/XtraDB. Tests show that PBXT’s performance is similar to InnoDB but, depending on your database designed and the application, it can be faster.

PBXT is a community project and, of course, we depend on users trying it out. In the long run, this will determine to what extent we are able to continue to develop and improve the engine.  So, despite this rather vague answer, we are hoping that more people try it out, and work with us to improve the engine as necessary.  My thanks to all who are already doing this!

I think I remember reports that PBXT (at an early stage) out performed InnoDB with INSERTS and UPDATES (but not SELECTS). That would make PBXT very interesting for non-SELECT-intensive applications (finance, production management etc.) in my opinion.  Is this the case, and do you have any recent benchmarks available?

This is no longer necessarily the case. For example a test (http://mysqlha.blogspot.com/2009/03/pbxt-is-fast-no-kidding.html) by Mark Callaghan shows that PBXT can actually out perform InnoDB with SELECTs under circumstances.

The implementation of full-durability has changed the performance characteristics of PBXT from “MyISAM-like” to more InnoDB-like. Originally PBXT was conceived as an engine that would be somewhere between MyISAM and InnoDB in both performance and features. The early version of PBXT was not fully durable (equivalent to innodb_flush_log_at_trx_commit=2).

A major change was completed at the beginning of last year with the implementation of full-durability. In doing this it was important to keep the log-based architecture which was the reason for the high write performance of earlier versions.

Traditional transactional implementations suffer from the problem that a backlog of asynchronous writes accumulate until it swamps the engine. There has been a lot of work on both InnoDB and XtraDB to solve this problem. The key words here are fuzzy and adaptive checkpointing (the former, originally implementation by Heiki for InnoDB, and the latter, an excellent addition to XtraDB).

Both methods improve the management of the asynchronous writes. The idea behind the log-based solution, on the other hand, is to avoid the accumulating a backlog of asynchronous writes, but writing synchronously.

Although write performance is comparable with InnoDB, I am not entirely convinced that PBXT’s implementation of the log-based I/O is optimal at this stage. This is ongoing work for PBXT 1.5.

Morgan notes: As well as Adaptive Checkpointing, Oracle has also been working on Adaptive Flushing for the InnoDB Plugin. The engine being ’swamped’ problem that Paul is referring to is best described visually – see this post for more info.

What were the hard decisions or trade-offs that you had to make when designing PBXT?

If you read the white paper from 2006 (http://primebase.org/download/pbxt_white_paper.pdf) you will notice that the original design was uncompromisingly MVCC-based.  Some of this has been changed to make PBXT more InnoDB-like, but other principles have remained.

Pure-MVCC does not do any locking. Read locks are not required because each transaction effectively gets its own snapshot of the database. And write locks are not acquired when updating. Instead, the application can hit a “optimistic lock error” if a record is updated by another user.

Now PBXT does acquire locks for 2 reasons: to support SELECT FOR UPDATE, and to avoid optimistic locking errors. This makes PBXT’s behavior identical to InnoDB in REPEATABLE READ mode.

On the other hand, there are currently no plans to implement InnoDB style “gap locking”. Gap locking effectively involves locking rows that do not exist. This, in turn, means that PBXT transactions are not SERIALIZABLE.  A result of this is that statement-based replication is not supported by the engine.

Another hard decision was not to implement clustered indexes which I mentioned in more details later.

How does online backup work in PBXT, and is incremental backup possible?

A recent version of PBXT (1.0.09) supports the MySQL Backup API which was originally implemented in MySQL 6.0. This feature is now scheduled for an upcoming version of MySQL 5.4.

The Backup API makes it possible to pull a consistent snapshot of an entire database even when tables use different engine types. The API does not yet support incremental backup, but this is planned.

Internally this feature is implemented by PBXT using an MVCC-based consistent snapshot.

Does PBXT have a maintenance thread like InnoDB’s main thread?

PBXT has several system threads, that are responsible for various maintenance tasks. The most important of these are the “Writer”, the “Sweeper” and the “Checkpointer”;

• The Writer transfers data from the transaction log to the database table files. This task runs constantly and is the main source of asynchronous I/O in the engine.
• The Sweeper is a thread unique to the PBXT architecture. It’s job is to clean up after a transaction. Basically it recognizes which record versions are no longer needed and deletes them from the database.
• The Checkpointer flushes the data written by the Writer to disk. It is also responsible for writing consistent snapshots of the index files to disk. The frequency of checkpoints, as with other engines like InnoDB, determines the recovery time.

Does PBXT support clustered indexes?

No, currently it does not. This is one of the original design decisions (as raised by a previous question).  Two things contributed to this decision:

• Supporting clustered indexes would have made the implementation of PBXT more complicated than I would have liked. My goal was to make PBXT as simple as possible, so that it is easy to maintain the code and add features.
• I believe clustered indexes are becoming less relevant with the rise of Solid State technology. As random read access times decrease clustering of data will become less and less important.

What is the page size in PBXT, and can it be tuned?

PBXT uses 16K pages for the index data and (approximately) 32K pages for the table data. Both sizes can be set using compile time switches. However, if the index page size is changed, then the indices need to be rebuilt, which can be done by REPAIR TABLE.  The table data page size does not require a rebuild because a page of records in the table is just a group of records (not an actual fixed length page).

Are there any differences in the PBXT implementation of MVCC that might surprise experienced InnoDB DBAs?  Also – In MVCC does it keep the versions in indexes, and can PBXT use MVCC for index scans?

If you are using InnoDB in REPEATABLE READ mode, then there is essentially no difference in the isolation paradigm between the two engines.

REPEATABLE READ is often preferred over SERIALIZABLE mode because it allows a greater degree of concurrency while still providing the necessary transaction isolation. So I do not consider the lack of serializability as a serious deficit in the engine. And, fortunately MySQL 5.1. supports row-based replication which makes it possible to do replication while using REPEATABLE READ.

PBXT does use MVCC to do index scans. Basically this means that all types of SELECTs can be done without locking.

Morgan notes: Indexes not using MVCC is one of the main differences in the Falcon storage engine.

PBXT supports row-level locking and foreign keys. Does this create any additional locking overhead that we should be aware of?

Firstly, PBXT does not acquire read locks. A normal SELECT does not lock at all. In addition, an UPDATE or DELETE only acquires a temporary row-lock. This lock is released when the row is updated or deleted because the MVCC system can detect that a row has been changed (and is therefore write locked).

This means that PBXT does not normally need to maintain long lists of row-level locks. This is also the case when a foreign key leads to cascading operations which can affect thousands of rows.

The only case you need to be aware of is SELECT FOR UPDATE. This operation acquires and holds a row-level lock for each row returned by the SELECT. These locks are all stored in RAM. The format is quite compact (especially when row IDs are consecutive) but this can become an issue if millions of rows are selected in this manner.

I should also mention that the consequence of this is that SELECT … LOCK IN SHARE MODE is currently not supported.

When I evaluate a storage engine my key acceptance criteria are things like backup, concurrency, ACID compliance and crash recovery. As a storage engine developer, what other criteria do you think I should be adding?

Yes, I think you have mentioned the most important criteria. What I can add to this list are 3 things that make developing a storage engine extremely demanding: performance, stability and data integrity.

Of course, as a DBA or database user these aspects are so basic that they are taken for granted.

But engine developers need to keep performance, stability and data integrity in mind constantly. The problem is, they compete with each other: increasing performance often causes instabilities that then have to be fixed. How to optimize the program without compromising data integrity is a constant question.

Relative to maintaining performance, stability and data integrity, adding features to an engine is easy. So I would say that these are the criteria that concern a developer the most.

MySQL supports a “pluggable storage engine API”, but it seems that not all the storage engine vendors are able to keep all their code at that layer (Infobright had to make major changes to MySQL itself).  What war stories can you report on in plugging into MySQL?

Unfortunately the “war” continues. I have already received several e-mails that PBXT does not compile with the recently released MySQL 5.1.41!

Any dot release can lead to this problem, and I think PBXT is fairly moderate with its integration into MySQL.

My main advantage: I have been able to avoid modifying any part of MySQL to make the engine work. This means that PBXT runs with the standard MySQL/MariaDB distribution.

But this has required quite a bit of creative work, in other words, hacks.

One of the main problems has been running into global locks when calling back into MySQL to do things like open a table, create a session structure (THD) or create a .frm file.

One extreme example of this is PBXT recovery. When MySQL calls the engine “init” method on startup it is holding the global LOCK_plugin lock. In init, the engine needs to do recovery. In PBXT’s case this means opening tables (reading a .frm file), which requires creating a THD. The code to create a THD in turn tries to acquire LOCK_plugin!

Unfortunately a thread hangs if it tries to acquire the same mutex twice, so this just does not work!

We went through quite a few iterations (MySQL code was also changing during the development of 5.1) before we came up with the current solution: create a background thread to do recovery asynchronously. So the thread can wait for the LOCK_plugin to be unlocked before it continues.

The affect is that the init function returns quickly, but the first queries that access PBXT tables may hang waiting for recovery to complete.

PBXT seems to have very few configuration parameters.  Was this an intentional design decision, and do you see it creating opportunities for you in organizations with less internal IT-expertise?

No, this is not by design.

While I try to only add tuning parameters that are absolutely necessary, PBXT is not specifically designed to be self-tuning, because I believe that is a very hard problem to solve in general.

Tuning parameters are often added to an engine in response to performance problems in particular configurations. This is not necessarily a bad thing because it provides DBA’s with the tools they need.

My goal for PBXT in this regard is twofold:

1. I hope that most installations can get away with setting a minimum of tuning parameters, in particular, just the cache values.
2. On the other hand, I aim to provide expert tuning parameters for installations that need to extract maximum performance from the hardware. This work is ongoing.

Morgan notes: There are more in InnoDB/XtraDB now than there were three years ago. This is probably something that emerges over time as we get to understand more about an engine.

Entry posted by Morgan Tocker | No comment

Add to: | | | |

Savio makes a good case based on the recent launch of AWS’s Relational Database Service, based on MySQL, and Microsoft’s support for MySQL and Tomcat on Azure:

“When Amazon decided to offer MySQL via Amazon RDS, they did so without purchasing MySQL support from Sun. I’ve confirmed that Microsoft Azure is supporting MySQL on Azure without paying Sun for a MySQL Enterprise subscription.”

Clearly there is a threat to open source vendors from cloud-based services. Meanwhile I have previous argued that the cloud and open source are complementary. Can both positions be valid?

I believe so, and I think it’s important to look at the technologies involved. Certainly, the ability of cloud platform providers to provide services based on infrastructure components such as MySQL and Tomcat threatens potential support revenues for on-premise deployments, but SugarCRM’s launch of CRM Applications on Windows Azure proves that just because the code is open source, does not mean that the cloud platform provider will automatically cut the vendor out of the picture.

Perhaps the difference with SugarCRM is that it is application, rather than infrastructure. Perhaps it is also the fact that SugarCRM has been proactive about exploring on-demand and cloud delivery models.

One of the reasons AWS was able to deliver the a managed MySQL service on EC2 was, frankly, because Sun had not already done so. All the realtional database vendors made their products available as AMIs on AWS in 2008 and since then they have done almost nothing about innovating delivery options abound those AMIs.

Had Sun launched MySQL-as-a-service on EC2 it could have grabbed the market share that AWS will now grab with RDS. I’m not sure why Sun failed to do this, incidentally. FathomDB did it, although it lacked the market presence to prevent AWS stealing the limelight. I would argue that Sun/MySQL could have done so.

Savio argues that open source specialists faced with this dilemma should double-down on their investments in “proprietary features in the ‘enterprise version of the open source product, which are note available in the ‘free community’ version”.

Certainly that is one opportunity for differentiation, but I would also argue that open source specialist vendors should also be concentrating on working with cloud platform providers to bring managed service deployments to market before the platform providers beat them to it.

I think in the long-term we’ll see more vendors providing open source software for on-premise deployment while offering enterprise versions via paid managed service deployments on cloud platforms, along with services to help customers migrate their data/applications from one deployment option to the other.

And if you’re wondering why a cloud provider would bother working with an open source specialist vendor, rather than just taking their code, consider this: one of the cloud providers mentioned in this post pays for enterprise Linux support subscriptions rather than using a community Linux or supporting its Linux servers internally. And it isn’t Microsoft.

Cloud computing is undeniably a threat to the monetization of open source software, but it is also an opportunity. Be quick or be dead.

SQL Server

There are lots of good technical posts this week. The SSIS Junkie has some observations and a straw poll on sort transform arbitration. He writes, “This post was prompted by a thread on the MSDN SSIS forum today where the poster was asking how he could replicate the behaviour of SSIS’s Sort transform using T-SQL, specifically he wanted to know how the Sort transform chooses what data to pass through when the ‘Remove Duplicates’ option is checked.”

Another poll, courtesy of Tibor Karaszi: do you perform log backup for the model database?

Eric Johnson has a lesson in looping through rows in a table in SSIS 2008, which begins, “When writing code against a SQL Server, as we usually are doing in SSIS Packages, you often need to iterate over all the rows in a table. This can be done using an SSIS Foreach Loop Container, but the how is not obvious.”

Simple-Talk’s Tony Davis wonders, Do Scalar UDFs give SQL Server a Bad Name? “Many developers seem to regard SQL Server as if it were a science-fiction alien planet where unsuspecting crew-members in blue jumpers occasionally die horribly; everything is suddenly unsafe, and potentially malicious: nothing really works properly and so any serious code should be kept well away from it. Is this developer ignorance, or is their fear justified?”

From Merrill Aldrich comes a trick question — part quattro. Spoiler: TPH is an evil trap. As a commenter says: “Very interesting, I’ve never heard this vehement an argument against TPH before.”

Thinking outside the box lets us in on how to tell if you are running on a virtual environment, with a handy little bit of code.

Kalen Delaney elucidates UPDATE Locks, “ . . . a hybrid of SHARED and EXCLUSIVE locks. [Contrary] to what you might think, UPDATE locks are not just acquired for UPDATE operations.”

Stephen Forte shows how SQL Server R2 Does SQL Azure.

Now that PASS 2009 in Seattle, Washington has passed, it’s time to fondly look back on it. Kendal Van Dyke shares his experiences and some photos in Looking Back – PASS Summit 2009 Day 4.

The Rambling DBA, Jonathan Kehayias does so with the benefits of attending PASS realized: “ . . . as a testament to the value of attending this conference the very first session I attended, diagnosed and provided information for a problem that has existed in one of my servers for many weeks but was impossible to diagnose unless you knew what you were looking at.”

Thomas LaRock, SQL Rockstar concurs: “If you want to grow your skills, then you need to connect, learn, and share with others. And there is no better place to do that than at PASS.”

Not that PASS was the only game in town. Just down the coast a bit, Baron Schwartz gives his recap of Portland OpenSQL Camp 2009.

Selena Marie Deckelmann reports that OpenSQLCamp was awesome!

PostgreSQL

Pavel Stehule has some news of a longtime plpgsql misfeature removed. He writes, “plpgsql is good language – simple, robust with good error diagnostic. But it had one bizarre behave. plpgsql connects two worlds – procedural ADA like code and SQL statements. Usually there are not problems. But there are one exception – collision of identifiers.”

In the latest in his Waiting for 8.5 series, Hubert Lubacziewski introduces and tests something new: TRIGGERS on columns.

MySQL

On code.openark.org, Shlomi Noach is surprised by questions or queries. He writes, “I was used to checking for the ‘questions’ global status variables . . .  So, for example, I could run com_select/questions to learn the SELECT ratio out of all queries.  . . .  Apparently, as of 5.0.72-5.0.76 & 5.1.31 this has changed. A new status variable was introduced, called ‘queries’.” What’s the difference? Is this good or bad? Shlomi and his readers kick it around.

Venu Anuganti also was surprised, in his case by InnoDB Tablespace Corruption: “When . . . InnoDB crashes, it automatically recovers during the next start by rolling back/forward based on what was pending and un-flushed/un-committed changes at the time of crash.  . . .  [On] one of the servers; we ran out of disk space . . . on data directory . . . and server was running for few hours in this mode . . .  [It] became un-available and not responding after a while. Only option left was to kill the server process and its PID along with cleaning the stuff to get the space back. After I (re)started the server . . .  the tablespace is corrupted.”

From Arnold Daniels comes a version of versioning MySQL data, which Arnold introduces thus: “ . . . You’re probably using a versioning control system . . . to safeguard your data. Advantages of using a VCS are that you can walk to the individual changes for a document, see who made each change and revert back to specific revision if needed. These are features which would also be nice for data stored in a database. With the use of triggers we can implement versioning for data stored in a MySQL db.” Example code follows.

On the MySQL Perfomance Blog, Morgan Tocker qualifies his earlier piece on why you don’t want to shard: “What I didn’t mention was that if you’ve established that you will need to eventually shard, is it better to just get it out of the way early? My answer is almost always no. That is to say I disagree with a statement I’ve been hearing recently; ’shard early, shard often’”

Morgan also mentions that interviews for InfiniDB and TokuDB are next: “I’d like to announce that Robert Dempsey (InfiniDB storage engine) and Bradley C. Kuszmaul (TokuDB storage engine) have also accepted an interview. If you have any questions about either storage engine, please post them here by Friday 20th November.”

Here’s a new blog to watch, particularly if you’re new to MySQL or Drizzle: Kent Bozlinski’s Learning Drizzle. Kent says, “I’m not really scared of rain after living in Seattle for seven years. I am a little scared of sticking my neck out and writing about something which (for the moment) I know almost nothing about.” The post is Drizzle is Scary (A Little).

Oracle

Maybe you’ve already heard about the fabulous unpopularity of the new My Oracle Support. Daniel Fink comes back with some data and commentary on just that, with his My Oracle Support Survey Results.

Richard Foote asks, An index only performs how much work???, the result of looking into exactly why index rebuilds can improve performance so significantly.

Kerry Osborne gives a lesson on fixing bad index hints in SQL Profiles (automatically). He says, “With 10g and 11g, it appears the goal [or Outlines] has swung away from the “locking” concept and towards allowing the optimizer more flexibility.  . . . I must say that I find this decision to be irritating at best.  . . .  One of the main offenders in this regard is the use of a new format available for index hints as of 10g.”

Inside the Oracle Optimizer covers similar turf in answering to the question, What should I do with old hints in my workload?, or more specifically, “When moving from 10g to 11g, should hints in existing SQL be removed?”

Tom Kyte wants your opinions on comparative window functions: “ . . . they could be getting better in the near future.  . . . analytics [could be allowed] to access the current row value to be compared against any other row value in a defined window.  . . .  I’ve already supplied them with my feedback (which started with “this is an awesome idea”) – and you can too – by posting it here. They’ll be checking back to see what you say.”

If you like staying on top of fresh things, perhaps John Piwowar’s method of retrieving Oracle patches with wget would also appeal to you.

And that is all for now. If you think I’ve missed a worthwhile DB blog from this week, please mention it in a comment.

Log Buffer will be back in a week’s time. See you then!

Follow 451 CAOS Links live @caostheory on Twitter and Identi.ca
“Tracking the open source news wires, so you don’t have to.”

For the latest on Oracle’s acquisition of MySQL via Sun, see Everything you always wanted to know about MySQL but were afraid to ask

# Google launched the Chromium OS open source project, a prelude to the Chrome OS, while Canonical confirmed that it is contributing to the development of Chrome OS.

# Terracotta acquired the Quartz open source job scheduling and workload management software project.

# The European Commission extended its Oracle/Sun review deadline until Jan 27 at the request of Oracle

# Mozilla revealed its 2008 revenue was $78.6m, up 5%. Expenses were $49.4m, up 48%.

# Roberto Galoppini wrote about open source value creation and consumption.

# Exadel and Actuate formed a strategic alliance to promote the use of BIRT in enterprise applications.

# Savio Rodrigues explained how Microsoft Azure, like Amazon’s Web Services, will capture open source revenue streams.

# MuleSoft released Mule Data Integrator for data mapping and transformation.

# MindTouch launched a version of its collaboration platform for the cloud.

# Tristan Renaud, on software pricing, asked to hide or not to hide?

# Mik Kersten published growing open source ecosystems: the install story.

# CodePlex Foundation announced its first gallery/project - The ASP.NET Ajax Library.

# Zenoss released version 2.5 of its commercially licensed Zenoss Enterprise product.

# ActiveState launched Firefly, a hosted project management and collaboration offering based on Trac.

# Opengear released Opengear Monitor, a new centralized monitoring system based on Nagios.

# Infobright and Talend teamed up with Jaspersoft for open source data warehousing/integration/BI appliances.

# WaveMaker launched WaveMaker 6.0, pitching it as an open source cloud development platform.

# Vyatta partnered with OpenVPN on an auto-configuring VPN offering for branch offices and remote workers.

# Matt Asay explained why pro-open source policies do not always mean more open source.

# Microsoft announced Windows Azure Tools for Eclipse and SDKs for PHP and Java.

# The European Space Agency (ESA) wants to build a repository for hosting and developing it’s open source applications.

# SugarCRM announced its CRM Applications will available be on Windows Azure.

# Linux.com published an interview with Tim Golden, Open Source Software Infrastructure Strategist at Bank of America.

# Roman Stanek raised the issue of openness in commercial open source software pricing.

# Tarus Balog raised some interesting questions about Zenoss Core/Enterprise licensing.

So, why I'm writing now? ;-) Just because I'm pretty sure there will be a second session! :-) I cannot say you date & place yet because it'll directly depend on the number of persons willing to attend. And if you have a such willing, please, right now express it by email to Olivier.Beutels(at)sun.com !

List of topics covered during Performance Session:

• MySQL history/development milestones
• MySQL Architecture overview
• MySQL Storage Engines overview (Pros/Cons)
  - MyISAM
  - InnoDB
  - NDB
  - PBXT
  - XtraDB
  - Falcon

• MySQL Bottlenecks:
  - Data model/design
  - Multi-threaded model
  - Storage engine
  - Internals
  - etc.
  
  
• MySQL Performance:
  - Solutions
  - Scalability
  - etc.
  
  
• InnoDB Performance:
  - Bottlenecks
  - Internals
  - Google SMP patches
  - XtraDB
  - etc.
  
  
• MySQL 5.4
  - Improvements
  - Internals
  - I/O capacity
  - I/O threads
  - Thread concurrency model
  - Benchmark results
  - etc.
  
  
• New / Incoming InnoDB improvements:
  - Ahead /Adaptive flushing /checkpoint
  - Purge lag
  - Splitted locks
  - Multiple rollbacks
  - etc.
  
  
• MySQL Performance Tuning:
  - Monitoring
  - Configuration
  - Best practices
  - DTrace
  - ZFS
  - etc.
  
  
• MySQL & dim_STAT
  - mysqlSTAT
  - mysqlLOAD
  - innodbSTAT
  - innodbIO
  - innodbMUTEX

After I left Pythian I was looking at a couple of other places to begin working with, but finally settled on staying in the mediterranean region, however with a small relocation.

Thus, this post is now long overdue, but I am working for a spanish company called Tuenti, which is the largest social network in spain. I am quite excited and have now settled down in Madrid, so I will be using this website in order to blog about challenges, problems, solutions and everything that comes with running a large scale deployment of MySQL.

Stay tuned for the first technical post, it’s on its way already!

It’s not about MySQL versus PostgreSQL, it’s about using the right tool for the job.

Or how about this one?

You need to take the important factors into account before you decide whether [hot new fad] or [trusty old solution] is best suited for your application.

Both are signs that someone might be trying to sound important. In situations like this, I’ve noticed that the people I look up to usually don’t make weighty-sounding statements about other people’s systems. They talk about what they are qualified to talk about: either they say something about their own systems, or if it’s warranted and invited, they ask intelligent questions about other people’s systems.

People who only have vacuous generalities to contribute don’t talk about their own systems, because if they actually worked on systems that qualified them to contribute to the conversation, they’d have something of substance to say. And they don’t ask questions, because they don’t know what to ask. (The third option — say nothing — seems unacceptable, I guess.)

For the love of all that is good, please do not look back through my blog archives. I don’t want to know.

Related posts:

1. Recap of Enterprise LAMP Summit and Camp Last week
2. Seeking input on a badness score for query execution Suppose th
3. Migrating US Government applications from Oracle to MySQL I just ret

Related posts brought to you by Yet Another Related Posts Plugin.

Adding the following to the /etc/sudoers file enabled the ability to restart MySQL.

User_Alias	DBA = rbradfor, user2, etc
Host_Alias 	DB_SERVERS = server1.example.com, server2.example.com, etc
Cmnd_Alias	MYSQL = /etc/init.d/mysqld, /usr/sbin/tcpdump

DBA DB_SERVERS = MYSQL

As you can see I also got tcpdump, which I find valuable to monitor via mk-query-digest.

Next, permissions for log files.

This particular benchmark has two favorite parts, that go with each other together really well:

I didnt change absolutely any parameters for the servers, eg didn’t change the innodb_buffer_pool_size or key_buffer_size.

And..

If you need speed just to fetch a data for a given combination or key, Redis is a solution that you need to look at. MySQL can no way compare to Redis and Memcache. …

Seriously, how does one repay for all the damage of such idiotic benchmarks?

P.S. I’ve ranted at benchmarks before, and will continue doing so.

If you want to change the set of characters that are considered word characters, you can do so in two ways…
* Modify the MySQL source … and recompile MySQL.
* Modify a character set file: This requires no recompilation.

Thus sayeth the Manual. What it doesn’t say is just what to modify in that character set file. I had to ask MySQL developer Alexander Barkov for help to get it right.

In the user comments on that manual page, John Navratil has provided most of the answer, but there’s an important step missing: adding a new collation.

The “normal” collations like latin1_swedish_ci are compiled in. But we don’t really want to mess with those anyway, because then the parser might become confused: you still want “-” to indicate subtraction everywhere except inside your fulltext search, right?

First, find your character set files. Typically they’re under [MySQL Install Path]\share\charsets, but you can double check that with SHOW VARIABLES LIKE 'character_sets_dir';

Edit Index.xml. Find the section for the character set you want to use (I’m using latin1), and add a new collation, with a new name and an unused id:

<collation name="latin1_ft_ci" id="62"/>

Edit the character set file (latin1.xml in my example). Near the top you’ll find a <ctype> array. There’s a leading 00 (which is there to offset the array to 257 characters for some legacy EOF convention, per the manual.) After that, you’ll find 256 bytes which identify the type of each character in the set. Find the hex value for the character you want (MySQL’s HEX() function is handy for this.) The value for “:” is 0×3A. Find that position in the array. Remember to start at 0×00, so 0×3A is the fourth row down, and the eleventh in from the left. Change the 10 there (which means spacing character) to 01 (which means upper case letter.) You’ll find the rest of the possible character types in the manual.

Scroll down in the same file and find the collations. Copy and paste the whole map from whichever one you normally use (like latin1_swedish_ci) and change the name to match the one we created in the Index.xml file (like latin1_ft_ci).

Restart the server and check that the new collation appears:

SHOW COLLATION LIKE 'latin1_ft_ci';

Finally, alter your table to use the new collation for the fulltext column:

ALTER TABLE tblName MODIFY textColumn TEXT COLLATE latin1_ft_ci;

And now your fulltext search ought to work.

Before and after:

mysql> SELECT * FROM t1 WHERE MATCH(d) against ('00:12:34:56:78:9A');
Empty set (0.00 sec)

mysql> ALTER TABLE t1 MODIFY d TEXT COLLATE latin1_ft_ci;
Query OK, 4 row affected (0.03 sec)
Records: 4  Duplicates: 0  Warnings: 0

mysql> SELECT * FROM t1 WHERE MATCH(d) against ('00:12:34:56:78:9A');
+-------------------+
| d                 |
+-------------------+
| 00:12:34:56:78:9A |
+-------------------+
1 row in set (0.01 sec)

• Redis
• Memcached
• Tokyo Tyrant / Tokyo Cabinet
• MySQL 5.1.40 (MyISAM)
• MySQL 5.1.40 (with Innodb Plugin 1.0.4), compiled into source of MySQL

My test had the following criteria

• 2 client boxes
• All clients connecting to the server using Python
• Used Python's threads to create concurrency
• Each thread made 10,000 open-close connections to the server
• The server was
  • Intel(R) Pentium(R) D CPU 3.00GHz
  • Fedora 10 32bit
  • Intel(R) Pentium(R) D CPU 3.00GHz
  • 2.6.27.38-170.2.113.fc10.i686 #1 SMP
  • 1GB RAM
• Used a md5 as key and a value that was saved
• Created an index on the key column of the table
• Each server had SET and GET requests as a different test at same concurrency

Results please !

read more

Date : 24 novembre 2009

Session 1 : 9h30-10h45

• Architecture MySQL : Parsing, exécution, optimizer, query cache, binlog…etc
• Architecture Innodb : Clustered index, bufferpool, hash index, insert buffer, locking model, MVCC, recovery log, checkpoint…etc. Comparaison avec les autres moteurs de stockage (PBXT, MySQL Cluster) et bases de données (PostgreSQL, Oracle ..).

Pause/Café : 10h45-11h00.

Session 2 : 11h00-12h30

• Évolutions récentes : Google SMP patches,  XtraDB, Oracle innodb plugin 1.0.4
• Apports de MySQL 5.4
• Améliorations à venir :  Ahead flushing / Adaptive checkpoint,  Purge lag / Purge thread, Splitted locks, etc…Explication technique des améliorations et des gains de performance obtenus.

Nous parlerons aussi de MySQL Enterprise et du MySQL Query analyzer qui vous permet d’optimiser un environnement MySQL.

Cette matinée se veut conviviale et interactive avec Q&A.

Unlike MyISAM Innodb does not have to keep open file descriptor when table is open – open table is purely logical state and appropriate .ibd file may be open or closed. Furthermore besides MySQL table_cache Innodb maintains its own (called data dictionary) which keeps all tables ever accessed since table start – there is no variable to control its size and it can take significant amount of memory in some edge cases. Percona patches though provide innodb_dict_size_limit to restrict growth of data dictionary.

So I started with same series of test and creating 100.000 tables with single integer column. The process of creating tables took about 45 minutes which is a lot more than MyISAM and the total size on disk was 12GB in .ibd files plus some space allocated in system tablespace. So if you create Innodb tables you better store some data in them otherwise there will be a huge waste of space.

I used MySQL 5.4.2 for tests which should be as good as it gets in terms of optimizations in this space.

To keep test alligned to my previous experiments I was running with table_open_cache=64 and tried innodb_open_files=64 and 16384.

Reading 100.000 tables first time after MySQL time takes about 500 seconds (so it is some 200 tables/sec) – first time Innodb actually populates data dictionary. The second time we do same operation it takes about 25 seconds (4.000 tables/sec) which is quite a difference. As we can see even in case table it fully in Innodb data dictionary the operation is slower than MyISAM tables. Though the difference can be related to the size of set of empty tables which is about 10 times smaller for MyISAM.

I found no significant difference whatever limit of open files was, which is not surprising as logical operation of opening file is rather fast on local file system – one can open/close file hundreds of thousands times per second.

To verify this I tried doing “open table” test for only 10K out of 100K tables – the performance was about the same, taking 1sec (on the second time) . Whenever innodb_open_files_limit was 64 (virtually all misses ) or 16384 (all hits) performance was the same.

As I mentioned Data Dictionary can take considerable amount of memory – In my case after reading all tables I got “Dictionary memory allocated 392029720″ which means very simple single tables takes about 4KB of space in data dictionary. More complicated tables can take a bit more.

So innodb_open_files does not affect performance a lot on reads – what is about writes ? I tried again very simple test inserting the row in each of 100K tables. This test ran about 180sec first time and about 260sec second time (with innodb_flush_log_at_trx_commit=0) go giving 550 and 380 updates/sec appropriately. Why was second time slower and not faster ? Because on the second run there were a lot of dirty pages in innodb buffer pool which had to be flushed before recycling. First ran however was done with clean buffer pool (after reading all tables once)

Same as with select case I could not see any measurable difference between two tested innodb_open_files values.

Finally I decided to test the crash recovery – does it make any difference ?

The crash recovery in Innodb is nasty if you have a lot of tables:

091118 18:43:36 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files…
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer…
InnoDB: Doing recovery: scanned up to log sequence number 12682768136
091118 18:47:44 InnoDB: Starting an apply batch of log records to the database…

If Innodb detects it is not shut down properly it will scan all .ibd files which took a bit over 4 minutes for 100K tables but which obviously can take a lot more if there are more tables or they are less cached than in this case. This part of data recovery does not depends on amount of records which need to be applied just about number of tables.

With innodb_open_files=64 I got bunch of warning messages during recovery:

091118 18:47:44 InnoDB: Warning: too many (67) files stay open while the maximum
InnoDB: allowed value would be 64.
InnoDB: You may need to raise the value of innodb_max_files_open in
InnoDB: my.cnf.
InnoDB: fil_sys open file LRU len 0
091118 18:47:44 InnoDB: Warning: too many (68) files stay open while the maximum
InnoDB: allowed value would be 64.
InnoDB: You may need to raise the value of innodb_max_files_open in
InnoDB: my.cnf

So we can see Innodb may with to have so many open files during recovery stage and it will open more files than allowed if needed.

Both scanning open files and applying logs took about 9 minutes in this setup. This number of course can change a lot depending on hardware log file size workload and even when crash happen (how many unflushed changes we had)

Repeating test with innodb_open_files=16384 I got about same crash recovery speed though with no warnings.

So it looks like innodb_open_files_limit=300 is not being that large liability even with large number of tables and you can also safely increase this number if you like – there is no any surprises such as surprised slow downs for replacing open files in the list. I guess Heikki knows how to implement LRU in the end

Entry posted by peter | 2 comments

Add to: | | | |

An mqd test on sample file slow023.txt began to differ after some pretty extensive code changes of late:

< # Query 1: 0 QPS, 0x concurrency, ID 0x8E38374648788E52 at byte 0 ________
---
> # Query 1: 0 QPS, 0x concurrency, ID 0x2CFD93750B99C734 at byte 0 ________

The ID which depends on the query’s fingerprint has changed. It’s very important that we don’t suddenly change these on users because these IDs are pivotal in trend analyses with mqd’s --review-history option. First some background info on the recent code changes and then the little story about how I tracked down the source of this change.

mqd internals used to run like this: call parser module (like SlowLogParser) and pass it an array of callbacks which it ran events through. Now this has changed so there’s a single, unified pipeline of “callbacks” (they’re technically no longer callbacks). The first process in the pipeline is usually a parser module which returns each event and then mqd keeps pumping the events through the pipeline (in contrast to before where the parser module did the pumping). So “obviously” this has nothing to do with query fingerprinting or ID making which is done in code that has not changed. Thus, this “bug” was very perplexing at first.

First step: see what value make_checksum(), which makes the query IDs, used to get and gets now by using the Perl debugger:

DB<3> x $item
0 'select count(*) as a from x '


DB<12> x $item
0 'select count(*) as a from x'

The difference is that single trailing space. But why has this space suddenly disappeared in the new (later) rev? Something in fingerprint() must have changed, which is the sub that makes that query. Use the debugger again to step through fingerprint() while a watch is set on the var:

1574: $query =~ s/\A\s+//; # Chop off leading whitespace
DB<6>
Watchpoint 0: $query changed:
old value: ' select count(*) as A from X
'
new value: 'select count(*) as A from X
'
QueryRewriter::fingerprint(bin/mk-query-digest:1575):
1575: chomp $query; # Kill trailing whitespace
DB<6>
QueryRewriter::fingerprint(bin/mk-query-digest:1576):
1576: $query =~ tr[ \n\t\r\f][ ]s; # Collapse whitespace
DB<6>
Watchpoint 0: $query changed:
old value: 'select count(*) as A from X
'
new value: 'select count(*) as A from X '

Notice that the var did not change after the line “# Kill trailing whitespace” was executed. The trailing newline was removed and reduced to a single trailing space when “# Collapse whitespace” was executed. The new rev:

1585: $query =~ s/\A\s+//; # Chop off leading whitespace
DB<4>
Watchpoint 0: $query changed:
old value: ' select count(*) as A from X
'
new value: 'select count(*) as A from X
'
QueryRewriter::fingerprint(../mk-query-digest:1586):
1586: chomp $query; # Kill trailing whitespace
DB<4>
Watchpoint 0: $query changed:
old value: 'select count(*) as A from X
'
new value: 'select count(*) as A from X'

Notice how chomp in the new rev removed all trailing whitespace; the result of chomp has changed, but why? In case you didn’t know, chomp actually chomps any trailing $INPUT_RECORD_SEPARATOR, not just newlines. It just so happens that many of the parser modules change $INPUT_RECORD_SEPARATOR.

The root of this subtle but very important change is due to the fact that the parser modules no longer call the pipeline callbacks. When they did, their changes to $INPUT_RECORD_SEPARATOR were visible to the callbacks, and operations like fingerprint() are part of the callbacks. Now that they do not, their changes to $INPUT_RECORD_SEPARATOR are “invisible” and operations like fingerprint() see a different (i.e. the default) $INPUT_RECORD_SEPARATOR.

Conclusion in brief: an issue of scope at the beginning of mk-query-digest affects chomp causing fingerprint() and make_checksum() to generate different query IDs at the end of the script.

We are pleased to announce the availablity of the 0.9.5.1 alpha release of InfiniDB Community Edition.  This is our latest alpha release and is not recommended for production work. 

New functionality we’ve added with 0.9.5.1 includes:  

Vote UP / Vote DOWN]]> Wed, 18 Nov 2009 20:10:36 +0000 Calpont & InfiniDB Project Team tag:everythingmysql.ning.com,2009-11-18:3993569:BlogPost:857 http://everythingmysql.ning.com/xn/detail/3993569%3ABlogPost%3A857 There have been times that I needed to restore one database from a dump file that contains all of the databases from the server. There are a few ways to accomplish this goal. One way would be to grep through the entire file for the table schema and insert statements. There are some problems with this method in some environments. For example, what if a table name in the targeted databases had the same name in another database on the same instance? The logic for grep now gets increasingly more complex. It’s a good thing that the developers at MySQL already thought of this and implemented it, --one-database. In the example below I have created 4 databases, test, test[1-3] and filled each of them with 10K rows. I run mysqldump for all databases and put them in a file called test.all.sql then truncate test2.tbl1. Using mysql with the --one-database option I restore the data to its original state. Example: mysql> select min(anum), max(anum) from test.tbl1; +-----------+-----------+ | min(anum) | max(anum) | +-----------+-----------+ | 0 | 10000 | +-----------+-----------+ 1 row in set (0.01 sec) mysql> select min(anum), max(anum) from test1.tbl1; +-----------+-----------+ | min(anum) | max(anum) | +-----------+-----------+ | 10001 | 20000 | +-----------+-----------+ 1 row in set (0.01 sec) mysql> select min(anum), max(anum) from test2.tbl1; +-----------+-----------+ | min(anum) | max(anum) | +-----------+-----------+ | 20001 | 30000 | +-----------+-----------+ 1 row in set (0.01 sec) mysql> select min(anum), max(anum) from test3.tbl1; +-----------+-----------+ | min(anum) | max(anum) | +-----------+-----------+ | 30001 | 40000 | +-----------+-----------+ 1 row in set (0.01 sec) shell> mysqldump -uroot -S mysql.sock --opt --all-databases > test.all.sql mysql> use test2; mysql> select count(*) from tbl1; +----------+ | count(*) | +----------+ | 10000 | +----------+ 1 row in set (0.01 sec) mysql> truncate table tbl1; Query OK, 0 rows affected (0.05 sec) mysql> select count(*) from tbl1; +----------+ | count(*) | +----------+ | 0 | +----------+ 1 row in set (0.00 sec) shell> mysql -uroot -S mysql.sock --one-database test2 < test.all.sql mysql> select min(anum), max(anum) from test.tbl1; +-----------+-----------+ | min(anum) | max(anum) | +-----------+-----------+ | 0 | 10000 | +-----------+-----------+ 1 row in set (0.01 sec) mysql> select min(anum), max(anum) from test1.tbl1; +-----------+-----------+ | min(anum) | max(anum) | +-----------+-----------+ | 10001 | 20000 | +-----------+-----------+ 1 row in set (0.01 sec) mysql> select min(anum), max(anum) from test2.tbl1; +-----------+-----------+ | min(anum) | max(anum) | +-----------+-----------+ | 20001 | 30000 | +-----------+-----------+ 1 row in set (0.01 sec) mysql> select min(anum), max(anum) from test3.tbl1; +-----------+-----------+ | min(anum) | max(anum) | +-----------+-----------+ | 30001 | 40000 | +-----------+-----------+ 1 row in set (0.01 sec) As you can see test2.tbl1 has the original data back in and we did not have to write our own script to accomplish this.
It’s a good thing that the developers at MySQL already thought of this and implemented it, --one-database. In the example below I have created 4 databases, test, test[1-3] and filled each of them with 10K rows. I run mysqldump for all databases and put them in a file called test.all.sql then truncate test2.tbl1. Using mysql with the --one-database option I restore the data to its original state.

Example:

mysql> select min(anum), max(anum) from test.tbl1;
+-----------+-----------+
| min(anum) | max(anum) |
+-----------+-----------+
| 0 | 10000 |
+-----------+-----------+
1 row in set (0.01 sec)

mysql> select min(anum), max(anum) from test1.tbl1;
+-----------+-----------+
| min(anum) | max(anum) |
+-----------+-----------+
| 10001 | 20000 |
+-----------+-----------+
1 row in set (0.01 sec)

mysql> select min(anum), max(anum) from test2.tbl1;
+-----------+-----------+
| min(anum) | max(anum) |
+-----------+-----------+
| 20001 | 30000 |
+-----------+-----------+
1 row in set (0.01 sec)

mysql> select min(anum), max(anum) from test3.tbl1;
+-----------+-----------+
| min(anum) | max(anum) |
+-----------+-----------+
| 30001 | 40000 |
+-----------+-----------+
1 row in set (0.01 sec)

shell> mysqldump -uroot -S mysql.sock --opt --all-databases > test.all.sql

mysql> use test2;

mysql> select count(*) from tbl1;
+----------+
| count(*) |
+----------+
| 10000 |
+----------+
1 row in set (0.01 sec)

mysql> truncate table tbl1;
Query OK, 0 rows affected (0.05 sec)

mysql> select count(*) from tbl1;
+----------+
| count(*) |
+----------+
| 0 |
+----------+
1 row in set (0.00 sec)

shell> mysql -uroot -S mysql.sock --one-database test2 < test.all.sql

mysql> select min(anum), max(anum) from test.tbl1;
+-----------+-----------+
| min(anum) | max(anum) |
+-----------+-----------+
| 0 | 10000 |
+-----------+-----------+
1 row in set (0.01 sec)

mysql> select min(anum), max(anum) from test1.tbl1;
+-----------+-----------+
| min(anum) | max(anum) |
+-----------+-----------+
| 10001 | 20000 |
+-----------+-----------+
1 row in set (0.01 sec)

mysql> select min(anum), max(anum) from test2.tbl1;
+-----------+-----------+
| min(anum) | max(anum) |
+-----------+-----------+
| 20001 | 30000 |
+-----------+-----------+
1 row in set (0.01 sec)

mysql> select min(anum), max(anum) from test3.tbl1;
+-----------+-----------+
| min(anum) | max(anum) |
+-----------+-----------+
| 30001 | 40000 |
+-----------+-----------+
1 row in set (0.01 sec)

As you can see test2.tbl1 has the original data back in and we did not have to write our own script to accomplish this.
PlanetMySQL Voting: Vote UP / Vote DOWN]]> Wed, 18 Nov 2009 20:09:09 +0000 Everything MySQL 960 at http://www.krisbuytaert.be/blog http://www.krisbuytaert.be/blog/got-interviewed by @botchagalupe on Virtualization, Open Source tools and DNS Problems Technorati Tags: dnsproblem drupal ha heartbeat linux-ha mysql pacemaker puppet virtualization xen Share with Shareomatic! Trackback URL for this post: http://www.krisbuytaert.be/blog/trackback/960 by @botchagalupe
on Virtualization, Open Source tools and DNS Problems

Technorati Tags: dnsproblem drupal ha heartbeat linux-ha mysql pacemaker puppet virtualization xen

Share with Shareomatic!

The Home Screen

The most prominent new addition in respect to previous Workbench releases is the new Home Screen. It allows you to access the main features of Workbench in a nice and easy way and is divided into 4 parts.

The upper Workbench Central panel features a few Links and Action Buttons to quickly access common resources.

The lower Workspace panel shows the main feature sets, grouped horizontally.

• SQL Development allows editing and execution of SQL queries and scripts, create or alter database objects and edit table data.
• Data Modeling covers the EER Modeling functionality you might already be familiar with from previous MySQL Workbench releases.
• Server Administration offers administrative tasks like starting/stopping the server, edit the database server configuration, create user accounts, do data dumps and much more.

To quickly get started, use the following steps.

Home Screen Tutorial

1. Create a new Connection – Before you can start any other tasks — using the SQL Editor, forward engineering your model, or managing your database server, you need to create a new Database Connection. You can do that by using the New Connection Wizard or – if you are an advanced user – by using the Manage Connections dialog.You will have to enter the usual connection parameters, like IP address and port the server is running on as well as your username and password.
2. Open Editor – After you have created a new connection, it will be displayed in the list of available Database Connections. Simply double-click on the list entry to open a SQL Editor and start querying.
3. Create a new model - If you want to start your work by designing a visual database design first, click the Create new EER Model action item. An empty model will be created for you, only featuring the my_db schema which you can easily rename to the desired name.
4. Create a New Server Profile – In order to administer your database server you have to register it within MySQL Workbench first.  Use the New Server Profile Wizard or – if you are an advanced user – use the Manage Server Profiles dialog.
5. Open Admin – Once the new Server Profile is registered and appears in the list box you can double-click it to open the Admin.

Let’s take a closer look at the SQL Editor.

SQL Editor

The SQL Editor features a straight forward interface that gets you going, immediately.

SQL Editor Tutorial

1. To give you instant access to your database objects we have added the Live Schema Overview panel to the SQL Editor. To start editing the data of an existing table, insert new data rows or manipulate existing data, simply double-click on the table.
2. You can also modify the table structure by right-clicking on the table and select Alter … . This also works for any other database object.
3. Enter your SQL queries and scripts on the SQL Statement panel. You can execute a single or multiple statements at the same time by clicking the Execution Arrow toolbar icon or hitting Ctrl+Return on the keyboard.
4. If you create new objects by using SQL statements make sure to click the Refresh toolbar icon to update the Live Schema Overview panel and the Schema Tree.

We assume you already know how to use the EER Modeling and therefore we proceed with the Admin.

Admin

The Admin also features a very clear design that makes it easy to access the desired features.

Admin Tutorial

1. After opening the Admin you can see the Server Status panel at the top. Use this panel to verify the current server status and to check the current Server Health graphs.
2. Select the corresponding Configuration tab that fits your current task.
3. After selecting the correct Configuration tab perform the necessary actions on the Configuration page.

After studying this tutorial you should have a basic idea how the use MySQL Workbench 5.2 most efficiently. Please write a comment if you have a tip for other users.

We are proud to announce that we have reached the Beta 1 for MySQL Workbench 5.2. Beta 1 is “functionally complete” and shows the new features for the upcoming version of Workbench GUI product.

The team has worked very hard to reach the goal of including all our new features within this first public beta version. We know there is still a lot of fine tuning and stabilization to be done over the coming weeks to get it solid on all platforms, but this release marks an important milestone.  Your beta feedback is key to this.

MySQL Workbench 5.2 Beta 1 provides:

1. Data Modeling
2. Query (upgrade from MySQL Query Browser)
3. Admin (upgrade from MySQL Administrator)

If you are a current user of MySQL Query Browser or MySQL Administrator, we look forward to your feedback on all the new capabilities we are delivering in a single unified MySQL Workbench

As always, you will find binaries for the various platforms on our download pages.

Please get your copy from our Download

http://dev.mysql.com/downloads/workbench/

To get started quickly, please take a look at this short tutorial.

MySQL Workbench 5.2 Beta Tutorial

http://wb.mysql.com/?p=406

Please be aware that this release is the initial a beta version – so please don’t use it on your production servers! Also note, MySQL Workbench files saved with version 5.2 cannot be opened with previous versions of our program.

The files for several platforms have been pushed to our main server and
should be available on our mirrors.

Blog postings and general information – including build instructions for Linux – can be found on our Workbench Developer Central site.

Workbench Developer Central

http://wb.mysql.com

If you need any additional info or help please get in touch with us.
Post in our forums, leave comments on our blog pages or if you want to talk to us directly you can visit us on our IRC channel #workbench on irc.freenode.net.

Again, thank you for trying out the Workbench beta, we look forward to your feedback and bug reports.

- The MySQL Workbench Team

I've been reading through the requested features for the new client on the wiki here:

• Proposed Client Features
• BOOTS Console API Design

I think all the stuff on that link is excellent so far. I'd also like to request a feature that I think will be a really cool timesaver for DBAs and developers using Drizzle.

Macro Support

Remember, "way back when" you used Microsoft Excel and were able to start recording your actions, then when you stopped recording, Excel would store a "macro" of your actions that you could subsequently replay?

I think this would be incredibly useful for folks who do repetitive work in the console.

Sure, I know, I know...the first reaction folks will say is "but HEY, you guys removed stored procedures!" Yeah, yeah... but the feature I'm proposing here is different from stored procedures in the following ways:

1. It's entirely client-side. There is no server-side storage/cache, processing, parsing, or anything.
2. It's not limited to a small subset of SQL that stored procedures (at least in MySQL) are currently limited to. Anything the new client can do would be able to go into a macro.
3. Since the client is in Python, the macros are themselves re-writable in a scripting language. This gives the recorded macros incredible flexibility.
4. No fussing with SQL stored procedure permissions at runtime (you know, the silly INVOKER/DEFINER crap)
5. Ability to interact with result sets in the macro. Just try doing that easily in a SQL stored procedure. Using CURSORs is incredibly clunk and ugly. Applying a Python function or closure/lambda on each of a result set is elegant and easy.

Imagine the following rough example interface...

drizzle> RECORD MACRO "sales_report_with_email" (to_email);
macro recording started.

drizzle> mode python;
in python mode.

python> import datetime
python> today= datetime.datetime.now().isoformat()
python> filename= "%s-%s-%s" % ("sales", to_email, today)
python> Ctrl-D

drizzle> SELECT * FROM sales
         WHERE manager = @to_email; > csv(@filename);
drizzle> mode python;
In python mode.

python> report_txt= open(filename, "r+b").read()
python> import smtplib
python> mailserver = smtplib.SMTP('localhost')
python> mailserver.sendmail('theboss@company.com', to_email, report_txt)
python> mailserver.quit()
python> print "Mail sent to %s\n" % to_email
python> Ctrl-D

drizzle> STOP MACRO;
Macro "sales_report_with_email" saved.

drizzle> macro("sales_report_with_email", "myboss@company.com");
Mail sent to myboss@company.com

Pretty powerful, eh?

If you follow the flow above, you will notice the only real trick to solve is passing the macro's arguments into the console's variable array, and from the console's variable array into the Python interpreter's variable scope. But this is a fairly simple problem to solve...

Thoughts? Suggestions? If you've got comments, please feel free to share here, or on the Drizzle Discussion mailing list, or even update the wiki pages posted above. Thanks!

Date : 24 novembre 2009
Lieu : Centre de Conférences, Capital 8, 32 Monceau, 75008 Paris.
Accueil : 9h00 (petit déjeuner et café seront offerts).
Prix : Cette matinée est gratuite !

Session 1 : 9h30-10h45

• Architecture MySQL : Parsing, exécution, optimizer, query cache, binlog…etc
• Architecture Innodb : Clustered index, bufferpool, hash index, insert buffer, locking model, MVCC, recovery log, checkpoint…etc. Comparaison avec les autres moteurs de stockage (PBXT, MySQL Cluster) et bases de données (PostgreSQL, Oracle ..).

Pause/Café : 10h45-11h00.

Session 2 : 11h00-12h30

• Évolutions récentes : Google SMP patches,  XtraDB, Oracle innodb plugin 1.0.4
• Apports de MySQL 5.4
• Améliorations à venir :  Ahead flushing / Adaptive checkpoint,  Purge lag / Purge thread, Splitted locks, etc…Explication technique des améliorations et des gains de performance obtenus.

Nous parlerons aussi de MySQL Enterprise et du MySQL Query analyzer qui vous permet d’optimiser un environnement MySQL.

Cette matinée se veut conviviale et interactive avec Q&A.

Le nombre de places étant limité, inscrivez-vous dès aujourd’hui auprès de olivier.beutels@sun.com

Until today... ;)

Preparing the database

Okay, first I need a table to store my blobs. In this example I'll store images in my database because images usually look better in a tutorial than some random raw data.

mysql> CREATE TABLE images (
       id INT(10) UNSIGNED NOT NULL AUTO_INCREMENT,
       image MEDIUMBLOB NOT NULL,
       PRIMARY KEY (id)
       );
Query OK, 0 rows affected (0.02 sec)

In general you don't want to store images in a relational database. But that's another discussion for another day.

Storing the blob

To make a long story short, here's the code to store a blob using MySQLi:

<?php
	$mysqli=mysqli_connect('localhost','user','password','db');

	if (!$mysqli)
		die("Can't connect to MySQL: ".mysqli_connect_error());

	$stmt = $mysqli->prepare("INSERT INTO images (image) VALUES(?)");
	$null = NULL;
	$stmt->bind_param("b", $null);

	$stmt->send_long_data(0, file_get_contents("osaka.jpg"));

	$stmt->execute();
?>

If you already used MySQLi, most of the above should look familiar to you. I highlighted two pieces of code, which I think are worth looking at:

1. The $null variable is needed, because bind_param() always wants a variable reference for a given parameters. In this case the "b" (as in blob) parameter. So $null is just a dummy, to make the syntax work.

2. In the next step I need to "fill" my blob parameter with the actual data. This is done by send_long_data(). The first parameter of this method indicates which parameter to associate the data with. Parameters are numbered beginning with 0. The second parameter of send_long_data() contains the actual data to be stored.

While using send_long_data(), please make sure that the blob isn't bigger than MySQL's max_allowed_packet:

mysql> SHOW VARIABLES LIKE 'max_allowed_packet';
+--------------------+----------+
| Variable_name      | Value    |
+--------------------+----------+
| max_allowed_packet | 16776192 | 
+--------------------+----------+
1 row in set (0.00 sec)

If your data exceeds max_allowed_packet, you probably don't get any errors returned from send_long_data() or execute(). The saved blob is just corrupt!

Simply raise the value max_allowed_packet to whatever you'll need. If you're not able to change MySQL's configuration, you'll need to send the data in smaller chunks:

	$fp = fopen("osaka.jpg", "r");
	while (!feof($fp)) 
	{
 	   $stmt->send_long_data(0, fread($fp, 16776192));
	}

Usually the default value of 16M should be a good start.

Retrieving the blob

Getting the blob data out of the database is quite simple and follows the usual way of MySQLi:

<?php
	$mysqli=mysqli_connect('localhost','user','password','db');

	if (!$mysqli)
		die("Can't connect to MySQL: ".mysqli_connect_error());

	$id=1;  
	$stmt = $mysqli->prepare("SELECT image FROM images WHERE id=?"); 
	$stmt->bind_param("i", $id);

	$stmt->execute();
	$stmt->store_result();

	$stmt->bind_result($image);
	$stmt->fetch();

	header("Content-Type: image/jpeg");
	echo $image; 
?>

Connect to the database, prepare the SQL statement, bind the parameter(s), execute the statement, bind the result to a variable, and fetch the actual data from the database. In this case there is no need to worry about max_allowed_packet. MySQLi will do all the work:

By the way...

If you want to insert a blob from the command line using MySQL monitor, you can use LOAD_FILE() to fetch the data from a file:

mysql> INSERT INTO images (image) VALUES( LOAD_FILE("/home/oswald/osaka.jpg") );

Be aware that also in this case max_allowed_packet limits the amount of data you're able to send to the database:

mysql> SHOW VARIABLES LIKE 'max_allowed_packet';
+--------------------+-------+
| Variable_name      | Value |
+--------------------+-------+
| max_allowed_packet | 7168  | 
+--------------------+-------+
1 row in set (0.00 sec)

mysql> INSERT INTO images (image) VALUES( LOAD_FILE("/home/oswald/osaka.jpg") );
ERROR 1048 (23000): Column 'image' cannot be null
mysql> SET @@max_allowed_packet=16777216;
Query OK, 0 rows affected (0.00 sec)

mysql> SHOW VARIABLES LIKE 'max_allowed_packet';
+--------------------+----------+
| Variable_name      | Value    |
+--------------------+----------+
| max_allowed_packet | 16777216 | 
+--------------------+----------+
1 row in set (0.00 sec)

mysql> INSERT INTO images (image) VALUES( LOAD_FILE("/home/oswald/osaka.jpg") );
Query OK, 1 row affected (0.03 sec)
PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN]]>
    Wed, 18 Nov 2009 10:35:20 +0000
    Kai Seidler
    MySQL
  

  
    
    http://www.mysqlperformanceblog.com/?p=1751
    http://www.mysqlperformanceblog.com/2009/11/18/5-0-87-build20-percona-binaries/
    Dear Community,
We are pleased to present the 20th build of MySQL server with Percona patches.
Comparing to the previous release it has following new features:

The build is based on MySQL-5.0.87
innodb_rw_lock.patch is ported from InnoDB Plugin 1.0.3
To be compatible with RedHat RPM repository, the naming scheme has changed to
<rpm name>-<mysql version>-<percona build version>.<buildnumber>.<redhat version>.<architecture>.rpm
Example:
MySQL-server-percona-5.0.87-b20.29.rhel5.x86_64.rpm


See release notes for earlier changes.
Since the build 20 MySQL server with Percona patches is available in Percona RPM repository via YUM. To make it working add a file Percona.repo in /etc/yum.repos.d with following content
PLAIN TEXT
CODE:




[percona]


name=CentOS-$releasever - Percona


baseurl=http://repo.percona.com/centos/$releasever/os/$basearch/


gpgcheck=0 






As usual you can download binaries and sources with the patches here
http://www.percona.com/mysql/5.0.87-b20/
There is Debian packages repository is also available. See release page for configuration and usage guideance.
The Percona patches live on Launchpad : https://launchpad.net/percona-patches and you can report bug to Launchpad bug system:
https://launchpad.net/percona-patches/+filebug. The documentation is available on our Wiki
For general questions use our Pecona-discussions group, and for development question Percona-dev group.
For support, commercial and sponsorship inquiries contact Percona.

innodb_rw_lock.patch

    
    Entry posted by Aleksandr Kuzminsky |
      2 comments
    Add to:  |  |  |  | 
    Dear Community,

We are pleased to present the 20th build of MySQL server with Percona patches.
Comparing to the previous release it has following new features:

The build is based on MySQL-5.0.87
innodb_rw_lock.patch is ported from InnoDB Plugin 1.0.3
To be compatible with RedHat RPM repository, the naming scheme has changed to
<rpm name>-<mysql version>-<percona build version>.<buildnumber>.<redhat version>.<architecture>.rpm
Example:
MySQL-server-percona-5.0.87-b20.29.rhel5.x86_64.rpm


See release notes for earlier changes.
Since the build 20 MySQL server with Percona patches is available in Percona RPM repository via YUM. To make it working add a file Percona.repo in /etc/yum.repos.d with following content
PLAIN TEXT
CODE:




[percona]


name=CentOS-$releasever - Percona


baseurl=http://repo.percona.com/centos/$releasever/os/$basearch/


gpgcheck=0 






As usual you can download binaries and sources with the patches here

http://www.percona.com/mysql/5.0.87-b20/
There is Debian packages repository is also available. See release page for configuration and usage guideance.
The Percona patches live on Launchpad : https://launchpad.net/percona-patches and you can report bug to Launchpad bug system:
https://launchpad.net/percona-patches/+filebug. The documentation is available on our Wiki
For general questions use our Pecona-discussions group, and for development question Percona-dev group.
For support, commercial and sponsorship inquiries contact Percona.

innodb_rw_lock.patch

    

    
Entry posted by Aleksandr Kuzminsky |
      2 comments
    
Add to:  |  |  |  | 

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN]]>
    Wed, 18 Nov 2009 10:04:54 +0000
    MySQL Performance Blog
    announce
    mysql
  

  
    
    http://blogs.sun.com/mysqlf/entry/mysql_university_memcached_functions_for
    http://blogs.sun.com/mysqlf/entry/mysql_university_memcached_functions_for
    This
Thursday (November 19th, 14:00 UTC), Patrick Galbraith will present memcached Functions for MySQL (UDFs). This session is about a suite of functions available to use with MySQL that allow you to store, retrieve and delete data, as well as most of the functions and operations that are available with libmemcached, such as server connectivity to the client, server status, client behaviors, and more. You can combine the fetching of data from one or more tables with the fetching of data from memcached and be able to apply any SQL operations on that result set such as LIMIT, sorting and other conditional operations. 
  For MySQL University sessions, point your browser to this page. You need a browser with a working Flash plugin. You may register for a Dimdim account, but you don't have to. (Dimdim is the conferencing system we're using for MySQL University sessions. It provides integrated voice streaming, chat, whiteboard, session recording, and more.) All MySQL University sessions are recorded, that is, slides and voice can be viewed as a Flash movie (.flv). You can find those recordings on the respective MySQL University session pages which are listed on the MySQL University home page.  
  MySQL University is a free educational online program for engineers/developers. MySQL University sessions are open to anyone, not just Sun employees. Sessions are recorded (slides and audio), so if you can't attend the live session you can look at the recording anytime after the session.  
  Here's the schedule for the rest of this year:  
   
    November 26: The Spider Storage Engine (Giuseppe Maxia) 
    December 3: Practical Full-Text Search in MySQL (Bill Karwin) 
   
  The schedule is not engraved in stone at this point. Please visit http://forge.mysql.com/wiki/MySQL_University#Upcoming_Sessions for the up-to-date list. On that page, you can also find the starting time for many time zones.
    This
Thursday (November 19th, 14:00 UTC), Patrick Galbraith will present memcached Functions for MySQL (UDFs). This session is about a suite of functions available to use with MySQL that allow you to store, retrieve and delete data, as well as most of the functions and operations that are available with libmemcached, such as server connectivity to the client, server status, client behaviors, and more. You can combine the fetching of data from one or more tables with the fetching of data from memcached and be able to apply any SQL operations on that result set such as LIMIT, sorting and other conditional operations.
 
  
For MySQL University sessions, point your browser to this page. You need a browser with a working Flash plugin. You may register for a Dimdim account, but you don't have to. (Dimdim is the conferencing system we're using for MySQL University sessions. It provides integrated voice streaming, chat, whiteboard, session recording, and more.) All MySQL University sessions are recorded, that is, slides and voice can be viewed as a Flash movie (.flv). You can find those recordings on the respective MySQL University session pages which are listed on the MySQL University home page.
  
  
MySQL University is a free educational online program for engineers/developers. MySQL University sessions are open to anyone, not just Sun employees. Sessions are recorded (slides and audio), so if you can't attend the live session you can look at the recording anytime after the session.
  
  
Here's the schedule for the rest of this year:
  
  
 
    
November 26: The Spider Storage Engine (Giuseppe Maxia)
 
    
December 3: Practical Full-Text Search in MySQL (Bill Karwin)
 
  
 
  
The schedule is not engraved in stone at this point. Please visit http://forge.mysql.com/wiki/MySQL_University#Upcoming_Sessions for the up-to-date list. On that page, you can also find the starting time for many time zones.

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN]]>
    Wed, 18 Nov 2009 10:04:20 +0000
    Stefan Hinz
    Sun
    memcached
    mysql
    udf
    university
  

  
    
    http://venublog.com/2009/11/17/innodb-tablespace-corruption/
    http://venublog.com/2009/11/17/innodb-tablespace-corruption/
    Over the weekend, I experienced a strange issue (even though its not new) with the InnoDB tablespace (ibdata) corruption. When in general InnoDB crashes, it automatically recovers during the next start by rolling back/forward based on what was pending and un-flushed/un-committed changes at the time of crash.
But for some reason, one of the server; we ran out of disk space (yeah, no alerts) on data directory; where we store everything (tablespace, logs and data); and server was running for few hours in this mode (disk full); and it became un-available and not responding after a while. Only option left was to kill the server process and its PID along with cleaning the stuff to get the space back.  After I (re)started the server; server failed to start with the following error..

InnoDB: Error: trying to access page number 1098759810 in space 0,
InnoDB: space name /data/ibdata1,
InnoDB: which is outside the tablespace bounds.
InnoDB: Byte offset 0, len 16384, i/o type 10.

Means the tablespace is corrupted.. By enabling the monitor; I noticed the following..

InnoDB: Error: data dictionary entry for table DBXX/tableXX is corrupt!
InnoDB: Index field 0 is delete marked.
091117 20:32:47InnoDB: Assertion failure in thread 1148791104 in file dict0load.c line 503
InnoDB: Failing assertion: ut_memcmp&#40;buf, field, len&#41; == 0

Looks like a modification to primary key on that table which was never persisted&#8230;as it failed to write anything to disk just before the crash; wondering why the engine did not trap it earlier as it will not mark as persistent unless a write to log returns success (even if the modified entries are in memory, the change is written in log as everything else is rolled back)
Anyway, I recovered using innodb_force_recovery=4 and dumped and re-loaded it as it failed sub-sequent restarts to start again even after the first recovery (well, should not start as dictionary is wrong)&#8230;
Time for me to simulate this scenario on Drizzle and Maria for fun (Just got the source of Maria couple of days back, and should start contributing the code) to see how it works out in the coming days.
    Over the weekend, I experienced a strange issue (even though its not new) with the InnoDB tablespace (ibdata) corruption. When in general InnoDB crashes, it automatically recovers during the next start by rolling back/forward based on what was pending and un-flushed/un-committed changes at the time of crash.

But for some reason, one of the server; we ran out of disk space (yeah, no alerts) on data directory; where we store everything (tablespace, logs and data); and server was running for few hours in this mode (disk full); and it became un-available and not responding after a while. Only option left was to kill the server process and its PID along with cleaning the stuff to get the space back.  After I (re)started the server; server failed to start with the following error..

InnoDB: Error: trying to access page number 1098759810 in space 0,
InnoDB: space name /data/ibdata1,
InnoDB: which is outside the tablespace bounds.
InnoDB: Byte offset 0, len 16384, i/o type 10.

Means the tablespace is corrupted.. By enabling the monitor; I noticed the following..

InnoDB: Error: data dictionary entry for table DBXX/tableXX is corrupt!
InnoDB: Index field 0 is delete marked.
091117 20:32:47InnoDB: Assertion failure in thread 1148791104 in file dict0load.c line 503
InnoDB: Failing assertion: ut_memcmp(buf, field, len) == 0

Looks like a modification to primary key on that table which was never persisted…as it failed to write anything to disk just before the crash; wondering why the engine did not trap it earlier as it will not mark as persistent unless a write to log returns success (even if the modified entries are in memory, the change is written in log as everything else is rolled back)
Anyway, I recovered using innodb_force_recovery=4 and dumped and re-loaded it as it failed sub-sequent restarts to start again even after the first recovery (well, should not start as dictionary is wrong)…
Time for me to simulate this scenario on Drizzle and Maria for fun (Just got the source of Maria couple of days back, and should start contributing the code) to see how it works out in the coming days.

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN]]>
    Wed, 18 Nov 2009 05:10:23 +0000
    Venu Anuganti
    Database
    MySQL
    InnoDB crash
    InnoDB crash recovery
    Recover InnoDB TableSpace Corruption
  

  
    
    http://mike.kruckenberg.com/archives/2009/11/digg-moved-from-mysql-to-cassandra.html
    http://mike.kruckenberg.com/archives/2009/11/digg-moved-from-mysql-to-cassandra.html
    At today's Web 2.0 Expo keynote with Kevin Rose and Jay Adelson they mentioned they'd done a lot of architecture work on Digg including moving things from MySQL to Cassandra.

I guess something like this was on the table in 2006, mentioned in this O'Reilly article, but the comments clear it up as more of a "we've thought about it and are sticking with MySQL."

This whole idea of NoSQL, and that developers would rank performance over normalization and ability to use joins is interesting. At today's NoSQL talk there was a good Q/A where folks got to ask questions like "what if you need transactions" or "how consistent is data replicated to other servers." NoSQL adopts the line about choosing "the right tool for the job", so the response to many of these questions is not what a died-in-the-wool RDBMS user would be looking to hear.

Interesting to see how this class of data storage evolves. The prediction at today's talk was that in 12 months most new web development would be happening on a NoSQL database. Hmm. The web moves fast, but I'm not sure about that one.
    At today's Web 2.0 Expo keynote with Kevin Rose and Jay Adelson they mentioned they'd done a lot of architecture work on Digg including moving things from MySQL to Cassandra.


I guess something like this was on the table in 2006, mentioned in this O'Reilly article, but the comments clear it up as more of a "we've thought about it and are sticking with MySQL."

This whole idea of NoSQL, and that developers would rank performance over normalization and ability to use joins is interesting. At today's NoSQL talk there was a good Q/A where folks got to ask questions like "what if you need transactions" or "how consistent is data replicated to other servers." NoSQL adopts the line about choosing "the right tool for the job", so the response to many of these questions is not what a died-in-the-wool RDBMS user would be looking to hear.

Interesting to see how this class of data storage evolves. The prediction at today's talk was that in 12 months most new web development would be happening on a NoSQL database. Hmm. The web moves fast, but I'm not sure about that one.

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN]]>
    Wed, 18 Nov 2009 01:18:26 +0000
    Mike Kruckenberg
    Technology
  

  
    
    http://themattreid.com/wordpress/?p=251
    http://feedproxy.google.com/~r/Themattreid/~3/FUCjETtnhHg/
    I&#8217;ve been messing around with the Kontrollbase schema for the last couple of days, writing various queries for the daily reporting scripts that will eventually be an automated pdf report. I&#8217;ll give you examples of two of the queries, the first being overall environment stats, and the second being single-host growth over time.
Overall environment stats
select ((((MAX(os_mem_used)) / 1024 ) / 1024) / 1024) max_os_mem_used, ((((MIN(os_mem_used)) / 1024 ) / 1024) / 1024) min_os_mem_used, ((((AVG(os_mem_used)) / 1024 ) / 1024) / 1024) avg_os_mem_used, ((((STDDEV_POP(os_mem_used)) / 1024 ) / 1024) / 1024) stdev_os_mem_used, ((((MAX(length_data + length_index)) / 1024 ) / 1024) / 1024) max_size, ((((MIN(length_data + length_index)) / 1024 ) / 1024) / 1024) min_size, ((((AVG(length_data + length_index)) / 1024 ) / 1024) / 1024) avg_size, ((((STDDEV_POP(length_data + length_index)) / 1024 ) / 1024) / 1024) stdev_size, MAX(num_connections) max_connections,  MIN(num_connections) min_connections, AVG(num_connections) avg_connections, STDDEV_POP(num_connections) stdev_connections, MAX(queries_per_second) max_qps, MIN(queries_per_second) min_qps, AVG(queries_per_second) avg_qps, STDDEV_POP(queries_per_second) stdev_qps from server_statistics;
*************************** 1. row ***************************
max_os_mem_used: 50.743488311768
min_os_mem_used: 0.023044586182
avg_os_mem_used: 1.5759627057922952
stdev_os_mem_used: 2.4064208226596184
max_size: 283.815660957247
min_size: 0.000492287800
avg_size: 10.8213909777686940
stdev_size: 39.9443980717105447
max_connections: 435
min_connections: 0
avg_connections: 16.9734
stdev_connections: 37.3269
stdev_os_mem_used: 2.4064208226596184
max_qps: 9243.6533203125
min_qps: 0.00011409764556447
avg_qps: 216.421064774444
stdev_qps: 1071.72792986232
1 row in set (0.00 sec)
single-host growth over time
mysql> select (select (((length_data + length_index) / 1024) / 1024) as curr_size from server_statistics where server_list_id='44' and CURDATE() 
    I’ve been messing around with the Kontrollbase schema for the last couple of days, writing various queries for the daily reporting scripts that will eventually be an automated pdf report. I’ll give you examples of two of the queries, the first being overall environment stats, and the second being single-host growth over time.

Overall environment stats

select ((((MAX(os_mem_used)) / 1024 ) / 1024) / 1024) max_os_mem_used, ((((MIN(os_mem_used)) / 1024 ) / 1024) / 1024) min_os_mem_used, ((((AVG(os_mem_used)) / 1024 ) / 1024) / 1024) avg_os_mem_used, ((((STDDEV_POP(os_mem_used)) / 1024 ) / 1024) / 1024) stdev_os_mem_used, ((((MAX(length_data + length_index)) / 1024 ) / 1024) / 1024) max_size, ((((MIN(length_data + length_index)) / 1024 ) / 1024) / 1024) min_size, ((((AVG(length_data + length_index)) / 1024 ) / 1024) / 1024) avg_size, ((((STDDEV_POP(length_data + length_index)) / 1024 ) / 1024) / 1024) stdev_size, MAX(num_connections) max_connections,  MIN(num_connections) min_connections, AVG(num_connections) avg_connections, STDDEV_POP(num_connections) stdev_connections, MAX(queries_per_second) max_qps, MIN(queries_per_second) min_qps, AVG(queries_per_second) avg_qps, STDDEV_POP(queries_per_second) stdev_qps from server_statistics;

*************************** 1. row ***************************

max_os_mem_used: 50.743488311768

min_os_mem_used: 0.023044586182

avg_os_mem_used: 1.5759627057922952

stdev_os_mem_used: 2.4064208226596184

max_size: 283.815660957247

min_size: 0.000492287800

avg_size: 10.8213909777686940

stdev_size: 39.9443980717105447

max_connections: 435

min_connections: 0

avg_connections: 16.9734

stdev_connections: 37.3269

stdev_os_mem_used: 2.4064208226596184

max_qps: 9243.6533203125

min_qps: 0.00011409764556447

avg_qps: 216.421064774444

stdev_qps: 1071.72792986232

1 row in set (0.00 sec)
single-host growth over time

mysql> select (select (((length_data + length_index) / 1024) / 1024) as curr_size from server_statistics where server_list_id='44' and CURDATE() <= Creation_time order by Creation_time asc limit 1) as 0_day_size_mb, (select (((length_data + length_index) / 1024) / 1024) as curr_size from server_statistics where server_list_id='44' and DATE_SUB(CURDATE(),INTERVAL 30 DAY) <= Creation_time order by Creation_time asc limit 1) as 30_day_size_mb, ( (select (((length_data + length_index) / 1024) / 1024) as curr_size from server_statistics where server_list_id='44' and CURDATE() <= Creation_time order by Creation_time asc limit 1) - (select (((length_data + length_index) / 1024) / 1024) as curr_size from server_statistics where server_list_id='44' and DATE_SUB(CURDATE(),INTERVAL 30 DAY) <= Creation_time order by Creation_time asc limit 1)) as difference, ( select (select ((select (((length_data + length_index) / 1024) / 1024) as curr_size from server_statistics where server_list_id='44' and CURDATE() <= Creation_time order by Creation_time asc limit 1) - (select (((length_data + length_index) / 1024) / 1024) as curr_size from server_statistics where server_list_id='44' and DATE_SUB(CURDATE(),INTERVAL 30 DAY) <= Creation_time order by Creation_time asc limit 1)) as difference) / (select(select (((length_data + length_index) / 1024) / 1024) as curr_size from server_statistics where server_list_id='44' and CURDATE() <= Creation_time order by Creation_time asc limit 1) as 0_day_size_mb ) * 100) as percent_growth;

+---------------+----------------+--------------+----------------+

| 0_day_size_mb | 30_day_size_mb | difference   | percent_growth |

+---------------+----------------+--------------+----------------+

| 9100.43986130 |  8199.39263916 | 901.04722214 | 9.901139240197 |

+---------------+----------------+--------------+----------------+

1 row in set (0.01 sec)

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN]]>
    Tue, 17 Nov 2009 20:19:14 +0000
    Matt Reid
    Kontrollbase
    Schema design
  

  
    
    http://tokutek.com/?p=806
    http://tokutek.com/2009/11/new-tokudb-2-2-0-feature-more-query-progress-information/
    
Last spring, we added a feature that allows the user to see the progress of writes in a statement. Vadim liked it. In 2.2.0, in &#8220;show processlist&#8221;, we add progress information on reads.

Here is an example of what &#8220;show processlist&#8221; displays on an update:

mysql&gt; show processlist \G
*************************** 1. row ***************************
Id: 1
User: root
Host: localhost
db: test
Command: Query
Time: 7
State: Queried about 1576008 rows, Updated about 197000 rows
Info: update foo set a=9 where a=8


Here is an example of what &#8220;show processlist&#8221; displays on an insert that requires a query:

mysql&gt; show processlist \G
*************************** 1. row ***************************
Id: 1
User: root
Host: localhost
db: test
Command: Query
Time: 6
State: Queried about 1542001 rows, Inserted about 771000 rows
Info: insert into bar select* from foo where a &lt; 5


And here is an example of what show processlist displays in a query:

mysql&gt; show processlist \G
*************************** 1. row ***************************
Id: 1
User: root
Host: localhost
db: test
Command: Query
Time: 4
State: Queried about 2790000 rows
Info: select sum(a) from foo


As we see in the examples above, the work required to execute some statements are a combination of reads and writes. Having progress information on both helps provide the user with a better understanding of what is going on in the system.
    
Last spring, we added a feature that allows the user to see the progress of writes in a statement. Vadim liked it. In 2.2.0, in “show processlist”, we add progress information on reads.


Here is an example of what “show processlist” displays on an update:
mysql> show processlist \G
*************************** 1. row ***************************
Id: 1
User: root
Host: localhost
db: test
Command: Query
Time: 7
State: Queried about 1576008 rows, Updated about 197000 rows
Info: update foo set a=9 where a=8


Here is an example of what “show processlist” displays on an insert that requires a query:
mysql> show processlist \G
*************************** 1. row ***************************
Id: 1
User: root
Host: localhost
db: test
Command: Query
Time: 6
State: Queried about 1542001 rows, Inserted about 771000 rows
Info: insert into bar select* from foo where a < 5


And here is an example of what show processlist displays in a query:
mysql> show processlist \G
*************************** 1. row ***************************
Id: 1
User: root
Host: localhost
db: test
Command: Query
Time: 4
State: Queried about 2790000 rows
Info: select sum(a) from foo


As we see in the examples above, the work required to execute some statements are a combination of reads and writes. Having progress information on both helps provide the user with a better understanding of what is going on in the system.

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN]]>
    Tue, 17 Nov 2009 20:05:16 +0000
    Tokuview Blog
    TokuView
    features
    mysql
    query+progress
    TokuDB
  

  
    
    http://blogs.the451group.com/opensource/?p=1281
    http://feedproxy.google.com/~r/451opensource/~3/RQYnYJcXKg8/
    Larry Augustin confirmed as SugarCRM CEO. Red Hat&#8217;s Fedora Project is 12. And more.
Follow 451 CAOS Links live @caostheory on Twitter and Identi.ca
&#8220;Tracking the open source news wires, so you don&#8217;t have to.&#8221;
For the latest on Oracle&#8217;s acquisition of MySQL via Sun, see Everything you always wanted to know about MySQL but were afraid to ask
# Larry Augustin was confirmed as full-time CEO of SugarCRM. 
# Red Hat, by way of the Fedora Project, announced the launch of Fedora 12.
# Microsoft is to release the .NET Micro Framework under the Apache 2.0 license.
# Simon Phipps introduced the concept of his Software Freedom Scorecard. Plans to propose as joint OSI and FSF initiative.
# Microsoft admitted inadvertent GPLv2 violation in Windows download tool, plans to publish code. 
# Lucid Imagination updated its LucidWorks Certified Distribution for Lucene 2.9. 
# Nexenta Systems announced NexentaStor 2.2, based on the ZFS file system. 
# Univa released version 5.0 of its UniCluster infrastructure and workload management software stack.
# A Canadian law firm suggested purchasers of tech companies are avoiding GPLv3 code to protect their patents. 
# Fluendo updated its Codec Pack of multimedia Linux/Unix codecs. 
# SGI launched the Altix UV Linux-based supercomputer.
# JetBrains announced the general availability of RubyMine 2.0, based on the open source IntelliJ Platform. 
# BonitaSoft’s BPM software is being integrated into eXo Platform DMS’s document management module. 
# Distributor Tech Data created Open Tech, a new program to attract open source ISVs and resellers. 
# WSO2 launched Cloud Platform including Cloud Virtual Machines; Cloud Connectors; and Governance-as-a-Service. 
# The QualiPSo Project released QualiPSo Factory, a SOA-based forge. 
# Sourcesense partnered with Sonatype to provide training, support, and customization services for Maven, Nexus.

    Larry Augustin confirmed as SugarCRM CEO. Red Hat’s Fedora Project is 12. And more.

Follow 451 CAOS Links live @caostheory on Twitter and Identi.ca

“Tracking the open source news wires, so you don’t have to.”
For the latest on Oracle’s acquisition of MySQL via Sun, see Everything you always wanted to know about MySQL but were afraid to ask
# Larry Augustin was confirmed as full-time CEO of SugarCRM. 
# Red Hat, by way of the Fedora Project, announced the launch of Fedora 12.
# Microsoft is to release the .NET Micro Framework under the Apache 2.0 license.
# Simon Phipps introduced the concept of his Software Freedom Scorecard. Plans to propose as joint OSI and FSF initiative.
# Microsoft admitted inadvertent GPLv2 violation in Windows download tool, plans to publish code. 
# Lucid Imagination updated its LucidWorks Certified Distribution for Lucene 2.9. 
# Nexenta Systems announced NexentaStor 2.2, based on the ZFS file system. 
# Univa released version 5.0 of its UniCluster infrastructure and workload management software stack.
# A Canadian law firm suggested purchasers of tech companies are avoiding GPLv3 code to protect their patents. 
# Fluendo updated its Codec Pack of multimedia Linux/Unix codecs. 
# SGI launched the Altix UV Linux-based supercomputer.
# JetBrains announced the general availability of RubyMine 2.0, based on the open source IntelliJ Platform. 
# BonitaSoft’s BPM software is being integrated into eXo Platform DMS’s document management module. 
# Distributor Tech Data created Open Tech, a new program to attract open source ISVs and resellers. 
# WSO2 launched Cloud Platform including Cloud Virtual Machines; Cloud Connectors; and Governance-as-a-Service. 
# The QualiPSo Project released QualiPSo Factory, a SOA-based forge. 
# Sourcesense partnered with Sonatype to provide training, support, and customization services for Maven, Nexus.

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN]]>
    Tue, 17 Nov 2009 16:50:01 +0000
    The 451 Group
    Links
    Software
    .NET Micro Grameowrk
    451 group
    451caostheory
    451group
    altix
    apache
    Bonitasoft
    caostheory
    eXo platform
    fedora
    fluendo
    gplv2
    GPLv3
    JetBrains
    Larry Augustin
    Linux
    Lucene
    lucid imagination
    lucidworks
    matt aslett
    mattaslett
    matthew aslett
    matthe
  

  
    
    http://pooteeweet.org/blog/0/1589
    http://pooteeweet.org/blog/0/1589
    This is just a quick start for a brainstorming of what we all hate in todays CMS (I am including portal/community software here as well and I guess most also applies to web shops) software out there. I have written a very small CMS application myself ages ago so I do not have experience in what its really like writing and maintaining a big one. All I know is that its insanely painful to deal with any of them, though if your site is all about having admins managing tons of static content or end users wanting to interact, there is little way around these ugly beasts. I guess it all boils down to how to persist changes made through and admin panel. Somewhat related is the issue of scalability which to me mainly boils down to how easily can the storage logic be changed without changing the business logic on top.

The biggest gripe that results from these ever so powerful admin panels is the tendency to have these settings stored in the database. In combination with non universal auto generated primary keys it means that not only will it be hard to keep development and production in sync. Some things done inside the admin panel is classic content but a fair bit is configuration which we traditionally want to store inside our software code management repository. Not sure how, but my dream CMS would know the difference and commit configuration to my SCM and it would not depend on meaningless integer ID's that only have meaning inside a single database.

The other big thing I dread is the fact that while most people these days have bought into the MVC concept, few people actually choose a technology that really enables their model layer to offer a true separation of how their data is persisted. I think we all agree that one of the key selling points of all the currently popular CMS is the large number of out of the box features and available plugins. However for the most part they end up using the EAV pattern if they use an RDBMS. In the PHP space I am not sure if we even have a feature rich CMS based on document stores yet, then again I also do not see these as the holy grail either. Document stores are nice in a lot of ways, mainly because they offer a clean solution to having to deal with hierarchical data and more importantly flexibility in managing different content types. But at the same time they loose a lot of the power of an RDBMS that allows for powerful reporting, data manipulation via SQL. Of course the same can be achieved by custom coding but not with the same speed both in development and execution.

Anyways I digress, all I want to say is that my dream CMS gives me as much flexibility in managing data persistence. So no stupid EAV pattern (*), no hardcoded relational tree algorithms or ways to leverage RDBMS specific features from the chosen RDBMS .. be it Oracle, MySQL, PostgreSQL or whatever. But also no insisting that a document store is the holy grail for sorting any kind of data. As such I see much appeal in content stores (we are developing PHP implementation for a content store here at Liip for this very reason) or at the very least using a real ORM like Doctrine (and no none of your active records on steroids qualify).

I guess the last one is very obvious and at different levels this has been done well in existing CMS solutions or not: Define a solid API for adding additional functionality. I think the biggest challenge here is figuring how to balance the want for a low barrier to entry for non developers versus enabling developers to properly design their software. Simple usually means few hooks that are insanely (and increasingly) wide. At the other end you get a situation where everything is done inside layers and layers of inheritance which means you need to open/customize a bazillion number of files to do anything (that cannot be done through the admin panel).

This point is easiest to fail at, since its very grey-ish. At least to me its obviously bogus to mix content with configuration or to use EAV to store custom content types. But its not so easy to balance out ease of use versus flexibility. Either things are too hard for non developers or they are too simple for real developers. Either the app does not gain a large enough user base or it gets too many non developers destroying any sense of consistency (resulting in mass suicide of the only people who are able to fix things). I guess nobody can get it right the first time, then again looking at the current CMS it seems that nobody managed to get to the right balance through evolution either.

(*) Yes my answer to all people using the EAV pattern is to allow the admin panel to generate and execute DDL.
    This is just a quick start for a brainstorming of what we all hate in todays CMS (I am including portal/community software here as well and I guess most also applies to web shops) software out there. I have written a very small CMS application myself ages ago so I do not have experience in what its really like writing and maintaining a big one. All I know is that its insanely painful to deal with any of them, though if your site is all about having admins managing tons of static content or end users wanting to interact, there is little way around these ugly beasts. I guess it all boils down to how to persist changes made through and admin panel. Somewhat related is the issue of scalability which to me mainly boils down to how easily can the storage logic be changed without changing the business logic on top.


The biggest gripe that results from these ever so powerful admin panels is the tendency to have these settings stored in the database. In combination with non universal auto generated primary keys it means that not only will it be hard to keep development and production in sync. Some things done inside the admin panel is classic content but a fair bit is configuration which we traditionally want to store inside our software code management repository. Not sure how, but my dream CMS would know the difference and commit configuration to my SCM and it would not depend on meaningless integer ID's that only have meaning inside a single database.

The other big thing I dread is the fact that while most people these days have bought into the MVC concept, few people actually choose a technology that really enables their model layer to offer a true separation of how their data is persisted. I think we all agree that one of the key selling points of all the currently popular CMS is the large number of out of the box features and available plugins. However for the most part they end up using the EAV pattern if they use an RDBMS. In the PHP space I am not sure if we even have a feature rich CMS based on document stores yet, then again I also do not see these as the holy grail either. Document stores are nice in a lot of ways, mainly because they offer a clean solution to having to deal with hierarchical data and more importantly flexibility in managing different content types. But at the same time they loose a lot of the power of an RDBMS that allows for powerful reporting, data manipulation via SQL. Of course the same can be achieved by custom coding but not with the same speed both in development and execution.

Anyways I digress, all I want to say is that my dream CMS gives me as much flexibility in managing data persistence. So no stupid EAV pattern (*), no hardcoded relational tree algorithms or ways to leverage RDBMS specific features from the chosen RDBMS .. be it Oracle, MySQL, PostgreSQL or whatever. But also no insisting that a document store is the holy grail for sorting any kind of data. As such I see much appeal in content stores (we are developing PHP implementation for a content store here at Liip for this very reason) or at the very least using a real ORM like Doctrine (and no none of your active records on steroids qualify).

I guess the last one is very obvious and at different levels this has been done well in existing CMS solutions or not: Define a solid API for adding additional functionality. I think the biggest challenge here is figuring how to balance the want for a low barrier to entry for non developers versus enabling developers to properly design their software. Simple usually means few hooks that are insanely (and increasingly) wide. At the other end you get a situation where everything is done inside layers and layers of inheritance which means you need to open/customize a bazillion number of files to do anything (that cannot be done through the admin panel).

This point is easiest to fail at, since its very grey-ish. At least to me its obviously bogus to mix content with configuration or to use EAV to store custom content types. But its not so easy to balance out ease of use versus flexibility. Either things are too hard for non developers or they are too simple for real developers. Either the app does not gain a large enough user base or it gets too many non developers destroying any sense of consistency (resulting in mass suicide of the only people who are able to fix things). I guess nobody can get it right the first time, then again looking at the current CMS it seems that nobody managed to get to the right balance through evolution either.

(*) Yes my answer to all people using the EAV pattern is to allow the admin panel to generate and execute DDL.

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN]]>
    Tue, 17 Nov 2009 15:47:00 +0000
    Lukas Smith
    general
  

  
    
    http://infinidb.org/infinidb-blog/join-tomorrows-webinar-on-building-fast-queryanalytic-databases.html
    http://infinidb.org/infinidb-blog/join-tomorrows-webinar-on-building-fast-queryanalytic-databases.html
    I'll be doing a Webinar tomorrow (Nov 18) with some of the MySQL folks on how to build fast read-intensive and analytic databases, so join us if you can.&nbsp; We'll go over some key decision points that can make big differences in the performance of query-based applications and also talk a little about InfiniDB and our approach.Register at: http://www.mysql.com/news-and-events/web-seminars/display-459.html . Hope to see you there...
    I'll be doing a Webinar tomorrow (Nov 18) with some of the MySQL folks on how to build fast read-intensive and analytic databases, so join us if you can.  We'll go over some key decision points that can make big differences in the performance of query-based applications and also talk a little about InfiniDB and our approach.

Register at: http://www.mysql.com/news-and-events/web-seminars/display-459.html . Hope to see you there...

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN]]>
    Tue, 17 Nov 2009 15:45:52 +0000
    Robin Schumacher
  

  
    
    http://www.xaprb.com/blog/?p=1298
    http://www.xaprb.com/blog/2009/11/17/recap-of-portland-opensql-camp-2009/
    I was at OpenSQL Camp 2009 in Portland last weekend.  I thought the event was very well done.  On Friday we had a pizza party at Old Town Pizza, which was awesome.  Saturday and Sunday were breakfast, sessions, lunch (yum), and sessions and hacking.  These were held at souk, a co-working space.  After 5PM, people got together for dinner, beer, etc.

I presented on mk-query-digest &#8212; a live demo of features requested by the audience.  Sessions from others that I thought were particularly good included ones on CouchDB and MongoDB.  I mixed up the time and missed the session from Tokutek on how fractal tree indexes work.  I&#8217;ll try to watch the video if that one was taped.

During the hackathons, Daniel and I worked on Maatkit.  We are laying groundwork for a more powerful mk-query-digest.

As you may know, I created OpenSQL Camp.  But I was not involved in organizing this or the previous event in Germany, which I think is great.  I talked briefly with Eric and Selena about seeing if we could put together a recipe to make the process easy for folks to organize their own.  We should be able to lay out checklists and timelines of major things &#8212; location, shirts, sponsorship, budgeting, food.  Eric and Selena got great food, much better than the Panera catering I had for the first event.  Those kinds of decisions and results should be recorded.  It would be great to be able to treat it like a franchise so anyone could just add water and make their own.

I also might be willing to help organize another on the East Coast, perhaps as soon as next year if I can reduce my workload enough to have the time.  I&#8217;d probably want to do something in or near Washington DC, which is a more convenient location with better public transport than my hometown of Charlottesville.

It all started out as a response to complaints about MySQL&#8217;s annual conference not being a user&#8217;s conference, but nobody actually doing anything about it.  I decided to do something about it, in a more inclusive way.  And judging by the attendees and talks at the two I&#8217;ve gone to, people were happy to say yes to that.  I think if there are continued events, that&#8217;s the ultimate measure of success.

Related posts:OpenSQL Camp events in 2009 There are Going to OpenSQL Camp US 2009 I&#8217;veOpenSQL Camp 2009 plans announced I&#8217;m 
Related posts brought to you by Yet Another Related Posts Plugin.
    I was at OpenSQL Camp 2009 in Portland last weekend.  I thought the event was very well done.  On Friday we had a pizza party at Old Town Pizza, which was awesome.  Saturday and Sunday were breakfast, sessions, lunch (yum), and sessions and hacking.  These were held at souk, a co-working space.  After 5PM, people got together for dinner, beer, etc.


I presented on mk-query-digest — a live demo of features requested by the audience.  Sessions from others that I thought were particularly good included ones on CouchDB and MongoDB.  I mixed up the time and missed the session from Tokutek on how fractal tree indexes work.  I’ll try to watch the video if that one was taped.

During the hackathons, Daniel and I worked on Maatkit.  We are laying groundwork for a more powerful mk-query-digest.

As you may know, I created OpenSQL Camp.  But I was not involved in organizing this or the previous event in Germany, which I think is great.  I talked briefly with Eric and Selena about seeing if we could put together a recipe to make the process easy for folks to organize their own.  We should be able to lay out checklists and timelines of major things — location, shirts, sponsorship, budgeting, food.  Eric and Selena got great food, much better than the Panera catering I had for the first event.  Those kinds of decisions and results should be recorded.  It would be great to be able to treat it like a franchise so anyone could just add water and make their own.

I also might be willing to help organize another on the East Coast, perhaps as soon as next year if I can reduce my workload enough to have the time.  I’d probably want to do something in or near Washington DC, which is a more convenient location with better public transport than my hometown of Charlottesville.

It all started out as a response to complaints about MySQL’s annual conference not being a user’s conference, but nobody actually doing anything about it.  I decided to do something about it, in a more inclusive way.  And judging by the attendees and talks at the two I’ve gone to, people were happy to say yes to that.  I think if there are continued events, that’s the ultimate measure of success.

Related posts:
OpenSQL Camp events in 2009 There are 
Going to OpenSQL Camp US 2009 I’ve
OpenSQL Camp 2009 plans announced I’m 

Related posts brought to you by Yet Another Related Posts Plugin.

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN]]>
    Tue, 17 Nov 2009 15:32:50 +0000
    Baron Schwartz (xaprb)
    Conferences
    OpenSQL Camp
    PostgreSQL
    SQL
  

  
    
    http://www.clusterdb.com/?p=681
    http://www.clusterdb.com/mysql-cluster/mysql-cluster-geographic-replication-deep-dive-webinar/
    I will be presenting a free Webinar on Geographic Replication for MySQL Cluster at 9:00 am (UK time) on Tuesday 24 November.
Multi-Master Replication for HA with MySQL Cluster

MySQL Cluster has been deployed into some of the most demanding web, telecoms and enterprise /
government workloads, supporting 99.999% availability with real time performance and linear write scalability.
You can register on-line here.
Tune into this webinar where you can hear from the MySQL Cluster product management team provide a detailed &#8220;deep dive&#8221; into one of MySQL Cluster&#8217;s key capabilities &#8211; Geographic Replication.
In this session, you will learn how using Geographic Replication enables your applications to:

Achieve higher levels of availability within a data center or across a WAN
Locate data closer to users, providing lower latency access
Replicate to other MySQL storage engines for complex data analysis and reporting of real time data
Gow to get started with Geographic Replication

Tuesday, November 24, 2009: 10:00 Central European time

Tue, Nov 24:  09:00 Western European time
Tue, Nov 24:  11:00 Eastern European time

The presentation will be approximately 1 hour long, including on-line Q&amp;A.
    I will be presenting a free Webinar on Geographic Replication for MySQL Cluster at 9:00 am (UK time) on Tuesday 24 November.
Multi-Master Replication for HA with MySQL Cluster

MySQL Cluster has been deployed into some of the most demanding web, telecoms and enterprise /

government workloads, supporting 99.999% availability with real time performance and linear write scalability.
You can register on-line here.
Tune into this webinar where you can hear from the MySQL Cluster product management team provide a detailed “deep dive” into one of MySQL Cluster’s key capabilities – Geographic Replication.
In this session, you will learn how using Geographic Replication enables your applications to:

Achieve higher levels of availability within a data center or across a WAN
Locate data closer to users, providing lower latency access
Replicate to other MySQL storage engines for complex data analysis and reporting of real time data
Gow to get started with Geographic Replication

Tuesday, November 24, 2009: 10:00 Central European time

Tue, Nov 24:  09:00 Western European time
Tue, Nov 24:  11:00 Eastern European time

The presentation will be approximately 1 hour long, including on-line Q&A.

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN]]>
    Tue, 17 Nov 2009 14:15:44 +0000
    Andrew Morgan
    MySQL Cluster
    MySQL Replication
    HA
    High Availability
    MySQL
  

  
    
    tag:blogger.com,1999:blog-14455177.post-6936738247882504971
    http://mikaelronstrom.blogspot.com/2009/11/new-version-of-benchmark-scripts-also.html
    I updated my benchmark scripts this week. These scripts can nowrun:- Sysbench benchmarks for MySQL and Drizzle- DBT2 benchmarks for MySQL and MySQL Cluster- TPC-W benchmark for MySQL ClusterThere is also a number of scripts to start and stopMySQL, Drizzle and MySQL Cluster nodes.In this version I added Drizzle support for sysbench and alsoadded a README-AUTOMATED file that describes the steps neededto set-up a completely automated sysbench run for MySQLand Drizzle.To run a MySQL sysbench benchmark one needs the DBT2 tarball,the sysbench tarball and a MySQL tarball (gzipped tarballs).The tarball is found on www.iclaustron.com in the downloadssection and this version is named dbt2-0.37.47.tar.gz.
    run:

- Sysbench benchmarks for MySQL and Drizzle
- DBT2 benchmarks for MySQL and MySQL Cluster
- TPC-W benchmark for MySQL Cluster

There is also a number of scripts to start and stop
MySQL, Drizzle and MySQL Cluster nodes.

In this version I added Drizzle support for sysbench and also
added a README-AUTOMATED file that describes the steps needed
to set-up a completely automated sysbench run for MySQL
and Drizzle.

To run a MySQL sysbench benchmark one needs the DBT2 tarball,
the sysbench tarball and a MySQL tarball (gzipped tarballs).

The tarball is found on www.iclaustron.com in the downloads
section and this version is named dbt2-0.37.47.tar.gz.

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN]]>
    Tue, 17 Nov 2009 13:00:00 +0000
    Mikael Ronstr&ouml;m
  

  
    
    http://www.facebook.com/note.php?note_id=183311655932
    http://www.facebook.com/note.php?note_id=183311655932
    I pushed more patches to the Facebook patch for MySQL. Expect more patches as several people are on the team (Domas committed a change today). We will soon be publishing patches for MySQL 5.1. The patches include:

Protect InnoDB LRU during full table scan. This is a backport from the InnoDB plugin version 1.0.5 (pending release in MySQL 5.1.41) and it fixes bug 45015. This adds the my.cnf variables innodb_old_blocks_time and innodb_old_blocks_pct.
Add my.cnf variable innodb_mmap_buffer_pool to allocate memory for InnoDB bufer pool using mmap. Read the code. This is a work in progress.
Provide a fast and non-blocking alternative to FLUSH TABLES WITH READ LOCK. Read this for context. This adds START TRANSACTION WITH CONSISTENT INNODB SNAPSHOT that starts an Innodb transaction and returns the master binlog filename and offset. The command does not block on long running queries. When this is used for backups, replication should be restarted at those return values. The new START command returns 1 row with 2 columns (File, Position) on success. This fixes bug 48124 until MySQL provides the proper fix.
Add my.cnf variable innodb_flush_neighbors_on_checkpoint. This is TRUE by default to match default InnoDB behavior. When FALSE, dirty neighbor pages in an extent are not written to disk during a checkpoint when the min LSN of the oldest dirty page must be advanced. This only changes the behavior when all pages with an old modified LSN are flushed to enforce the checkpoint constraint. This does not modify what is done when pages are flushed for other reasons. This can reduce the rate at which page writes are done. This reverts part of the change added in MySQL 4.0.14. I have written about this several times including here. We need more data from production to determine whether this is useful.
Add FLIFO scheduling for InnoDB. This is described in another note.

    Facebook patch for MySQL. Expect more patches as several people are on the team (Domas committed a change today). We will soon be publishing patches for MySQL 5.1. The patches include:

Protect InnoDB LRU during full table scan. This is a backport from the InnoDB plugin version 1.0.5 (pending release in MySQL 5.1.41) and it fixes bug 45015. This adds the my.cnf variables innodb_old_blocks_time and innodb_old_blocks_pct.
Add my.cnf variable innodb_mmap_buffer_pool to allocate memory for InnoDB bufer pool using mmap. Read the code. This is a work in progress.
Provide a fast and non-blocking alternative to FLUSH TABLES WITH READ LOCK. Read this for context. This adds START TRANSACTION WITH CONSISTENT INNODB SNAPSHOT that starts an Innodb transaction and returns the master binlog filename and offset. The command does not block on long running queries. When this is used for backups, replication should be restarted at those return values. The new START command returns 1 row with 2 columns (File, Position) on success. This fixes bug 48124 until MySQL provides the proper fix.
Add my.cnf variable innodb_flush_neighbors_on_checkpoint. This is TRUE by default to match default InnoDB behavior. When FALSE, dirty neighbor pages in an extent are not written to disk during a checkpoint when the min LSN of the oldest dirty page must be advanced. This only changes the behavior when all pages with an old modified LSN are flushed to enforce the checkpoint constraint. This does not modify what is done when pages are flushed for other reasons. This can reduce the rate at which page writes are done. This reverts part of the change added in MySQL 4.0.14. I have written about this several times including here. We need more data from production to determine whether this is useful.
Add FLIFO scheduling for InnoDB. This is described in another note.

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN]]>
    Tue, 17 Nov 2009 05:09:20 +0000
    Mark Callaghan
  

  
    
    http://www.mysqlperformanceblog.com/?p=1743
    http://www.mysqlperformanceblog.com/2009/11/16/chicago-atlanta-last-two-training-locations-of-the-year/
    A final update on training for this year - registration for InnoDB/XtraDB training in Chicago (8th December) and Atlanta (10th December) are now open.
While in Atlanta I'll be giving a talk at the Atlanta PHP User Group on Optimizing MySQL Performance (details to be posted to their website shortly).  If you're in Chicago and would like me to speak at your group on 7th-8th December, let me know!
    
    Entry posted by Morgan Tocker |
      No comment
    Add to:  |  |  |  | 
    A final update on training for this year - registration for InnoDB/XtraDB training in Chicago (8th December) and Atlanta (10th December) are now open.

While in Atlanta I'll be giving a talk at the Atlanta PHP User Group on Optimizing MySQL Performance (details to be posted to their website shortly).  If you're in Chicago and would like me to speak at your group on 7th-8th December, let me know!
    

    
Entry posted by Morgan Tocker |
      No comment
    
Add to:  |  |  |  | 

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN]]>
    Mon, 16 Nov 2009 22:37:24 +0000
    Morgan Tocker
    mysql
  

  
    
    http://www.mysqlperformanceblog.com/?p=1736
    http://www.mysqlperformanceblog.com/2009/11/16/interviews-for-infinidb-and-tokudb-are-next/
    I forwarded on a list of questions about PBXT to Paul McCullagh today.  While Paul's busy answering them, I'd like to announce that Robert Dempsey (InfiniDB storage engine) and Bradley C. Kuszmaul (TokuDB storage engine) have also accepted an interview.  If you have any questions about either storage engine, please post them here by Friday 20th November.
    
    Entry posted by Morgan Tocker |
      No comment
    Add to:  |  |  |  | 
    I forwarded on a list of questions about PBXT to Paul McCullagh today.  While Paul's busy answering them, I'd like to announce that Robert Dempsey (InfiniDB storage engine) and Bradley C. Kuszmaul (TokuDB storage engine) have also accepted an interview.  If you have any questions about either storage engine, please post them here by Friday 20th November.

    

    
Entry posted by Morgan Tocker |
      No comment
    
Add to:  |  |  |  | 

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN]]>
    Mon, 16 Nov 2009 21:53:12 +0000
    Morgan Tocker
    community
    mysql
    storage engine
  

  
    
    http://www.facebook.com/note.php?note_id=183022275932
    http://www.facebook.com/note.php?note_id=183022275932
    The 10 largest tables in one production database use 17.8G with InnoDB, 11.1G with MyISAM and 8.6G with packed MyISAM. From tests done by Ryan show that InnoDB compression reduces our database to 59% of the original size. Assuming this holds for the top 10 tables (and I have not tested that), then the result would be 10.5G.

This can save me a lot of work. I expected to add support to MySQL for using different storage engines on the same partitioned table. With that feature, I could use compressed MyISAM for the inactive data and InnoDB for the active data. Now I want to use compressed InnoDB for everything.
    tests done by Ryan show that InnoDB compression reduces our database to 59% of the original size. Assuming this holds for the top 10 tables (and I have not tested that), then the result would be 10.5G.

This can save me a lot of work. I expected to add support to MySQL for using different storage engines on the same partitioned table. With that feature, I could use compressed MyISAM for the inactive data and InnoDB for the active data. Now I want to use compressed InnoDB for everything.
PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN]]>
    Mon, 16 Nov 2009 20:12:05 +0000
    Mark Callaghan
  

  
    
    http://www.mysqlperformanceblog.com/?p=1730
    http://www.mysqlperformanceblog.com/2009/11/16/shard-early-shard-often/
    I wrote a post a while back that said why you don't want to shard.  In that post that I tried to explain that hardware advances such as 128G of RAM being so cheap is changing the point at which you need to shard, and that the (often omitted) operational issues created by sharding can be painful.
What I didn't mention was that if you've established that you will need to eventually shard, is it better to just get it out of the way early?  My answer is almost always no. That is to say I disagree with a statement I've been hearing recently; "shard early, shard often".  Here's why:

There's an order of magnitude better performance that can be gained by focusing on query/index/schema optimization.  The gains from sharding are usually much lower.
If you shard first, and then decide you want to tune query/index/schema to reduce server count, you find yourself in a more difficult position - since you have to apply your changes across all servers.

Or to phrase that another way:
I would never recommend sharding to a customer until I had at least reviewed their slow query log with mk-query-digest and understood exactly why each of the queries in that report were slow.  While we have some customers who have managed to create their own tools for shard automation, it's always easier to propose major changes to how data is stored before you have a cluster of 50+ servers.
    
    Entry posted by Morgan Tocker |
      5 comments
    Add to:  |  |  |  | 
    I wrote a post a while back that said why you don't want to shard.  In that post that I tried to explain that hardware advances such as 128G of RAM being so cheap is changing the point at which you need to shard, and that the (often omitted) operational issues created by sharding can be painful.

What I didn't mention was that if you've established that you will need to eventually shard, is it better to just get it out of the way early?  My answer is almost always no. That is to say I disagree with a statement I've been hearing recently; "shard early, shard often".  Here's why:

There's an order of magnitude better performance that can be gained by focusing on query/index/schema optimization.  The gains from sharding are usually much lower.
If you shard first, and then decide you want to tune query/index/schema to reduce server count, you find yourself in a more difficult position - since you have to apply your changes across all servers.

Or to phrase that another way:

I would never recommend sharding to a customer until I had at least reviewed their slow query log with mk-query-digest and understood exactly why each of the queries in that report were slow.  While we have some customers who have managed to create their own tools for shard automation, it's always easier to propose major changes to how data is stored before you have a cluster of 50+ servers.
    

    
Entry posted by Morgan Tocker |
      5 comments
    
Add to:  |  |  |  | 

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN]]>
    Mon, 16 Nov 2009 19:59:16 +0000
    Morgan Tocker
    mysql
    tips
    shard
    sharding