Does your dataset consist of InnoDB tables with large BLOB data such that the data is stored in external BLOB pages? Was the dataset created in MySQL version 5.1 and below and without using the InnoDB plugin, or with InnoDB plugin but with MySQL version earlier than 5.1.55? If the answer to both the questions are "YES" then it could very well be that you have a hidden corruption lying around in your dataset.

The post Beware of MySQL BLOB Corruption in Older Versions appeared first on ovais.tariq.

This article discusses the storage (inline and external) of field data in the InnoDB storage engine. All fields of variable length types like VARCHAR, VARBINARY, BLOB and TEXT can be stored inline within the clustered index record, or stored externally in separate BLOB pages outside of the index record (but within the same tablespace). All of these fields can be classified as large objects. These large objects are either binary large objects or character large objects. The binary large objects do not have an associated character set, whereas the character large objects do.

Within the InnoDB storage engine there is no difference between the way character large objects and binary large objects …

Some of you may be aware that MySQL is increasingly using an Oracle-internal bug tracker. You can see these large bug numbers mentioned alongside smaller public bug numbers in recent MySQL release notes. If you’re particularly unlucky, you  just get a big Oracle-internal bug number. For a recently fixed bug, I dug further, posted up on the Percona blog: http://www.mysqlperformanceblog.com/2011/11/20/bug12704861/

Possibly interesting reading for those of you who interested in InnoDB, MySQL, BLOBs and crash recovery.

I’m sure you are aware that the last Percona server release includes a new improved MEMORY storage engine for MySQL.
This new engine is based on Dynamic Row Format and offers some of great features, specialy for VARCHAR, VARBINARY, TEXT and BLOB fields in MEMORY tables.

But because this new MEMORY engine by Percona has some limitations and because Percona server hasn’t used it for its internal temporary tables yet, I would like to talk about what can be the real benefits to have a brand new MEMORY engine based on Dynamic row format specialy for internal memory tables.

Just remember or discover how MySQL uses …

Having reviewed different table structures designed by different people, I have come to the conclusion that binary and non-binary string data types are used without consideration of the consequences of choosing either one. The confusion stems from the fact that both non-binary and binary string data appear to store characters because they can be saved as quoted string.

Some of you may have noticed that blob streaming has been merged into the main Drizzle tree recently. There are a few hooks inside the Drizzle kernel that PBMS uses, and everything else is just in the plug in.

For those not familiar with PBMS it does two things: provide a place (not in the table) for BLOBs to be stored (locally on disk or even out to S3) and provide a HTTP interface to get and store BLOBs.

This means you can do really neat things such as have your BLOBs replicated, consistent and all those nice databasey things as well as easily access them in a scalable way (everybody knows how to cache HTTP).

This is a great addition to the AlsoSQL arsenal of Drizzle. I’m looking forward to it advancing and being adopted (now much easier that it’s in the main repository)

• …

Another (AFAIK) undocumented part of the Storage Engine API:

We all know what a normal row looks like in Drizzle/MySQL row format (a NULL bitmap and then column data):

Nothing that special. It’s a fixed sized buffer, Field objects reference into it, you read out of it and write the values into your engine. However, when you get to BLOBs, we can’t use a fixed sized buffer as BLOBs may be quite large. So, the format with BLOBS is the bit in the row is a length of the blob (1, 2, 3 or 4 bytes – in Drizzle it’s only 3 or 4 bytes now and soon only 4 bytes once we fix a bug that isn’t interesting to discuss here). The Second part of the in-row part is a pointer to a location in memory where the BLOB is stored. So a row that has a BLOB in it looks something like this:

…

I ran into a very interesting issue with a customer recently. The customer was storing JPEGs in MySQL (as BLOBs obviously) and was wanting to setup some additional MySQL slaves. Nothing crazy there. So I did the normal steps of scheduling a mysqldump with the --all-databases and --master-data options. Trouble is, the next day when I start importing the dump, 'mysql' bombed up with "ERROR at line X: Unknown command '\0'". I used 'sed' to pull out the lines around the error and didn't really notice anything out of the ordinary. After a bit of Googling, one suggestion was to do a simultaneous backup and restore (basically a "mysqldump .. | mysql -h ..."). In this case, since I was using an existing slave to do the backups, this was something that was feasible to do. Unfortunately, the result was the same.

It turns out that 'mysqldump' will not always product a dump that is always BLOB friendly. To get around this, the '--hex-blob' option …

Raj Kissu is in the papers today. He completed a Google Summer of Code 2008 project with MySQL, hacking on blob streaming for phpMyAdmin. In fact, his project was so good, he has commit access to the phpMyAdmin tree :-)

Today, The Star had an article about him (and two other students), titled For the love of code. He said:

But Raj has already heard ­inquiries for his project. “A ­company that has developed a transactional engine using MySQL server have clients who want to test BlobStreaming,” he said.

Kudos Raj. I think we’ll see more great work from Paul McCullagh and Barry Leslie, as more happens with Scalable BLOB Streaming for MySQL happens.

In fact, Barry Leslie …

• Paul McCullagh presents
• BLOB
  • invented by Jim Starkey
  • Basic Large OBject
  • Binary Large OBject
  • photos, films, mp4 files, pdfs, etc
• how MySQL handles BLOBs
  • mysql client send buffer -> receive buffer on the server (max_allowed_packet)
  • streaming a BLOB
    • continuous data stream
    • stream BLOB data directly in and out of the database
    • store BLOBs of any size (>4GB) in the database
    • create a scalable back-end that can handle any throughput and storage requirements. Wouldn't need to know in advance how big the database will get
    • provide an open system that can be used by all engines
    • provide extensions for …