by Charles He, Soumyadip Banerjee, Tao Tao, & Krishna Puttaswamy

Image Credit: NASA/JPL (Raghvendra Sahai). This image of the Red Rectangle was produced from data in the Hubble Space Telescope (HST) archive, taken with the Wide Field Planetary Camera 2 onboard the HST as part of Guest Observer program 7297.

Last year Airbnb grew to a point that a scalable and distributed storage …

What is JSON
JSON is an text based, human readable format for transmitting data between systems, for serializing objects and for storing document store data for documents that have different attributes/schema for each document. Popular document store databases use JSON (and the related BSON) for storing and transmitting data.

Problems with JSON in MySQL
It is difficult to inter-operate between MySQL and MongoDB (or other document databases) because JSON has traditionally been very difficult to work with. Up until recently, JSON is just a TEXT document. I said up until recently, so what has changed? The biggest thing is that there are new JSON UDF by Sveta Smirnova, which are part of the MySQL 5.7 Labs releases. Currently the JSON UDF are up to version 0.0.4. While these new UDF are a welcome edition to the MySQL database, they don't solve the really tough …

What is JSON
JSON is an text based, human readable format for transmitting data between systems, for serializing objects and for storing document store data for documents that have different attributes/schema for each document. Popular document store databases use JSON (and the related BSON) for storing and transmitting data.

Problems with JSON in MySQL
It is difficult to inter-operate between MySQL and MongoDB (or other document databases) because JSON has traditionally been very difficult to work with. Up until recently, JSON is just a TEXT document. I said up until recently, so what has changed? The biggest thing is that there are new JSON UDF by Sveta Smirnova, which are part of the MySQL 5.7 Labs releases. Currently the JSON UDF are up to version 0.0.4. While these new UDF are a welcome edition to the MySQL database, they don't solve the really tough …

A customer of ours had an interesting problem regarding a query that was taking too long, around 55s. Looking at the query with the query profiler we found that it was spending most of its time in the "statistics" phase. Now the query was pretty complex, it contained nearly 20 tables with INNER JOINs, LEFT JOINs and even some subqueries. However the tables were small and fetching all the data shouldn't have taken the 55 seconds the query was taking. The problem was that the optimiser was spending too much time evaluating and finding the optimal execution plan.

There are two options in MySQL with which you can control the optimiser's behaviour a bit. The first one is optimizer_prune_level. The pruner discards non-optimal execution plans early without evaluating them fully. It is turned on by default and is not recommended to turn off unless there's a really good reason. For testing purposes we turned the pruner off for this query, but …

The InnoDB full-text search capability is an exciting feature. The full-text search itself is generally useful to have in an RDBMS. If an application is using all InnoDB tables except for one that is used for full-text searches, now that last table can be switched to InnoDB. If putting the full-text data in a MyISAM table led to scalability problems, duplication, or a less-than-ideal schema design, now those issues can be addressed.

In this post, I’ll take you through some of the basics of setting up and querying an InnoDB FULLTEXT search index. I’ll leave the scalability and performance aspects to Jimmy’s and Vinay’s blog posts, and just use some toy-sized data for demonstration purposes.

Creating a Table with a …

There are two search engines that have promise that made themselves public in 2010: DuckDuckGo and Blekko. DuckDuckGo has active search spam removal, you can access it via secure HTTP (HTTPS), and is a search engine that also relies on crowd sourced data.

Web of Trust has reputation ratings of over millions of websites, and has an active community of about 15 million users now. Best of all, there’s no bots doing these ratings, but community members (trust metrics are crowd sourced).

DuckDuckGo and Web of Trust have a partnership now, so you can simply change the settings to display WoT ratings instead of the favicons when …

Sphinx search is a full text search engine, commonly used with MySQL.

There are some misconceptions about Sphinx and its usage. Following is a list of some of Sphinx’ properties, hoping to answer some common questions.

• Sphinx is not part of MySQL/Oracle.
• It is a standalone server; an external application to MySQL.
• Actually, it is not MySQL specific. It can work with other RDBMS: PostgreSQL, MS SQL Server.
• And, although described as “free open-source SQL full-text search engine”, it is not SQL-specific: Sphinx can read documents from XML.
• It is often described as “full text search for InnoDB”. This description is misleading. Sphinx indexes text; be it from any storage engine or external source. It solves, in a way, the issue of “FULLTEXT is only supported by MyISAM”. Essentially, it provided full-text …

Background Knowledge

The Search Plugin for TaskFreak! created by DaDaemon and xdu v0.0.1 (March 26, 2007) was designed to create a simple, quick search capability of the tasks title and description. As well it only searched through he current task view (tasks visible at the time) and tasks that are not completed. For some this was not what was desired and would rather have the Search Plugin search through all tasks weather completed or not and as well search through the comments of tasks along with the title and description. I’ll show you how this is done using Searcher, bchristie and davidlmansfield instructions posted on the TaskFreak! Forums.

Solution – Add the Ability to Search All Tasks

Edit the “index.php” located in …

This Thursday (December 3rd, 16:00 UTC – note the different time), Bill Karwin will talk about Practical Full-Text Search in MySQL. He'll introduce and compare five different approaches of full-text search with MySQL, using built-in functionality as well as third-party tools. It's interesting to see how vastly performance can vary, depending on which tools you use for which purposes.

For MySQL University sessions, point your browser to this page. You need a browser with a working Flash plugin. You may register for a Dimdim account, but you don't have to. (Dimdim is the conferencing system we're using for MySQL University sessions. It provides integrated voice …

This Thursday (December 3rd, 16:00 UTC – note the different time), Bill Karwin will talk about Practical Full-Text Search in MySQL. He'll introduce and compare five different approaches of full-text search with MySQL, using built-in functionality as well as third-party tools. It's interesting to see how vastly performance can vary, depending on which tools you use for which purposes.

For MySQL University sessions, point your browser to this page. You need a browser with a working Flash plugin. You may register for a Dimdim account, but you don't have to. (Dimdim is the conferencing system we're using for MySQL University sessions. It provides integrated voice …