There are two search engines that have promise that made themselves public in 2010: DuckDuckGo and Blekko. DuckDuckGo has active search spam removal, you can access it via secure HTTP (HTTPS), and is a search engine that also relies on crowd sourced data.

Web of Trust has reputation ratings of over millions of websites, and has an active community of about 15 million users now. Best of all, there’s no bots doing these ratings, but community members (trust metrics are crowd sourced).

DuckDuckGo and Web of Trust have a

  [Read more...]

Sphinx search is a full text search engine, commonly used with MySQL.

There are some misconceptions about Sphinx and its usage. Following is a list of some of Sphinx’ properties, hoping to answer some common questions.

• Sphinx is not part of MySQL/Oracle.
• It is a standalone server; an external application to MySQL.
• Actually, it is not MySQL specific. It can work with other RDBMS: PostgreSQL, MS SQL Server.
• And, although described as “free open-source SQL full-text search engine”, it is not SQL-specific: Sphinx can read documents from XML.
• It is often described as “full text search for InnoDB”. This description is misleading. Sphinx indexes text; be it from any storage engine or external source. It solves, in a way, the issue of “FULLTEXT is

  [Read more...]

Background Knowledge

---

The Search Plugin for TaskFreak! created by DaDaemon and xdu v0.0.1 (March 26, 2007) was designed to create a simple, quick search capability of the tasks title and description. As well it only searched through he current task view (tasks visible at the time) and tasks that are not completed. For some this was not what was desired and would rather have the Search Plugin search through all tasks weather completed or not and as well search through the comments of tasks along with the title and description. I’ll show you how this is done using Searcher, bchristie and davidlmansfield instructions posted on the TaskFreak! Forums.

Solution – Add the Ability to  [Read more...]

In the past few weeks I've been implementing advanced search at Plaxo, working quite closely with Solr enterprise search server. Today, I saw this relatively detailed comparison between Solr and its main competitor Sphinx (full credit goes to StackOverflow user mausch who had been using Solr for the past 2 years). For those still confused, Solr and Sphinx are similar to MySQL FULLTEXT search, or for those even more confused, think Google (yeah, this is a bit of a stretch, I know).

Similarities

• Both Solr and Sphinx satisfy all of your requirements. They're fast and designed to index and search large bodies of data efficiently.
• Both

  [Read more...]

I just discovered today in the MySQL 5.1 Reference Manual a handy set of additional indexes in the System Navigation section.

• Index
• Standard Index
• C Function Index
• Command Index
• Function Index
• INFORMATION_SCHEMA Index
• Transaction Isolation Level Index
• JOIN Types Index
• Operator Index
• Option Index
• Privileges Index
• SQL Modes Index
• Status Variable Index
• Statement/Syntax Index
• System Variable Index

Perhaps they have been around for some time and I’ve not noticed, but there are much better then searching when you know the content type as per the index list on what you are searching for.

You guys are generating an amazing amount of feedback on your blogs. Matt mentioned in the April Wrap-Up that there were 8.6 million comments! Comments are flying in every second of the day.

And have you ever had one of those blog posts that was good, but the real action was in the comments? The blog post is only half the story, it’s the feedback from everyone else that fills in the rest. To make it easier to find the second half of these stories we’ve added comment search to WordPress.com search.

Select the comments options from the WordPress.com search page and we’ll hunt through the millions of comments that have been added to WordPress.com blogs to find what you are looking

  [Read more...]

One of the things that has always bothered me about replication is that the binary logs are written to disk and then read from disk.

There is are two threads which are for the most part, unaware of each other.

One thread reads the remote binary logs, and the other writes them to disk.

While the Linux page buffer CAN work to buffer these logs, the first write will cause additional disk load.

One strategy, which could seriously boost performance in some situations, would be to pre-read say 10-50MB of data and just keep it in memory.

If a slave is catching up, it could have GIGABYTES of binary log data from the master. It would then write this to disk. These reads would then NOT come from cache.

Simply using a small buffer could solve this problem.

One HACK would be to use a ram drive or tmpfs for logs. I assume that the log thread will block if

  [Read more...]

As I’ve written in my previous post “1/3 Implementing an AutoSuggest feature using MySQL fulltext indices”, it’s possible to use the MySQL/MyISAM full-text index to extract search words for an AutoSuggest feature with great performance (because the index tree is used actually). This tool, called myisam_suggest, is my first implementation of this. Download Here: myisam_suggest.c [...]

Spinn3r is hiring for an experienced Senior Systems Administrator with solid Linux and MySQL skills and a passion for building scalable and high performance infrastructure.

About Spinn3r:

Spinn3r is a licensed weblog crawler used by search engines, weblog analytic companies, and generally anyone who needs access to high quality weblog data.

We crawl the entire blogosphere in realtime, remove spam, rank, and classifying blogs, and provide this information to our customers.

Spinn3r is rare in the startup world in that we’re actually profitable. We’ve proven our business model which gives us a significant advantage in future

  [Read more...]

Download the PDF: http://www.mysqlfulltextsearch.com/full_text.pdf

Default search by relevance, default sort is by relevance

Boolean search is also popular. cats AND dogs. No default sorting, so you need to order the results yourself

Phrase search

MySQL Full Text Index, only available with MyISAM, and it supports natural language and Boolean search. ft_min_word_len - 4 characters per word by default is indexed. Frequency based ranking, doesn’t count distance between words

SELECT * FROM articles WHERE MATCH (title,body) AGAINST (’database’ IN NATURAL LANGUAGE MODE);

For Boolean, you use AGAINST (’cat AND dog’ IN BOOLEAN MODE).

n-gram fulltext plugin for CJK languages are available as plugins

DRBD and MySQL FullText

  [Read more...]

A

A

So, the whole world changed at dealnews when Yahoo! linked us. We realized that our current infrastructure was not scaling very well. We had to make a change.

The Problem

Even though we were using all sorts of cool techniques, the server architecture was really still just a bunch of web servers all serving the same content. In addition to that, our existing systems as the time used a pull method. When a request came in, memcache was checked, if the data was not there, it was fetched from our main MySQL server. So, when there is no data in the cache or when it expires, this was very bad. Like when Yahoo! hit us. Some cache item would expire and 60,000 users would hit a page and each page would try and create the cache item.

  [Read more...]

I wrote previously about looking for a more powerful search solution, and I mentioned that Xapian wasn’t quite so convenient in indexing my data. I then chose to experiment with sphinx a little more, and proceeded to create a number of search engines and indexed a number of data sources in order to decide which direction to go. Unfortunately, while sphinx was convenient and still provides an excellent backend for basic search indexes, I’m revisiting Xapian once again based on it’s more-than-anticipated flexibility. I was brief in my explanation of Xapian however, and didn’t mention some of the more important and powerful aspects of it.

Xapian provides an API

Xapian is primarily an API for search indexing/data retrieval. They do provide a handy utility called Omega (available here) for

  [Read more...]

I’m finally taking the time to continue this series =)

• Part One
• Part Two
• Part Three

Lyrics Grep

For testing purposes, I went ahead and scraped about 60,000 song lyrics off of a number of sites and developed a simple search engine for them. The script that did the scraping is pretty nasty (to handle equally nasty HTML that I had to parse through), so I’m going to refrain from posting that script and save myself some embarrassment. Make a pot of coffee, sit down, and write one yourself (or something else that’s similar enough).

Database Schema

I created a new database called lyricsgrep with the

  [Read more...]

Note: Part I is located at the page that describes part one

Disclaimer
Just a minor clarification to anyone that was confused: I am currently experiencing Sphinx for the first time. Everything I’m writing about is new to me as well, for the most part. So far, I’m drooling over some of it’s capabilities; I may come back in a month and rip it a new ass hole.

Back to Configuration… (not really, this is the bitching section)

In preparation for my previous post about Sphinx, I had originally played with a number of configuration options, and even encountered a couple of issues that caused my confused butt to have to debug a number of things, and even recompile

  [Read more...]

I’ve been looking for a good solution to manage full text search for large data chunks (2GB - 100GB). I’ve written a couple of solutions using Xapian with limited success, but unfortunately I haven’t been satisfied with it overall. Performance was good, but there were a number of issues with flexibility that have me ultimately looking for another solution.

At my usual day job, the topic was brought up and I mentioned Xapian and Lucene as solutions, however we’re looking to stay away from Java as it’s not currently in our architecture, and as I stated before: Xapian doesn’t quite have the capabilities I’m looking for to handle even my own systems. Someone brought up Sphinx as something that was being looked into, and I jumped into the typical research

  [Read more...]