Yesterday I ran into the article which sheds some light on FaceBook search implementation. As we're recently a lot into search having implemented a bunch of search projects ourselves and helped number a of customers with their full text search needs I decided to post my thoughts on this matter.
First I was surprised article talks about just 1TB of data. I knew FaceBook has much more content than that. Though it seems the article only speaks about searching user profiles and similar global objects not full database of posts and comments. At FaceBook scale it is easy to fit 1TB in distributed memory which makes a lot of things easier.
Second this I was surprised about is about writing crawler for a site.... why would you do that if there is already information in the database which is much faster to extract. Even if data needs to be aggregated from …
[Read more]