We used to use Google mini to index our site to serve search results to our users. It worked great for a while until sometime back it just stopped indexing. It would get a fatal error and stop index after couple urls. It was also not so kind about sending me an email to let me know that it has stopped indexing. It was still serving pages which were indexed prior to this so we didn?t realize it wasn?t indexing new content.
And than one day we logged in to the admin console to see whats going on with it and found out it wasn?t indexing. I checked the logs and found out that it was stopping due too too many 4xx errors. After looking at logs on our web server I …[Read more...]