We used to use Google mini to index our site to serve search
results to our users. It worked great for a while until sometime
back it just stopped indexing. It would get a fatal error and
stop index after couple urls. It was also not so kind about
sending me an email to let me know that it has stopped indexing.
It was still serving pages which were indexed prior to this so we
didn?t realize it wasn?t indexing new content.
And than one day we logged in to the admin console to see whats
going on with it and found out it wasn?t indexing. I checked the
logs and found out that it was stopping due too too many 4xx
errors. After looking at logs on our web server I found out that
it was stopping because Apache was giving back 406 response code
which is described at w3.org as:
The resource identified by the request is only capable of
generating response entities which have content characteristics
not acceptable according to the accept …
[Read more]