Google Crawls Its Own SERPs
Google August 21st, 2006 - By HaochiSERP = Search Engine Result Page
Google’s robots.txt disallow all bots from crawling “Google.com/search….”, but let’s see what Googlebot did. It indexed about 40,000 Google’s SERPs. These pages range from e^(pi*i)+1 to “kentucky friend chicken”. The Googlebot is also crawling some 3000 Google Picasa Web Albums’ URLs, which is also disallowed by the robots.txt. (Not sure what “Allow: /$” means to Googlebot though.)
August 21st, 2006 at 3:16 pm
Google do NOT crawl Picasa Web Albums, but they do include web album URLs that they find on other pages.
The net effect of this is that the URLs are known about, but none of the content of the page itself is - the only time these pages turn up in SERPs is if the word being searched for is part of the URL, or was in the text that was used to link to the URL.
I talk a lot more about this at
http://www.zmarties.com/picasa/blog/2006/08/how-did-search-get-lost.html