googlebot unavailable after

Google today officially introduced the new unavailable_after META tag value which allows site owners to include a appropriate unavailable_after META tag value in a web page that tells the Google when to drop that specific web page from the search results.

This is how the META tag should look like:

<meta name="GOOGLEBOT" content="unavailable_after: 31-Dec-2007 23:59:59 EST">

The date and time after unavailable_after: should follow the RFC 850 standard.

Google note that using this feature will only drop the page from the search result, but not from their index. If you want to remove the page completely from Google index, you should use the URL Removals tool in the Webmaster Central to request for a page to be removed.

robot tag http header google

Also, Google now allows site owners to include META tag information in the HTTP Header so that you can apply the X-Robots-Tags to non-HTML files.