Search Engine Handling of HTTP Status Codes
November 17, 2008
HTTP status codes are sent from a server when a request is made for a page. If the wrong code is sent when a search engine's crawler requests a page, it can cause problems with search engines. The status code of each request is included in log files and they can be viewed using a tool such as Web-Sniffer.
200 OK
This code tells the crawler that that page exists. An improperly configured server could return this code even when a page does not exist, which could cause search engines to be unable to determine what pages actually exist.
301 Moved Permanently
This code tells the crawler that the page that was at this address has moved permanently to a new address. When a page is moved, a 301 redirect should be setup for the old address so that any pagerank that page received is transferred to the new page. If a website is moved to a new domain name, the old address should be 301 redirected to the new domain. If multiple domain names direct a user to the same site, all but the main domain name should be 301 redirected.
302 Temporarily Moved
This code tells the crawler that the page at this address has temporarily moved. If a 302 redirect is used when a page is moved any pagerank that the page has received will not be transferred to the new page. A 302 redirect can be useful for displaying a simpler URL is search results. For example, a websites home page address might be www.example.com/en/html/home/, but with a 302 redirect it would instead be www.example.com.
304 Not Modified
This code tells the crawler that the page has not changed since it was last requested.
404 Not Found
This code tells the crawler that the page cannot be found, but may available in the future. This is the standard code returned if a file has been deleted. Because the search engine crawler cannot be sure if the page has been deleted or if it is temporarily missing it may request the file again in the future.
410 Gone
This code tells the crawler that a page is gone forever and search engine crawler should not request the page again. Most search engines treat this code as is if were a 404.
500 Internal Server Error
This code indicates that something went wrong on server while processing the request for the page.