Like it or love it, Google remains the dominant force in SERPS (Search Engine Result Pages). Ignoring it could result in lowered rankings and lower visits to your sites.
It is well worth investigating Google’s webmaster tools pages for a great resource in helping to optimise your sites – or SEO (Search Engine Optimisation) as it is known.
One of the features available is a robots.txt analysis tool.
Without turning this into a tutorial, robots.txt is a very important file. It can be configured to allow or disallow certain directories to be searched by various search engine spiders. It is not Google specific and is honoured by all the major search engine spiders. Very useful if you have, for example, an images directory that you do not want crawled and indexed by the search engines.
Putting the term ‘robots.txt’ (without the quotes) into your favourite search engine will produce some very helpful sites and some detailed explanations of what can and can’t be put into a robots.txt file.
So back to Googles robots.txt analysis tool. It used to be very basic and, if it encountered an error, only showed the first one in that line. It would not let you know of other errors in the line until the first one had been fixed.
Now those limitation have been lifted. A ‘new and improved’ robots.txt analysis tool is available.
Firstly, it now shows all the errors in a particular line of the file. This not only lets me know how bad I am at writing a robots.txt file, but it allows for quicker corrections to be made.
You can also test your sitemap.xml URL and enter relative URLs for quick and easy testing.
Do note that the changes you make in the robots.txt analysis tool are not actually made on your site’s robots.txt file. You will need to copy the changes to that file by hand.
However, it does make for very quick and easy testing of your proposed robots.txt file and with the addition of thorough error reporting, relative URLs and sitemap checking, this analysis tool could be very useful for the busy webmaster.
For further details (including other features) please visit the official Google Webmasters Blog entry.