In this part of the ‘How to SEO’ manual, Position Gold Ltd will be looking at the Robots.txt file and explaining in full, just what it does., how it can benefit your website and just what can be achieved by using it.

What is it and what does it do?

Robots.txt files are text based files which installed to the root of a websites coding through the File Transfer Protocol (FTP). Normally created through notepad or other similar text creating programmes and is saved as ‘robots.txt’. The robots.txt file is basically a file that is read by the search engine spiders. It tells the spiders just exactly what to crawl and index, but more importantly tells the spider what NOT to crawl and index. The spider then follows the robots.txt files instructions and crawls the site accordingly.

Why should a website have a robots.txt file?

This is very simple, the robots.txt file allows you to instruct the website what it can and cannot see and index. There are many different reasons for why you may want to restrict what the the search engine spiders can see; as follows. A website may have content that is repeated throughout the site, by disallowing the spiders from crawling this, you will be able to prevent any aspects of duplicate content issues. Also images are usually put into the robots.txt files, this is because there is no point in spidering images (unless images have been specifically targeted in SEO). When a spider is crawling a site, crawling images just takes too much time and so it is much more efficient to have the spider crawl all text rather than some text and images. Administrative sections of websites are also commonly disallowed from spiders as the information found there is not beneficial to the the user in any way.

To conclude…

The robots.txt file is basically a checklist that the search engine spiders must adhere to. It is a way of controlling and manipulating what the search engine spider sees, indexes and then ranks your website on. You should never disallow everything from being seen as your website will simply not rank for any of your targeted keywords.