All you need to know about Robots.txt files.

Although the robots.txt file is very important if you want to have good rankings within the search engines, many web sites don’t have this file. If your website doesn’t have a robots.txt file yet you should considered getting one. However if you already have a robots.txt file, it will do no harm reading this blog to find out more tips and make sure that the file doesn’t contain any errors.

What is robots.txt?

When a search engine crawls through your site, it looks for a special file on the site. That file is called robots.txt and will tell the search engines spider which web pages of your site should be indexed and which web pages should be ignored. Therefore the robots.txt file is a simple text file, which must be placed in your root directory. For example:
http://www.yoursite.com/robots.txt

How do I create a robots.txt file?

As mentioned before, the robots.txt file is a simple text file. To this all you need to do is open a simple text editor to create the file. The robots.txt file consists of so called ‘records’. These records consists if two fields the user agent line and the disallow lines. The robots.txt file will allow the search engines spider of Google to retrieve every page from your site. However it will ignore any files named ‘cgi-bin’. Therefore if you leave the disallow blank, you are telling the search engine that all files may be indexed. In any case, you must enter a disallow line for every user agent record.
In conclusion to this, there are many different tips and tricks available to help you gain the best use out of your robots.txt file. If you would like any additional information see Robots.txt tips and tricks.