Robot tags
The Robot META tags are vital within off page SEO as it determines what information a search engine spider will be able to crawl and index. This is an advantage as it means that, a websites’ administrator can control what is visible to a search engine.
This is useful as it gives the website administrator more power over the search engines. The Robots META tag is used to restrict what pages within a website get crawled. This is useful if there is duplicate content on a website, as with the robots META tag, it will be ignored. It is also useful if your website contains any administrative or any client information within a website that is confidential. A website may also wish not to disclose some information for marketing reasons, as not to give too much away about a product or service. Another reason for not allowing a spider to crawl parts of a site is that a web page within a site may not be ready to be released yet. I.e. the page has been written before an event, E.g. the Glastonbury website is created before the festival.
An example of Robots META tags
<meta name=”robots” content=”all” />
- all
- This is the default setting and enables all pages to be crawled and indexed
- none
- This stops any pages from being crawled or indexed
- index
- Enables the spider to index the page
- noindex
- This instructs the spider not to index the page
- follow
- This instructs the spider to follow links within the page
- nofollow
- This prevents the spider following links within the page
The Robots META tag is, in essence, a filtering tool that gives a webmaster total control of what is viewable within a web pages’ coding.
Related posts:
