Taking Apart the Meta Robots Tag

March 17th, 2011 by John

A lot of site owners avoid the Meta robots tag and their robots.txt files, simply because they don?t understand either of them. As a part of code, both can be shrouded in mystery to the average non-tech business owner. They really aren?t all that much of a mystery, though, and they can be used to assist your site?s search engine optimisation.

First things first: what is the Meta robots tag? The Meta robots tag is a part of your code that communicates with the search engines. Just as it says on the box, it is an HTML Meta tag that communicates with robots. Robots don?t always do what it says, notably malware robots that ignore it to get on with their malicious deeds, but it?s an important part of your site?s communication with the non-human forces on the net. The robots.txt file, on the other hand, is a file in your root directory. It does pretty much the same thing, and is effective on all robots.

Issue 1: Meta robots tag vs. robots.txt

Both of these communicate with robots, and are practically the same. The differences are significant, though. Some webmasters recommend the Meta robots tag because you don?t have to access the root directory to alter it. This makes it helpful if you don?t have access. On the other hand, many recommend using robots.txt because it is more reliable, and prevents crawling by search engines as well as preventing indexing. If you?re using one, you don?t need to use the other.

Issue 2: Why would you need to use either the Meta robots tag or robots.txt?

A lot of sites choose to leave these alone, but both the Meta robots tag and robots.txt can be used to sculpt the search engines? communication with your site. For example, if you have pages you don?t want Google to bother about, then a simple instruction not to index will mean the page is left alone. If you do this in your robots.txt file, the search engine spiders will leave it alone permanently, whereas if this command is in your Meta robots tag they will come back to check whether you?ve changed your mind.

Other things that can be handy in terms of search engine optimisation include the use of the nofollow tag, the instruction not to archive, which prevents a cached version appearing in the SERPs, and the instruction not to take snippets. When you have an instruction specific for Google in your Meta robots tag, you can address it to Googlebot.

Issue 3: Seriously? Using ?nofollow? in my own pages??

Using nofollows can help you to sculpt the search engines? experience of your site, for example when you want more attention paid to a particular internal link. The Meta robots tag and your robots.txt file can help your site, and they can hinder it. If you?re at all uncertain about what effect the changes you?re making to your code are going to have, it?s best to consult your SEO company.

Link to us

If you want to link to this blog, copy and paste the following HTML code to your website.

Leave a Reply

Source: http://www.searchengineoptimization.co.uk/seo-blog/on-page-seo/taking-apart-the-meta-robots-tag.html

Keeley Hazell Keith Urban Kelly Brook Kenny Chesney