Meta elements are HTML or XHTML elements that are used to provide metadata about a web page. Any Meta element has four attributes namely: content, http-equiv, name and scheme. So, now what is a Meta robot? A Meta robot is a tag that is used to keep certain content on web pages out of search engine indexes. Meta robot acts as a standard protocol for robot exclusion. An owner of a website may have some pages on their website that are not fully ready or that they don't want to share it with others for some reasons. This is when robot Meta tag is used.
The Meta robot format always consist of one or more keywords: NOINDEX, NOFOLLOW, ALL, INDEX, and FOLLOW.
The Meta robot element defines if a search engine would be allowed to index a page or not. There are two factors that work here:
- NOINDEX: The noindex value makes sure that a page is not indexed by a search engine. It tells search engines not to index specific pages.
- NOFOLLOW: The nofollow makes sure that the links are not crawled. It tells a search engine like yahoo or Google not to follow a link listed on specific pages.
- INDEX: This means that robots can include the given page during search service.
- FOLLOW: This allows robot to follow the links on given page.
Certain additional attributes of Meta robot like NOARCHIVE and NOSNIPPET are also used to tell search engines what not to do with Web pages content. The NOARCHIVE is used to make sure that search engines don't store a cached copy of pages. Similarly NOSNIPPET is used to let the search engines know not to show a snippet or description under the search engine listing. The Users also use Robots.txt file to prevent search engines from indexing the contents of their website.
The various advantages of using Meta Robot tag are:
- It prevents search engines from indexing the web pages.
- It prevents the search engines and other websites from indexing the login and admin pages of your website.
- It prevents search engines like Yahoo and Google from indexing your feed.
- It also helps in disabling the date based archives.
- You can add the Meta robot tags on individual pages and links also.
- It provides options like NOINDEX, NOFOLLOW, ALL, INDEX, and FOLLOW.
So a Meta robot is a tag that provides you with an option of keeping the search engines out of your web content and links.