iSocialWeb

Meta robots

What it is, what it is for and how it works

What Meta Robots are and what they are used for

The meta robots tag is an HTML tag placed in the head section of a page that tells search engine crawlers whether they may index that page and follow its links, using values such as noindex, nofollow or nosnippet. Unlike robots.txt, which controls crawling, it controls how a single page is indexed and shown in results.

Where and how is the meta robots tag implemented?

The meta robots tag must be placed in the <head> section of the web page in question and must include two attributes, name and content, to work properly.

Attribute “name”

The name attribute indicates which crawlers it applies to. With the value robots, the instruction targets all of them; with the name of a specific crawler (googlebot, googlebot-news, bingbot…), only that one.

Attribute “content”

The content attribute will indicate how it should behave based on the parameters used.

Meta Tag Example

Here is an example of a meta robots tag:

<head>
  <meta name="robots" content="noindex, follow" />
</head>

With name="robots" the instruction targets all crawlers, and content="noindex, follow" asks them not to index the page but to follow its links.

If you only want to give the instruction to Google, use the name of its crawler:

<meta name="googlebot" content="noindex" />

In this case only Googlebot will stop indexing the page; other search engines are not affected. If a page has both a robots tag and a googlebot tag, Google combines the directives of both and, when they conflict, applies the most restrictive one.

Indexing control parameters

These are the directives Google supports in the meta robots tag and in the X-Robots-Tag header. You can combine several of them, separated by commas:

  • all: no restrictions on indexing or serving. It is equivalent to index, follow, which is the default behaviour, so there is no need to declare it.
  • noindex: do not show the page in search results. More detail in the noindex tag.
  • nofollow: do not follow the links on the page, so they pass no page authority.
  • none: equivalent to noindex, nofollow.
  • nosnippet: do not show a text snippet or video preview for the page in search results.
  • indexifembedded: lets Google index the content of a page when it is embedded in another page through an iframe, even if the page has noindex. It only works together with noindex.
  • max-snippet:[number]: maximum number of characters of the text snippet. 0 means no snippet and -1 means no limit.
  • max-image-preview:[none|standard|large]: maximum size of the image preview that can be shown for the page.
  • max-video-preview:[number]: maximum length, in seconds, of the video preview. 0 allows only a static image and -1 means no limit.
  • notranslate: do not offer a translation of the page in search results.
  • noimageindex: do not index the images on the page.
  • unavailable_after:[date]: stop showing the page in search results after the given date (in a standard format such as ISO 8601).
  • noarchive: it asked search engines not to display a cached copy of the page. Google retired its cached-page links in 2024, so this directive no longer has any effect in Google.

Guidelines for different search engines

However, not all search engines support all directives. Google and Bing document the ones they recognise:

  • Google: all of the directives listed above, according to its robots meta tag specifications. noarchive no longer has any effect.
  • Bing: noindex, nofollow, noarchive (or its equivalent nocache), nosnippet and the max-snippet, max-image-preview and max-video-preview family.

If you work with other search engines or crawlers, check their documentation: a directive that a search engine does not recognise is simply ignored.

Example of use and interpretation of directives

The most common parameters that we are going to find and useful for SEO are the index tag, noindex tag, follow tag and the nofollow tag. These are their usual combinations:

<meta name="robots" content="index, follow" />

This meta tag tells Googlebot and other crawlers to index the content so it can appear in their results, and to follow all the links they find on this URL.

By default, when a URL has no meta robots tag, this is the instruction all bots assume when crawling it. So you do not need to add it for your content to be indexed.

<meta name="robots" content="noindex, follow" />

Here you tell the search engine not to index the URL, so it does not appear in its results, but to follow the links it finds while crawling the page.

It is usually used on pages with little value for users in search results that still contain links we want the search engine to follow to reach other sections of the website.

<meta name="robots" content="noindex, nofollow" />

The “noindex, nofollow” meta tag tells search engines not to index the page and not to follow the links on it either. It is equivalent to “none”.

When should you use this directive?

Generally when the page content adds no value in search results and you do not want the search engine to follow the links it contains either.

<meta name="robots" content="index, nofollow" />

Here you ask search engines to include the page in their index but not to follow the links on it. It is an unusual combination: if you only want specific links not to be followed, mark them one by one as a nofollow link.

Types of Robots Tags

  1. Meta robots tag: part of the HTML of the page, located in the <head> section.
  2. X-Robots-Tag header: sent by the web server in the HTTP response of the URL, along with the other headers.

This is what the X-Robots-Tag header looks like in an HTTP response:

HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex

The same directives work in both: noindex, nofollow and the rest can be used either in the meta robots tag or in the X-Robots-Tag header. What differs is how they are passed to crawlers.

The X-Robots-Tag header lets you apply directives to non-HTML files, such as PDFs, images or videos, and to many URLs at once. The header itself does not support regular expressions: the patterns (for example, “all PDFs”) are defined in the server configuration, such as Apache or Nginx.

For example, in Apache this rule adds noindex to every PDF on the site:

<Files ~ "\.pdf$">
  Header set X-Robots-Tag "noindex"
</Files>

Remark: It is not necessary to use meta robots and the X-Robots-Tag header on the same page, as it is redundant. If you do, Google combines the directives of both and applies the most restrictive one.

Why they are important

Together with the robots.txt file, which controls crawling, and canonical tags, these directives let you control the indexability of your website, that is, which pages appear in search results and how they are shown:

  • You decide, page by page, which URLs should appear in search engines and which should not, regardless of the folder structure.
  • You keep low-value pages or pages with duplicate content (internal search results, filters, thank-you pages…) from competing in the results with your important pages.
  • You control how your result is displayed: snippet length, the size of image and video previews, or whether a translation is offered.

Differences between Meta robots vs. Robots.txt

The main difference between them is that meta robots give indications about the indexing of pages, while robots.txt gives indications about crawling.

In other words, robots.txt gives crawling instructions for entire sections of a domain, such as categories, subfolders and files, but it does not remove pages that are already indexed.

For example:

If robots.txt blocks crawling of a category but the URLs in that category have the index tag because we want them indexed, Googlebot will not be able to access them: it will not read their content or their meta tags, so it cannot index them normally. At most, if they receive links, it will show the URL without a description.

Also bear in mind that noindex does not save crawl budget: Google has to crawl the page to read the directive. That said, over time it tends to crawl pages that have been noindexed for a long time less often.

SEO recommendations for Meta Robots application

All meta directives (robots or otherwise) are discovered when a URL is crawled. If robots.txt does not allow a URL to be crawled, any meta directive on the page (in the HTML or in the HTTP header) will not be seen and will, in effect, be ignored. With that in mind, these are our recommendations:

  • Do not block noindexed URLs in robots.txt. If Google cannot crawl them, it will never read the directive and the URL can stay indexed.
  • To remove a page from the index, use noindex and let it be crawled, or return a 404 or 410 status code if the page no longer exists. A Disallow in robots.txt does not deindex it.
  • Do not include noindexed URLs in the XML sitemap. The sitemap should only list the URLs you want indexed.
  • Avoid contradictory directives, such as index in the meta robots tag and noindex in the HTTP header, or two different meta robots tags on the same page. Google applies the most restrictive one.
  • Use the X-Robots-Tag header for PDFs, images and other non-HTML files, where you cannot add a meta tag.
  • Check the result with the URL Inspection tool in Google Search Console: it tells you whether Google has detected the noindex and whether the URL can be indexed.

In conclusion:

Meta robots tags are a great tool to control which pages of your website get indexed and how they are shown in search results.

Using them correctly together with robots.txt, canonicals, site architecture and internal linking makes search engines' work easier and helps only the pages you care about compete in the results.

Remember that meta robots work at URL level.

Important: Google and Bing treat noindex as a directive, not a suggestion: if they can crawl and read the page, they will not show it in their results. Other bots, however, may ignore it, so it is not a security measure. To protect private content, use a password or restrict access on the server.

Frequently Asked Questions

  • It depends on what you want to achieve. These are the 3 main ways to block Googlebot:

    • Robots.txt: stops sections or files from being crawled, but does not guarantee that those URLs drop out of the index.
    • Meta robots or X-Robots-Tag header with noindex: keeps a URL out of the search results. For it to work, the page must be crawlable.
    • Password protection: the only real way to stop anyone from accessing private content.
  • Meta robots tags should be used whenever you want to control indexing at the individual page level. In other words, to ensure that a URL is not indexed, always use the robots meta tag or the X-Robots-Tag HTTP header.