Meta Robots Tag
The Meta Robots tag is added directly in the HTML <head> section of a webpage.
It gives instructions like index, noindex, follow, or nofollow — telling search engines whether to index the page or follow its links.
Example of Meta Robots tag
<meta name =”robots” content=”index, follow”>
4 different variation of Meta Robots tag
<meta name="robots" content="index, follow">
✔ Search engines can index the page and follow all links.
(This is the default behavior even without the tag.)
<meta name="robots" content="index, nofollow">
✔ Search engines can index the page
✖ but will not follow the links or pass link equity.
<meta name="robots" content="noindex, follow">
✖ Search engines will not index the page
✔ but will still follow the links and pass link equity.
<meta name="robots" content="noindex, nofollow">
✖ Search engines will not index the page
✖ and will not follow any links on it.
X-Robots Tag
The X-Robots tag is applied at the HTTP header level rather than inside the HTML code.
It supports the same directives as Meta Robots but offers broader control, making it ideal for managing non-HTML files such as PDFs, images, videos, and other downloadable assets.
Example of X-Robots Tag
X-Robots-Tag: noindex, nofollow
