Skip to main content

Robots Meta

Most of your pages should be found. A few should not: a "thank you" page after a purchase, the results of your internal search box, a draft you are still writing, or a login screen. For those, you want to give search engines instructions.

There are two main ways to do that, and they are easy to mix up. The robots meta tag tells a search engine what to do with a page it has read. The robots.txt file tells it where it may go in the first place. This lesson explains both, and helps you pick the right one.


What You Will Learn​

  • What the robots meta tag does, and its common values
  • The difference between crawling and indexing
  • How robots.txt, X-Robots-Tag, nofollow links, and canonical links differ
  • When to use noindex, and when not to
  • How to limit snippets and previews
  • Why none of this is security
  • How to check your tags and your robots.txt

The Big Idea​

thank-you.html
<head>
  <title>Thank you | The Bread Bin</title>

  <!-- Keep this page out of search results, but let links be followed -->
  <meta name="robots" content="noindex, follow">
</head>

Tick the options, and read the tag, the header, and the plain-English meaning that come out.

Crawling and Indexing Are Different​

This is the most important idea in the lesson.

  • Crawling is when a search engine visits a page and downloads it.
  • Indexing is when it stores the page, so it can show it in results.
ToolControlsWhat it really says
robots.txtCrawling"Please do not visit these paths."
Robots meta tagIndexing (and more)"You may visit, but please do not list this page."
X-Robots-Tag headerIndexing (and more)The same, for files with no head, such as PDFs
rel="nofollow" on a linkOne link"Please do not treat this link as an endorsement."
Canonical linkWhich duplicate to prefer"This other address is the main one." See Canonical Links.
The classic trap

To keep a page out of results, you use noindex. But a crawler must read the page to see that tag. If you also block the page in robots.txt, the crawler never visits, so it never sees the noindex. The address can then still appear in results, usually without a description, if other pages link to it. So: use noindex, and leave the page crawlable.

The Robots Meta Tag​

<meta name="robots" content="noindex, nofollow">
  • name="robots" speaks to all search engines. Use a crawler's own name (googlebot, bingbot) to speak to just one.
  • content holds one or more instructions, separated by commas.
  • It goes in the head. In the body, it is ignored.
  • Without any tag, the default is index, follow. You never need to write that.

Common values​

ValueMeaning
indexThe page may be listed (the default)
noindexDo not list this page in results
followLinks may be followed (the default)
nofollowDo not follow the links on this page
noneSame as noindex, nofollow
noarchiveDo not show a cached copy
nosnippetDo not show a text snippet
max-snippet:NLimit the snippet to N characters
max-image-preview:largeAllow large image previews (or standard or none)
notranslateDo not offer to translate the page
unavailable_after: dateStop showing the page after a date

Support varies by search engine, and it changes. These are documented by Google, and many are respected by others. Check each search engine's own page.

Pages That Often Get noindex​

  • Thank-you and confirmation pages.
  • Internal search results, which are thin and endless.
  • Login, account, and checkout steps.
  • Drafts and staging pages (but protect them with a password too).
  • Thin, near-empty pages, such as tag pages with nothing on them.
  • Duplicate printer-friendly pages, when a canonical will not do.

Pages That Should Not​

  • Anything you want people to find. A stray noindex on a home page can remove it from results.
  • A page that also has a canonical pointing elsewhere. Choose one purpose. If the page is a duplicate, use a canonical. If it should simply not be listed, use noindex.
  • Pages you want to hide from people. noindex is a request. Use a login instead.

The launch-day mistake​

Developers often put noindex on a staging site, so it is not listed while it is being built. Then they launch, and forget to take it off. The live site silently disappears from search. Always check your live pages after a launch, and consider adding a test in your build that fails if production pages have noindex.

The robots.txt File​

robots.txt is a plain text file at the top of your site (https://thebreadbin.example/robots.txt). It lists which paths crawlers should skip.

robots.txt
User-agent: *
Disallow: /admin/
Disallow: /search
Allow: /search/help

Sitemap: https://thebreadbin.example/sitemap.xml
  • User-agent names the crawler. * means all of them.
  • Disallow lists a path to skip. Allow makes an exception.
  • The longest matching rule wins, and Allow wins a tie.
  • * matches any characters, and $ matches the end of the address.
  • Sitemap points to your sitemap file.

Try it, and see which rule wins:

http://127.0.0.1:5500/index.html


Allowed to crawl. Using the group for "*".

Winning rule: Allow: /search/help (the longest matching rule wins, and Allow wins a tie).

This is a simplified tester that follows the common rules (user-agent groups, the longest match, * and $). Real crawlers have small differences, so use the testing tools in Search Console and Bing Webmaster Tools for the real answer. A robots.txt file is public, and only well-behaved crawlers follow it.

Good uses of robots.txt​

  • Stopping crawlers from wasting time on endless, low-value paths (internal search, filters, calendars).
  • Keeping crawlers away from large, heavy files that you do not want fetched.
  • Pointing to your sitemap.

What it is not for​

  • Keeping a page out of results. Use noindex (and leave it crawlable).
  • Hiding private content. The file is public, and anyone can read it. A list of "secret" paths in robots.txt is a map for attackers.
  • Blocking your CSS and JavaScript. Search engines need them to understand your page.

X-Robots-Tag: For Files Without a Head​

A PDF or image has no HTML head, so a meta tag cannot go in it. A server can send the same instruction as an HTTP header:

X-Robots-Tag: noindex

The values are the same as in the meta tag. It is set in your server or hosting settings, and is often used for PDFs, images, and whole folders.

The meta tag can apply nofollow to every link on a page. More often, you want to mark single links with rel:

<a href="https://example.com/partner" rel="sponsored">Our partner</a>
<a href="https://example.com/comment-link" rel="ugc nofollow">A link from a comment</a>
<a href="https://example.com/unsure" rel="nofollow">A link we do not vouch for</a>
  • sponsored: paid or affiliate links.
  • ugc: links in user-generated content.
  • nofollow: a link you do not want to vouch for.

Search engines treat these as hints, not commands. Use them to be honest about your links.

Snippets and Previews​

You can limit how your page appears in results. Use these with care, because showing less often means fewer clicks.

  • nosnippet or max-snippet:N limit the text shown. See Meta Description.
  • max-image-preview:large allows large images, which can make a result more eye-catching.
  • noarchive stops cached copies being offered.
  • data-nosnippet is an HTML attribute that asks Google not to use a part of a page in snippets.

AI and Other Crawlers​

More and more crawlers, including those that collect data for AI systems, visit sites. Many publish a crawler name and say they follow robots.txt. You can then allow or block them by name:

robots.txt
User-agent: ExampleAIBot
Disallow: /

Names, rules, and promises change quickly, and not every crawler obeys. Read each company's current documentation. If something must stay private, put it behind a login.

Robots Rules in Docusaurus​

Docusaurus has a site-wide noIndex option in docusaurus.config.js, and lets you add tags to a single page with its Head component. A static robots.txt placed in the static folder is copied to the top of your built site. Behaviour changes between versions, so check the current documentation, and look at the built output.

Check Your Tags​

http://127.0.0.1:5500/index.html

Check it for real​

  1. "View source" on a page, and search for robots.
  2. Open /robots.txt on your site, and read it.
  3. Google Search Console's URL inspection tells you whether a page is indexed, and whether noindex or robots.txt is stopping it.
  4. The robots.txt tester in Search Console (or Bing Webmaster Tools) shows what a real crawler would do.
  5. Check response headers in your browser's developer tools (Network tab) for X-Robots-Tag.
  6. Search for site:yourdomain.com to see what is listed.
Changes take time

After you add or remove noindex, search engines must visit the page again to notice. That can take days or weeks. Search Console has a temporary removal tool for urgent cases, and a way to ask for a recrawl.

Good Habits​

  • Use noindex to keep a page out of results, and leave it crawlable.
  • Use robots.txt for crawl control, not for hiding.
  • Never block CSS or JavaScript that pages need.
  • Use a login for anything private.
  • Check your live site after every launch for stray noindex tags.
  • Keep one robots tag per page, in the head.
  • Be honest with rel values on links.
  • Check with Search Console to see what search engines did.

Robots Checklist​

  • Pages you want found have no noindex
  • Pages you do not want listed have noindex, and are not blocked in robots.txt
  • Private pages are behind a login, not only hidden
  • robots.txt exists, is short, and lists no secret paths
  • CSS, JavaScript, and images needed for rendering are not blocked
  • A sitemap is listed in robots.txt
  • No page has both noindex and a canonical pointing elsewhere
  • Staging has a password, and its noindex is removed on launch
  • Sponsored and user-submitted links have rel values
  • PDFs and images use X-Robots-Tag when needed

Security and Privacy Notes​

  • noindex and robots.txt are requests, not locks. Well-behaved crawlers follow them. Scrapers, attackers, and anyone with the link do not have to. Use authentication for private data.
  • robots.txt is public. Listing /admin/ or /private-backups/ tells everyone where to look. Protect those paths properly, and avoid naming them.
  • Pages can still be shared and archived. Copies, screenshots, and other archives may exist, whatever your tags say. noarchive only affects some search engines' caches.
  • Do not rely on noindex for removing private data. If something sensitive was published, remove it, change what needs changing, and use the search engines' removal tools. Takedown and "right to be forgotten" rules differ by country. This is not legal advice.
  • Never build robots tags from untrusted input. If a page's tag or header is created from a parameter or user data, an attacker could inject noindex or break your HTML. Use fixed, validated values, and escape the output.
  • Watch for tampering. If a hacked site gets noindex added, or robots.txt changed to Disallow: /, it can vanish from results. Monitor your live tags and file.
  • Respect others' rules. If you write a crawler or scraper, follow robots.txt, identify yourself, and limit your request rate.

Global Audience Notes​

  • Search engines differ by region. Baidu, Yandex, Naver, and others each document their own robots support and crawler names. Check the ones your readers use.
  • Apply rules to every language version. A forgotten noindex on the Spanish pages is as damaging as on the English ones.
  • Check country-specific hosts (.fr, /de/). Each needs the right robots.txt, since the file belongs to one host.
  • Legal rules about data, privacy, and removal differ across countries. Ask for advice if you handle personal data.
  • Notranslate lets you stop automatic translation for pages (or parts) where a machine translation could mislead, such as legal text or brand names. Use it sparingly, because many readers rely on translation.

Common Mistakes to Avoid​

  1. Blocking a page in robots.txt and expecting noindex to work.
  2. Leaving noindex on after launch.
  3. Using robots.txt to hide private pages.
  4. Listing secret paths in robots.txt.
  5. Typos such as noindx, or name="robot".
  6. Contradictory values, such as index, noindex.
  7. Putting the tag in the body.
  8. Using http-equiv="robots", which is not supported.
  9. noindex plus a canonical pointing elsewhere.
  10. Blocking CSS and JavaScript.
  11. Using nofollow on a whole page by accident.
  12. Never checking what search engines actually did.

Practice Time​

Try these in your browser and your editor:

  1. Use the builder to make a tag for a thank-you page, an internal search page, and a normal article.
  2. In the robots.txt tester, try each sample path. Why does /search/help work, while /search?q=rye is blocked?
  3. Add a rule that blocks /drafts/, and test it.
  4. Run the checker on the messy page, fix every problem, and run it again.
  5. Open /robots.txt on three sites you use. What do they block?
  6. Search a site you own for noindex. Is it where you expect it?
  7. Use URL inspection in Search Console on a page. Is it indexed?
  8. Find a PDF on your site, and work out how you would add X-Robots-Tag to it.
  9. Mark three kinds of links on a page with the right rel values.
  10. Bonus: write a short test for your build that fails if a production page contains noindex.

Quick Recap​

Click a question to check your answer.

1. What is the difference between crawling and indexing?

Crawling is visiting and downloading a page. Indexing is storing it so it can appear in results. robots.txt mainly controls crawling, and the robots meta tag controls indexing.

2. Which tag keeps a page out of search results?

noindex, in a robots meta tag or an X-Robots-Tag header. The page must stay crawlable, so the tag can be read.

3. Why is blocking a page in robots.txt a poor way to keep it out of results?

A blocked crawler cannot read the page, so it cannot see a noindex tag. The address can still appear in results, without a description, if other pages link to it.

4. Is noindex a way to keep a page private?

No. It is a request, and anyone with the link can still open the page. Use a login for private content.

5. What is X-Robots-Tag for?

It sends robots instructions as an HTTP header, which works for files with no HTML head, such as PDFs and images.

6. What launch-day mistake do many sites make?

They leave the noindex from staging on the live site, so the site vanishes from search. Check live pages after a launch.

Key Takeaways​

  • Crawling and indexing are different. robots.txt guides the first, and the robots tag guides the second.
  • To keep a page out of results, use noindex and keep it crawlable.
  • The default is index, follow. You rarely need to say it.
  • Use X-Robots-Tag for PDFs and images, and rel values for single links.
  • None of these are security. Use a login for anything private.
  • robots.txt is public. Never list secrets in it.
  • Check your live site for stray noindex tags, and use Search Console to see what search engines did.
  • Rules and crawler names change. Read the current documentation.

Next, learn how to describe your content so search engines understand it in detail, with Structured Data.