2 docs tagged with "crawling"

Learn the HTML basics that help search engines find and understand a page: the head essentials, real headings and text, crawlable links, images, clean URLs, mobile and speed basics, and how JavaScript can hide content. Includes a crawler's-view lab, a URL checker, and a live HTML SEO basics checker with a messy and a fixed page.

Learn how the robots meta tag asks search engines to index, skip, follow, or limit snippets for a page, and how it differs from robots.txt, X-Robots-Tag, rel=nofollow, canonical links, and real privacy. Includes a robots meta builder, a robots.txt tester that shows which rule wins, and a live robots meta checker that catches typos and conflicts.