Don't have an account? Sign up
If you're already signed up, please sign in
robots.txt — a file at a site's root that gives supported web crawlers rules about which URL paths they may request; it does not itself prevent indexing.
Crawling and indexing are different processes. A crawler first retrieves an allowed URL, and the search engine then decides whether to process and index it. robots.txt does not guarantee removal from results, while a noindex directive must remain crawlable to be seen.
robots.txt — a file at a site's root that gives supported web crawlers rules about which URL paths they may request; it does not itself prevent indexing.
The mechanism controls URL discovery, crawler access, preferred-page selection, or eligibility for indexing. Conflicting directives can produce an unexpected outcome.
Verify the HTTP response, robots.txt, meta robots or X-Robots-Tag, canonical, sitemap, and final URL after redirects. Use URL inspection and indexing reports.
