Skip to content
GetAllAPIs
APIsPricingNetworkDocsPlaygroundFree tools
Dashboard Get started
APIsPricingNetworkDocsPlaygroundFree toolsDashboardGuidesFAQ Get started

Guides

  • Audit a website from an AI agent over MCP
  • How to read an audit report
  • Find competitor topic gaps from public sitemaps
  • robots.txt, noindex or canonical: which one to use
  • SEO checks before and after a site migration
  • Fix broken internal links and redirect chains
Home / Guides / robots.txt, noindex or canonical: which one to use

robots.txt, noindex or canonical: which one to use

Updated 2026-11-17

All three affect what ends up in search results, but they act at different stages: robots.txt controls fetching, noindex controls indexing and canonical controls which duplicate is shown.

robots.txt Disallownoindexrel=canonical
Stops crawlingYesNoNo
Removes from resultsNot reliably; a blocked URL can still be listed from linksYes, once recrawledUsually, in favour of the canonical URL
StrengthDirectiveDirectiveHint
Typical useInternal search, carts, infinite filters, adminThank-you pages, thin tag pages, stagingTracking parameters, sorted views, syndicated copies

Combinations that backfire

  • Disallow + noindex on the same URL. The crawler may not fetch the page, so it never sees the noindex. Allow crawling until the page has dropped out, then block it if you must.
  • noindex + canonical to another page. Mixed signals: one says "drop this", the other says "this is a copy of that". Pick one.
  • Canonical to a redirect or an error. The hint points nowhere useful and is often ignored.
  • Disallow: / left over from staging. The audit flags this as robots.txt blocks Googlebot.

Checking a page

The free indexability checker reads a page's head and headers and reports its robots directives and canonical, including conflicts. The robots.txt tester answers whether a path may be fetched and which rule decided.

Run the full audit on your site. One call checks every item in the check reference and ranks the fixes.

Open the playground Endpoint details

Product

  • API catalog
  • Pricing
  • Site audit
  • Playground
  • Dashboard
  • Network

Free tools

  • All free tools
  • Title & description checker
  • robots.txt tester
  • Sitemap validator
  • hreflang checker
  • Heading outline
  • Structured data checker
  • Social preview
  • Indexability checker

Learn

  • Guides
  • Check reference
  • Use cases
  • Compare
  • Glossary

Developers

  • Docs
  • Quickstart
  • MCP server
  • Agent guide
  • llms.txt
  • OpenAPI

Company

  • About
  • FAQ
  • Changelog
  • Data sources
  • Contact
  • Security

Legal

  • Terms of Service
  • Privacy Policy
  • Refund Policy
  • Acceptable Use
  • Data Processing
GetAllAPIs © 2026 Mohamed Outerbah · [email protected]