robots.txt, noindex or canonical: which one to use
Updated 2026-11-17
All three affect what ends up in search results, but they act at different stages: robots.txt controls fetching, noindex controls indexing and canonical controls which duplicate is shown.
| robots.txt Disallow | noindex | rel=canonical | |
|---|---|---|---|
| Stops crawling | Yes | No | No |
| Removes from results | Not reliably; a blocked URL can still be listed from links | Yes, once recrawled | Usually, in favour of the canonical URL |
| Strength | Directive | Directive | Hint |
| Typical use | Internal search, carts, infinite filters, admin | Thank-you pages, thin tag pages, staging | Tracking parameters, sorted views, syndicated copies |
Combinations that backfire
- Disallow + noindex on the same URL. The crawler may not fetch the page, so it never sees the noindex. Allow crawling until the page has dropped out, then block it if you must.
- noindex + canonical to another page. Mixed signals: one says "drop this", the other says "this is a copy of that". Pick one.
- Canonical to a redirect or an error. The hint points nowhere useful and is often ignored.
- Disallow: / left over from staging. The audit flags this as robots.txt blocks Googlebot.
Checking a page
The free indexability checker reads a page's head and headers and reports its robots directives and canonical, including conflicts. The robots.txt tester answers whether a path may be fetched and which rule decided.
Run the full audit on your site. One call checks every item in the check reference and ranks the fixes.
Open the playground Endpoint details