Skip to content
GetAllAPIs
APIsPricingNetworkDocsPlaygroundFree tools
Dashboard Get started
APIsPricingNetworkDocsPlaygroundFree toolsDashboardGuidesFAQ Get started

Crawlability

  • Homepage returns an error
  • Homepage redirects to another host
  • Homepage redirect chain
  • robots.txt unreachable
  • No robots.txt
  • robots.txt blocks Googlebot
  • Linked or listed pages return errors
  • Broken internal links
  • Internal links point at redirects
Home / Checks / Crawlability

robots.txt unreachable

high Crawlability robots_unreachable

What we check

/robots.txt returned a 5xx status or timed out. Following RFC 9309, we then treat the whole site as disallowed, as major crawlers do.

Why it matters

A server error on robots.txt can make search engines pause crawling the entire site until it recovers.

How to fix it

Make /robots.txt return 200 with your rules, or 404 if you have none. Check that a firewall or bot filter is not blocking it.

How to confirm the fix

curl -sI https://yoursite.com/robots.txt returns 200 or 404, quickly. Glossary: 5xx.

In the API response

When this check fails, output.technical_issues contains an item with "id": "robots_unreachable", the affected URLs as evidence and the fix. If it is among the highest-priority problems it also appears in actions (as finding_id) and in plan.

Related checks

  • No robots.txt
  • robots.txt blocks Googlebot

Run the full audit on your site. One call checks every item in the check reference and ranks the fixes.

Open the playground Endpoint details

Product

  • API catalog
  • Pricing
  • Site audit
  • Playground
  • Dashboard
  • Network

Free tools

  • All free tools
  • Title & description checker
  • robots.txt tester
  • Sitemap validator
  • hreflang checker
  • Heading outline
  • Structured data checker
  • Social preview
  • Indexability checker

Learn

  • Guides
  • Check reference
  • Use cases
  • Compare
  • Glossary

Developers

  • Docs
  • Quickstart
  • MCP server
  • Agent guide
  • llms.txt
  • OpenAPI

Company

  • About
  • FAQ
  • Changelog
  • Data sources
  • Contact
  • Security

Legal

  • Terms of Service
  • Privacy Policy
  • Refund Policy
  • Acceptable Use
  • Data Processing
GetAllAPIs © 2026 Mohamed Outerbah · [email protected]