robots.txt unreachable
high Crawlability robots_unreachable
What we check
/robots.txt returned a 5xx status or timed out. Following RFC 9309, we then treat the whole site as disallowed, as major crawlers do.
Why it matters
A server error on robots.txt can make search engines pause crawling the entire site until it recovers.
How to fix it
Make /robots.txt return 200 with your rules, or 404 if you have none. Check that a firewall or bot filter is not blocking it.
How to confirm the fix
curl -sI https://yoursite.com/robots.txt returns 200 or 404, quickly. Glossary: 5xx.
In the API response
When this check fails, output.technical_issues contains an item with "id": "robots_unreachable", the affected URLs as evidence and the fix. If it is among the highest-priority problems it also appears in actions (as finding_id) and in plan.
Related checks
Run the full audit on your site. One call checks every item in the check reference and ranks the fixes.
Open the playground Endpoint details