Skip to content
GetAllAPIs
APIsPricingNetworkDocsPlaygroundFree tools
Dashboard Get started
APIsPricingNetworkDocsPlaygroundFree toolsDashboardGuidesFAQ Get started

Crawlability

  • Homepage returns an error
  • Homepage redirects to another host
  • Homepage redirect chain
  • robots.txt unreachable
  • No robots.txt
  • robots.txt blocks Googlebot
  • Linked or listed pages return errors
  • Broken internal links
  • Internal links point at redirects
Home / Checks / Crawlability

robots.txt blocks Googlebot

high Crawlability robots_blocks_google

What we check

We evaluate your robots.txt for the Googlebot user agent against the homepage path.

Why it matters

If Googlebot may not fetch the homepage, the site cannot be crawled normally and pages fall out of results over time. This is a common leftover from a staging site.

How to fix it

Remove the Disallow: / that applies to Googlebot or *. Keep blocks narrow (specific folders), never site-wide on production.

How to confirm the fix

Paste the file into the robots.txt tester with the Googlebot agent and path /. Glossary: Crawlability.

In the API response

When this check fails, output.technical_issues contains an item with "id": "robots_blocks_google", the affected URLs as evidence and the fix. If it is among the highest-priority problems it also appears in actions (as finding_id) and in plan.

Related checks

  • robots.txt unreachable
  • Pages marked noindex

Run the full audit on your site. One call checks every item in the check reference and ranks the fixes.

Open the playground Endpoint details

Product

  • API catalog
  • Pricing
  • Site audit
  • Playground
  • Dashboard
  • Network

Free tools

  • All free tools
  • Title & description checker
  • robots.txt tester
  • Sitemap validator
  • hreflang checker
  • Heading outline
  • Structured data checker
  • Social preview
  • Indexability checker

Learn

  • Guides
  • Check reference
  • Use cases
  • Compare
  • Glossary

Developers

  • Docs
  • Quickstart
  • MCP server
  • Agent guide
  • llms.txt
  • OpenAPI

Company

  • About
  • FAQ
  • Changelog
  • Data sources
  • Contact
  • Security

Legal

  • Terms of Service
  • Privacy Policy
  • Refund Policy
  • Acceptable Use
  • Data Processing
GetAllAPIs © 2026 Mohamed Outerbah · [email protected]