Skip to content
GetAllAPIs
APIsPricingNetworkDocsPlaygroundFree tools
Dashboard Get started
APIsPricingNetworkDocsPlaygroundFree toolsDashboardGuidesFAQ Get started

Crawlability

  • Homepage returns an error
  • Homepage redirects to another host
  • Homepage redirect chain
  • robots.txt unreachable
  • No robots.txt
  • robots.txt blocks Googlebot
  • Linked or listed pages return errors
  • Broken internal links
  • Internal links point at redirects
Home / Checks / Crawlability

No robots.txt

low Crawlability robots_missing

What we check

/robots.txt returns 404. Crawling is allowed, so this is informational.

Why it matters

Nothing breaks, but robots.txt is the standard place to declare your sitemap and keep crawlers out of search results pages, carts and admin paths.

How to fix it

Publish a short file: User-agent: *, any Disallow lines you need, and a Sitemap: line.

How to confirm the fix

Test the rules with the robots.txt tester. Glossary: robots.txt.

In the API response

When this check fails, output.technical_issues contains an item with "id": "robots_missing", the affected URLs as evidence and the fix. If it is among the highest-priority problems it also appears in actions (as finding_id) and in plan.

Related checks

  • Sitemap not declared in robots.txt
  • robots.txt unreachable

Run the full audit on your site. One call checks every item in the check reference and ranks the fixes.

Open the playground Endpoint details

Product

  • API catalog
  • Pricing
  • Site audit
  • Playground
  • Dashboard
  • Network

Free tools

  • All free tools
  • Title & description checker
  • robots.txt tester
  • Sitemap validator
  • hreflang checker
  • Heading outline
  • Structured data checker
  • Social preview
  • Indexability checker

Learn

  • Guides
  • Check reference
  • Use cases
  • Compare
  • Glossary

Developers

  • Docs
  • Quickstart
  • MCP server
  • Agent guide
  • llms.txt
  • OpenAPI

Company

  • About
  • FAQ
  • Changelog
  • Data sources
  • Contact
  • Security

Legal

  • Terms of Service
  • Privacy Policy
  • Refund Policy
  • Acceptable Use
  • Data Processing
GetAllAPIs © 2026 Mohamed Outerbah · [email protected]