Data sources
Where every number in a report comes from. Each report also lists its own sources, with the time each file was fetched.
Current data mode on this server: First-party crawl (site). Calls served in the last 30 days: 0, all by this one source. There is no automatic failover to a second source.
| Source | Kind | Used for | Status |
|---|---|---|---|
| The audited site | first-party | Pages (15 sampled by default), headers, links, redirects | Always |
| robots.txt and XML sitemaps | first-party | Crawl rules, URL inventory, sitemap errors | Always |
| Competitors' robots.txt, sitemaps and homepage | public | Topic and section gaps, inventory size, homepage tags | When you pass competitors |
| Google PageSpeed Insights | third-party, free | Lighthouse score, LCP, field data for the homepage | Off on this server |
| Licensed SEO data | third-party, licensed | Keyword volumes, rankings, backlinks | Not configured |
How the crawler behaves
- Identifies itself as
SiteAuditBotin the User-Agent. - Fetches the homepage you submit, then honours robots.txt for everything else (a 404 robots.txt means allowed, a 5xx means disallowed).
- At most a few dozen requests per audit, 4 in parallel, within a 60-second budget and 2 MB per page.
- Only public IP addresses are fetched; redirects are re-checked at every hop.
- Competitors are only touched through robots.txt, sitemaps and the homepage.
Only audit sites you own or are authorised to audit (see the Acceptable Use Policy). To block the crawler, add a User-agent: SiteAuditBot group with Disallow: /.
What we never use
No scraped search-engine result pages, no logged-in sessions, no personal data sources.