Audit API vs writing your own crawler
Fetching a page and reading its title is easy. The work is in the details: robots.txt handling (including what to do when the file errors), sitemap indexes and gzip files, redirect handling without loops, refusing private IP addresses, staying polite to the site, and turning raw findings into a ranked plan.
| A home-made crawler | GetAllAPIs | |
|---|---|---|
| robots.txt | Parse rules, unreachable files and per-agent groups yourself | Parsed per agent; 404 means allowed, 5xx means disallowed |
| Safety | Your server fetches any URL a user types | Public-IP-only fetching, redirect re-checks, size and time limits |
| Ranking | You decide what matters | Severity × reach ÷ effort, now/next/later |
| Maintenance | Yours | Ours |
| Cost | Engineering time and hosting | Per report |
Choose a home-made crawler when
You need checks specific to your platform, or run millions of pages where per-report pricing stops making sense.
Choose the audit API when
You want standard checks done properly now, and would rather spend the afternoon on fixes.
This page describes categories of tools in general terms; individual products differ. Our own limits are listed on each endpoint page and in every report's not_covered.
Run the full audit on your site. One call checks every item in the check reference and ranks the fixes.
Open the playground Endpoint details