Search

Website Crawl

Crawl up to 25 same-site pages from a start URL (robots.txt honoured); pay per page fetched.

live web.crawl $0.0125 per page ($12.50 per 1,000)

Pricing

$0.0125 per page ($12.50 per 1,000), paid from a prepaid USD wallet. The exact amount is on every response as cost_usd; max_cost_usd refuses anything higher; failed calls are refunded.

1,000 pages = $12.50 · 10,000 = $125.00 · 100,000 = $1,250.00

Reserved for the maximum pages you request; unused pages are refunded at once.

Run it

curl -s -X POST https://getallapis.com/v1/run/web.crawl \
  -H "Authorization: Bearer $GETALLAPIS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","max_pages":5}'
from getallapis_client import Client
client = Client()  # reads GETALLAPIS_API_KEY
result = client.run('web.crawl', {'url': 'https://example.com', 'max_pages': 5})
print(result['cost_usd'], result['output'])
getallapis run web.crawl '{"url":"https://example.com","max_pages":5}'

Quote first, free: POST /v1/quote/web.crawl with the same body. MCP: run_endpoint with endpoint="web.crawl".

Input

FieldTypeRequiredDefaultNotes
urlstringyesStart URL
max_pagesintegerno10Pages to fetch; you pay only for pages fetched
include_textbooleannotrue

Output

Returned in output of the run envelope (ok, request_id, endpoint, status, cost_usd, items, latency_ms, source, output).

FieldMeaning
pagesOne scrape result per fetched page.
countPages fetched (billed).
skipped_by_robotsLinks skipped because robots.txt disallows them.

Source and latency

GetAllAPIs fetcher (first party). Own HTTP fetcher; public pages only, robots.txt honoured, SSRF-guarded. One upstream per endpoint; no automatic failover.

Not enough calls measured yet.

Markdown reference · JSON · Playground