Website Crawl
Crawl up to 25 same-site pages from a start URL (robots.txt honoured); pay per page fetched.
live web.crawl $0.0125 per page ($12.50 per 1,000)
Pricing
$0.0125 per page ($12.50 per 1,000), paid from a prepaid USD wallet. The exact amount is on every response as cost_usd; max_cost_usd refuses anything higher; failed calls are refunded.
1,000 pages = $12.50 · 10,000 = $125.00 · 100,000 = $1,250.00
Reserved for the maximum pages you request; unused pages are refunded at once.
Run it
curl -s -X POST https://getallapis.com/v1/run/web.crawl \
-H "Authorization: Bearer $GETALLAPIS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com","max_pages":5}'
from getallapis_client import Client
client = Client() # reads GETALLAPIS_API_KEY
result = client.run('web.crawl', {'url': 'https://example.com', 'max_pages': 5})
print(result['cost_usd'], result['output'])
getallapis run web.crawl '{"url":"https://example.com","max_pages":5}'
Quote first, free: POST /v1/quote/web.crawl with the same body. MCP: run_endpoint with endpoint="web.crawl".
Input
| Field | Type | Required | Default | Notes |
|---|---|---|---|---|
url | string | yes | | Start URL |
max_pages | integer | no | 10 | Pages to fetch; you pay only for pages fetched |
include_text | boolean | no | true |
Output
Returned in output of the run envelope (ok, request_id, endpoint, status, cost_usd, items, latency_ms, source, output).
| Field | Meaning |
|---|---|
pages | One scrape result per fetched page. |
count | Pages fetched (billed). |
skipped_by_robots | Links skipped because robots.txt disallows them. |
Source and latency
GetAllAPIs fetcher (first party). Own HTTP fetcher; public pages only, robots.txt honoured, SSRF-guarded. One upstream per endpoint; no automatic failover.
Not enough calls measured yet.