Data sources and modes
On this page
Every endpoint maps to exactly one upstream provider through an adapter. The run envelope says which one in source (id, name, kind, demo).
source.id | Kind | Upstream | Endpoints |
|---|---|---|---|
site | first-party | Crawl of the audited site, robots.txt, sitemaps, competitors' public sitemaps | seo.audit, seo.keyword_gap |
web | first-party | Our own fetcher (public pages only, robots.txt honoured) | web.scrape, web.map, web.crawl, seo.domain_technologies |
github | official API | GitHub REST API | github.* |
hackernews | official API | Hacker News Firebase API + HN Search (Algolia) | hackernews.* |
bluesky | official API | Bluesky public AppView | bluesky.* |
sec | open data | SEC EDGAR | sec.filings |
apple | official API | iTunes Search API + App Store reviews RSS | appstore.* |
dns | official API | Cloudflare DNS over HTTPS | email.verify |
rss | public feed | The publication's RSS feed | substack.posts |
wikipedia | open data | Wikimedia APIs (CC BY-SA) | wikipedia.* |
openalex | open data | OpenAlex (CC0) | openalex.works |
ecb | open data | ECB euro reference rates | fx.rates |
dataforseo | licensed | Keyword volumes, rankings, backlinks (only when configured) | seo.* reports |
mock | demo | Fictional numbers for demos; demo: true | seo.* reports |
seo.content_brief needs search-result data, so in free mode it returns 501 not_available before any charge.
Coming soon
Endpoints whose only upstream is a paid licence, an API whose terms forbid resale, or scraping a platform against its terms are listed with status: "coming_soon", the planned upstream, its cost and the reason. They are never callable: 501 coming_soon, no charge.
No failover
One upstream per endpoint and no automatic failover. The catalog says failover: false. A call whose upstream fails is refunded and returns an error with billed: false.
Crawler behaviour
The crawler identifies as GetAllAPIsBot, fetches only public IP addresses, honours robots.txt for every page after a submitted audit homepage, and caps pages, bytes and time per call.
Human-readable version: /sources.
Plain Markdown for agents: /docs/data-sources.md · index: llms.txt