# Data sources and modes

Every endpoint maps to exactly one upstream provider through an adapter. The run envelope says which one in `source` (`id`, `name`, `kind`, `demo`).

| `source.id` | Kind | Upstream | Endpoints |
|---|---|---|---|
| `site` | first-party | Crawl of the audited site, robots.txt, sitemaps, competitors' public sitemaps | `seo.audit`, `seo.keyword_gap` |
| `web` | first-party | Our own fetcher (public pages only, robots.txt honoured) | `web.scrape`, `web.map`, `web.crawl`, `seo.domain_technologies` |
| `github` | official API | GitHub REST API | `github.*` |
| `hackernews` | official API | Hacker News Firebase API + HN Search (Algolia) | `hackernews.*` |
| `bluesky` | official API | Bluesky public AppView | `bluesky.*` |
| `sec` | open data | SEC EDGAR | `sec.filings` |
| `apple` | official API | iTunes Search API + App Store reviews RSS | `appstore.*` |
| `dns` | official API | Cloudflare DNS over HTTPS | `email.verify` |
| `rss` | public feed | The publication's RSS feed | `substack.posts` |
| `wikipedia` | open data | Wikimedia APIs (CC BY-SA) | `wikipedia.*` |
| `openalex` | open data | OpenAlex (CC0) | `openalex.works` |
| `ecb` | open data | ECB euro reference rates | `fx.rates` |
| `dataforseo` | licensed | Keyword volumes, rankings, backlinks (only when configured) | `seo.*` reports |
| `mock` | demo | Fictional numbers for demos; `demo: true` | `seo.*` reports |

`seo.content_brief` needs search-result data, so in free mode it returns `501 not_available` before any charge.

## Coming soon

Endpoints whose only upstream is a paid licence, an API whose terms forbid resale, or scraping a platform against its terms are listed with `status: "coming_soon"`, the planned upstream, its cost and the reason. They are never callable: `501 coming_soon`, no charge.

## No failover

One upstream per endpoint and no automatic failover. The catalog says `failover: false`. A call whose upstream fails is refunded and returns an error with `billed: false`.

## Crawler behaviour

The crawler identifies as `GetAllAPIsBot`, fetches only public IP addresses, honours robots.txt for every page after a submitted audit homepage, and caps pages, bytes and time per call.

Human-readable version: [/sources](/sources).
