# Fetching in depth

`POST /v1/fetch` turns a URL into text, metadata and provenance. This guide
explains what happens in between and how to steer it.

## The order of operations

1. **Normalise** the URL (lower-case host, default port and fragment dropped,
   tracking parameters such as `utm_*`, `gclid`, `fbclid` removed, remaining
   parameters sorted). This is the cache key.
2. **Cache.** A fresh entry that satisfies your request answers at once,
   with `cached: true` and `cost_usd: 0`. The cache is shared by all
   projects: HTML pages stay fresh for 7 days, PDFs for 30. A stale entry is
   revalidated with a conditional request: if the origin answers `304 Not
   Modified`, you get the cached copy with `cached: true` and
   `revalidated: true`, at no cost.
3. **Safety.** Private, loopback, link-local and metadata addresses are
   refused (`blocked_destination`), also after redirects. A host whose name
   does not exist (NXDOMAIN twice within 6 h) answers `error: "domain_dead"`
   at once for 6 h — or its archived copy, see below.
4. **robots.txt** is honoured for `CormorantBot` and `*`
   (`robots_disallowed`).
   Some hosts are fetched **only through the paid tiers**, never from
   Cormorant's own addresses: North Data (`northdata.com`, `northdata.de`).
   For them, `allow_paid: false` answers `paid_tier_required`, robots.txt is
   read through the proxy, and a site-wide throttle applies (see
   [North Data via Cormorant](/docs/northdata.md)).
5. **Domain memory.** A domain where every tier failed in 3 consecutive
   fetches is skipped for a while (10 min, then 30 min, 2 h, 6 h) and answers
   `error: "domain_unavailable"` at once, with `retry_after_s`. Any success
   ends the skip. Cormorant also remembers which tier (and Web Render
   difficulty) worked for a domain and starts there next time.
6. **Tiers** run until one produces a good page.

## Tiers and escalation

| Tier | What | Cost |
|---|---|---|
| `direct` | HTTP with Chrome's TLS/HTTP2 fingerprint and headers, redirects, compression | free |
| `browser` | Chromium (new headless) with per-site cookies; waits out JS interstitials | free |
| `archive` | recent Wayback Machine snapshot (≤ 365 days), only when blocked | free |
| `proxy` | Massive residential proxy | per GB |
| `proxy_render` | Massive Web Render | per call |

The ladder starts at `direct` (or at a tier learned for the domain, see
architecture). It escalates on network errors, HTTP 403/429/451/5xx and
bot-challenge pages (Cloudflare, Akamai, DataDome, PerimeterX, Imperva,
Kasada, generic interstitials). Thin main text escalates `direct → browser`
only. `archive` and the paid tiers are used only when our own tiers were
*blocked* — never for a page that is simply short. Definitive answers (404,
410, 401, …) are returned without escalation. No CAPTCHA is solved unless the
operator enabled it for the project (Massive option, see operations).

**Dead domains and missing pages.** A host whose name does not exist
(NXDOMAIN — not a temporary DNS failure such as SERVFAIL or a timeout, which
stay `blocked_destination` with reason `dns_error`) twice within 6 hours is
remembered as dead for 6 hours: fetches answer `error: "domain_dead"` at once,
with a single `{"tier": "dns", "reason": "domain_dead"}` attempt and no tier
work. When a fetch ends in `domain_dead` or a 404/410 and `allow_archive` is
true (and `tiers`, if given, includes `archive`), Cormorant looks for a Wayback
snapshot (≤ 365 days) and, if there is one, returns it instead: `tier:
"archive"`, `archived_at`, and `served_from_archive_because: "domain_dead"` or
`"not_found"`; `attempts` still shows the 404 or the dead host. These stand-in
copies are not cached. The archive's paywall rules apply (no copies for known
paywalled domains or of articles marked not free, `archive_blocked_paywall`).
Without a snapshot the answer stays `domain_dead` or `http_404`/`http_410`
(with an `archive_not_found` attempt), and only then are alternative sources
looked for. Other definitive statuses (401, …) never use the archive.

`render=never` uses `direct → archive → proxy`; `render=always` uses
`browser → proxy_render`.

`attempts[].reason` values include `ok`, `thin_content`, `not_modified`,
`challenge_<vendor>`, `http_<status>`, `timeout`, `tls_error`, `dns_error`,
`dns_nxdomain`, `domain_dead`, `network_*`, `not_configured`, `budget_exhausted`,
`archive_not_found`, `massive_captcha_rejected`, `massive_no_credits`, `paywall` (teaser
of a hard paywall, returned as is), `paywall_skip_paid`, `archive_blocked_paywall`, and
for the alternative-source steps (`tier: source`) `alternate_<source>` (with the copy's
URL in `detail`), `syndication_candidates_<n>`, `open_access_none`,
`open_access_miss`, `guardian_api_miss`, `publisher_feed_miss`,
`publisher_feed_teaser`, `common_crawl_none`, `common_crawl_not_free`,
`common_crawl_paywalled_domain`, `common_crawl_index_error`,
`common_crawl_record_error`, `source_timeout`. Web Render attempts carry
`detail` with the difficulty used (`low`, `medium`, `high`); other attempts
omit `detail`. Busy Web Render answers (`massive_render_503`) are retried
once and appear as two attempts.
## What you get back

| Field | What it is |
|---|---|
| `text` | The retained text: main content for HTML (trafilatura, readability fallback, then visible body text), pages joined by blank lines for PDFs, the decoded document for JSON/XML/plain text, empty for binary. Deterministic — the same bytes always give the same text, and cache hits return identical text, so you can anchor quotes against it. |
| `pdf_text_pages`, `pdf_page_count`, `text_truncated` | For PDFs: `[{"page": 1, "text": "…"}, …]`, the number of pages, and whether the text stops early (very long documents). |
| `meta` | For HTML pages: author, published/modified dates, language, canonical URL, publisher, site name, description, section and word count ([details](/docs/api.md#article-metadata-meta)). |
| `html` | The decoded HTML of HTML pages (sync only by default; see `include_html`). |
| `title` | Document title, if found. |
| `content_kind` | `html`, `pdf`, `text`, `json`, `xml` or `binary` — from the content type, sniffed when it is missing or generic. |
| `status`, `final_url`, `redirects` | The HTTP status, the URL after redirects (`null` when unknown, for Web Render) and each redirect hop. |
| `content_type`, `origin_content_type`, `headers` | What the body is, what the origin said it was, and an allowlist of the origin's response headers (never cookies). |
| `tier`, `attempts` | Which tier produced the page, and every attempt with its reason and duration. |
| `sha256`, `bytes`, `body_url`, `body_kind` | The exact retrieved bytes and where to download them. |
| `cached`, `retrieved_at`, `etag`, `last_modified`, `archived_at` | Freshness and provenance. |
| `cost_usd`, `error`, `request_id` | Spend, outcome and the id to quote. |

### Non-HTML content

`content_kind` tells you how to read `text`:

- `json` — `text` is the decoded document (still valid JSON);
- `xml` — feeds, sitemaps, XBRL…: one text node per line, `title` from the
  first `<title>`; external entities are never resolved;
- `text` — plain text, CSV, Markdown, decoded;
- `binary` — spreadsheets, archives, images…: `text` is empty, download the
  bytes from `body_url`.

All of these are successful fetches (`error: null`).

### PDFs

PDFs are accepted by default (`accept_pdf: false` turns them into
`error: "pdf_not_accepted"`). Text is extracted per page; use
`pdf_text_pages` to cite page numbers. Large reports may need a higher
`max_bytes` (default 15 MB, at most 50 MB); a bigger body is
`error: "too_large"`.

## Exact bytes and provenance

Every fetch stores the exact bytes it retrieved, content-addressed by
`sha256`, and returns `body_url` (null when object storage is unavailable).
`body_kind` says what the hash covers:

- `origin`: the HTTP entity body the origin (or the archive, for
  `tier = archive`) sent, after content-encoding (gzip/br) is removed — the
  bytes a plain HTTP client would hand you;
- `rendered_dom`: the document serialized by a browser (`browser`,
  `proxy_render`) after scripts ran — use it for text, not as proof of the
  origin's bytes. Non-HTML documents a browser downloads (PDFs) are `origin`.

`GET /v1/bodies/{sha256}` (same Bearer key) streams the bytes with the origin
`Content-Type`, `Content-Length`, `ETag: "<sha256>"`, `X-Body-Kind` and
`Cache-Control: immutable`; `If-None-Match` gives `304`; `HEAD` returns the
headers only. A project can download only bodies it has retrieved itself
(fresh or from the shared cache); anything else is `404`. Bodies are kept
while a cache entry points to them or a project retrieved them within the
last 30 days.
Downloading a body (Python and TypeScript continue the [Quickstart](/docs/quickstart.md) client):

```sh
curl -sS "$BODY_URL" -H "Authorization: Bearer $CORMORANT_API_KEY" -o report.pdf
```

```python
page = client.post(
    "/v1/fetch", json={"url": "https://www.example.com/annual-report-2025.pdf"}
).json()
if page["body_url"]:
    with client.stream("GET", page["body_url"]) as response:
        response.raise_for_status()
        with open("report.pdf", "wb") as out:
            for chunk in response.iter_bytes():
                out.write(chunk)
```

```ts
const bytes = await fetch(page.body_url, { headers }).then((r) => r.arrayBuffer());
```

For citations, keep `final_url`, `retrieved_at`, `sha256` and `body_kind`
with the quote. For `tier = archive`, `final_url` is the Wayback Machine copy
and `archived_at` the snapshot time.

## Cache semantics

- `cache_ttl_s` is the **maximum age you accept**. `0` bypasses the cache and
  forces a fresh fetch.
- A cache hit is used only if it satisfies your request: `render: "always"`
  needs a rendered copy and `render: "never"` an origin copy; HTML text must
  reach your `min_text_chars`; a cached body larger than your `max_bytes`
  answers `too_large`, exactly as a fresh fetch would; with `tiers` set,
  hits from other tiers are not used.
- Best-effort pages (every tier gave thin text) are never cached.
- Hits cost nothing, even if the page originally came from a paid tier.

## Per-request controls

| Field | Use it to |
|---|---|
| `render` | `auto` (default) climbs the ladder; `never` avoids browsers (`direct → archive → proxy`); `always` goes straight to a browser (`browser → proxy_render`) for JavaScript-only pages. |
| `allow_paid` | `false` never uses `proxy` or `proxy_render` — the fetch is free by construction. |
| `tiers` | Restrict the ladder, e.g. `["direct", "browser"]`. If nothing remains: `error: "no_tier_allowed"`. |
| `allow_archive` | `false` never answers from the Wayback Machine — neither when the live site blocks us nor in place of a dead domain or a 404/410 (`served_from_archive_because`). |
| `timeout_s` | A deadline (5–150 s) for the whole fetch across tiers. When it runs out you get `error: "timeout"` — or the best page found so far. |
| `min_text_chars` | Main text shorter than this (default 500) counts as thin and escalates `direct → browser` only. Lower it for short pages (product pages, abstracts). |
| `max_bytes`, `accept_pdf` | Size cap and PDF acceptance. |
| `country` | Geotargeting for the paid tiers (ISO 3166-1 alpha-2). |
| `accept_language` | The `Accept-Language` header / browser locale, e.g. `de-CH,de;q=0.9`. |
| `user_agent` | Your own User-Agent for `direct`, `browser` and `proxy` — some services (SEC EDGAR, Wikimedia) require a contact address. It must be on your project's allowlist, which the operator maintains; otherwise `403 user_agent_not_allowed`. |
| `include_html` | Return `html` (default `true` for `/v1/fetch`, `false` in batches). Turn it off when you only need text — responses get much smaller. |
| `project_ref` | Free text stored with your usage, e.g. the entity the fetch was for. |

A cost-capped, latency-capped fetch for an agent:

```json
{
  "url": "https://www.holcim.com/investors",
  "allow_paid": false,
  "timeout_s": 45,
  "include_html": false
}
```

A filing from a service that requires a contact User-Agent:

```json
{
  "url": "https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=0000320193",
  "user_agent": "Example Research contact@example.org",
  "tiers": ["direct"],
  "render": "never"
}
```

## Errors

The fetch-level `error` codes, what they mean and whether to retry them are in
[Errors & retries](/docs/errors.md).