# Errors & retries

Cormorant reports problems on two levels:

1. **HTTP status codes** — something is wrong with the *request*: the key,
   a limit, validation, or a wait that ran out.
2. **The `error` field** in search and fetch results — the request was fine,
   but the *web* said no: a 404, a robots.txt rule, a site that blocks every
   tier. These come with HTTP `200` (or as a batch item's `error`).

## HTTP status codes

| Status | Body | Meaning | Retry? |
|---|---|---|---|
| `422` | `{"detail": [{"loc": […], "msg": "…", "type": "…"}]}` or `{"detail": "…"}` | Invalid request (a field out of range, a bad date, a non-http `callback_url`). | No — fix the request. |
| `401` | `{"detail": "missing bearer token"}` / `{"detail": "invalid API key"}` | No key, or an unknown/revoked key. | No. |
| `403` | `{"detail": {"error": "customer_suspended", "message": "…"}}` | The account that owns the key's project is suspended (any `/v1` endpoint). | No — contact the operator. |
| `403` | `{"detail": {"error": "user_agent_not_allowed", "hint": "…"}}` | A `user_agent` that is not on the project allowlist. | No — ask the operator or drop the field. |
| `404` | `{"detail": "job not found"}` / `{"detail": "body not found"}` | Unknown job or body, or one that belongs to another project (or has expired). | No. |
| `409` | `{"detail": "idempotency_key already used for a different job"}` | Same `idempotency_key` with a different kind or item count. | No — use a new key. |
| `413` | `{"detail": "at most 50000 items per job"}` | Too many items (or a request body over 64 MB). | Split the job. |
| `403` | `{"detail": {"error": "key_category_not_allowed", "message": "…"}}` | The operator limited this key to other categories (e.g. search only). | No — use another key. |
| `429` | `{"detail": "rate limit exceeded"}` + `Retry-After: 1` | Project rate limit. | Yes, after `Retry-After`. |
| `429` | `{"detail": {"error": "key_rate_limited", …}}` + `Retry-After: 1` | The key's own requests-per-second limit. | Yes, after `Retry-After`. |
| `429` | `{"detail": {"error": "key_quota_exhausted", "category": "fetch", "limit": 100000, …}}` | The key used its monthly request quota for that category (`search`, `fetch`). `Retry-After` = seconds until the month resets. | Not before the reset — see `GET /v1/quota`. |
| `429` | `{"detail": {"error": "key_budget_exhausted", "limit": "50.00", "currency": "USD", …}}` | The key reached its monthly spend cap. `Retry-After` = seconds until the month resets. | Not before the reset. |
| `503` | `{"detail": "…"}` | A dependency is briefly unavailable (e.g. object storage while downloading a body). | Yes, with backoff. |
| `504` | `{"detail": {"error": "timeout", "request_id": "req_…"}}` | No result within the sync wait (150 s, or `timeout_s` + 15 s). | Yes, later — the work often completes in the background and the retry is a cache hit. Better: use a batch. |

## Fetch errors (`error` in the result)

| `error` | Meaning |
|---|---|
| `invalid_url` | not an http(s) URL |
| `blocked_destination` | private, loopback, link-local or metadata address (also after redirects) |
| `robots_disallowed` | robots.txt disallows the path for `CormorantBot`/`*` |
| `too_large` | body exceeds `max_bytes` (batch PDFs with `max_bytes` unset: 200 MB) |
| `pdf_not_accepted` | PDF with `accept_pdf=false` |
| `http_<status>` | definitive upstream status (e.g. `http_404`) |
| `all_tiers_failed` | every allowed tier failed or was blocked (`status` = last HTTP status) |
| `domain_unavailable` | every tier failed for this domain in 3 consecutive fetches: it is skipped for a while (10 min, then 30 min, 2 h, 6 h); `retry_after_s` says when to try again. Any success ends the skip |
| `domain_dead` | the host name does not exist (NXDOMAIN twice within 6 h); answered at once for 6 h, without trying. With `allow_archive`, a snapshot is returned instead when one exists (`served_from_archive_because: "domain_dead"`) |
| `timeout` | `timeout_s` ran out (see per-request controls), or a large PDF took longer than 5 minutes to download (attempt reason `pdf_download_timeout`) |
| `no_tier_allowed` | `tiers`/`allow_paid` left no tier to try |
| `politeness_timeout` | could not get a per-domain slot in time (transient) |
| `sec_not_configured` | SEC EDGAR URL, but the operator has not configured Cormorant's SEC identity: SEC is disabled |
| `rate_limited` | SEC EDGAR URL: the cluster-wide SEC rate limit (at most 10 requests/s for all clients) had no slot within 10 s; retry after `retry_after_s` (transient) |
| `sec_unavailable` | SEC EDGAR throttled (403/429/503: all SEC requests back off), failed (5xx) or could not be reached; retry after `retry_after_s` (transient) |
| `redirect_not_allowed` | SEC EDGAR URL that redirected to a host outside SEC (not followed) |
| `paid_tier_required` | a host fetched only through paid tiers (North Data) with `allow_paid: false` or a `tiers` list without `proxy`/`proxy_render` |
| `budget_exhausted` | such a host, and the project's budget leaves nothing for the paid tiers |

When no tier reaches `min_text_chars`, the attempt with the most text is
returned (no error).
| `error` | Retry? |
|---|---|
| `invalid_url`, `blocked_destination` | No. |
| `robots_disallowed` | No — the site asked crawlers not to fetch this path. |
| `too_large`, `pdf_not_accepted`, `no_tier_allowed` | Not as is — change `max_bytes`, `accept_pdf`, `tiers` or `allow_paid`. |
| `http_404`, `http_410` and other definitive `http_<status>` | No (perhaps much later, if the page may reappear). |
| `all_tiers_failed` | Later (hours), or with paid tiers allowed if you had `allow_paid: false`. Look at `attempts` to see what blocked. |
| `timeout` | Yes, with a larger `timeout_s` — or in a batch. |
| `domain_unavailable` | Not before `retry_after_s` seconds — earlier requests get the same answer at once, without trying. Try another source meanwhile. |
| `domain_dead` | No (hours at the earliest) — the host name does not exist. Look for the page elsewhere; with `allow_archive` (the default) you already got the archived copy when there is one. |
| `politeness_timeout` | Yes, after a short pause: the domain was busy. Batches handle this for you. |
| `rate_limited`, `sec_unavailable` | Yes, after `retry_after_s` (SEC EDGAR's shared rate limit or backoff). Batches handle this for you. |
| `sec_not_configured` | No — SEC EDGAR is not enabled on this deployment; ask the operator. |
| `redirect_not_allowed` | No — an SEC EDGAR URL redirected outside SEC. |

Batch items are retried automatically for transient problems (connection
errors, politeness timeouts, provider rate limits); what ends up in an item's
`error` is final for that job.

## Search errors

A search whose providers could not answer returns `results: []` and `error` —
for example `budget_exhausted` (no budget left for the paid fallback),
`brave_http_429` (the fallback was rate-limited) or a SearXNG failure such as
`searxng_timeout`. These are transient: retry later with backoff. A search
that the engines answered but that found nothing is **not** an error:
`results: []` with `error: null` — rephrase the query instead of retrying.

## Backoff

Retry only what is retryable, cap the number of attempts, and back off
exponentially with jitter. Respect `Retry-After` when present.

```python
import random
import time

import httpx

RETRY_STATUS = {429, 502, 503, 504}  # but not key_quota/budget_exhausted: see below


def post_with_retries(client: httpx.Client, path: str, body: dict, attempts: int = 5) -> dict:
    for attempt in range(attempts):
        try:
            response = client.post(path, json=body)
        except httpx.TransportError:
            response = None
        if response is not None and response.status_code not in RETRY_STATUS:
            response.raise_for_status()  # 4xx other than 429: do not retry
            return response.json()
        if attempt == attempts - 1:
            break
        retry_after = response.headers.get("retry-after") if response is not None else None
        delay = (
            float(retry_after) if retry_after else min(60, 2**attempt) * random.uniform(0.5, 1.5)
        )
        time.sleep(delay)
    raise RuntimeError(f"{path} failed after {attempts} attempts")
```

```ts
const RETRY_STATUS = new Set([429, 502, 503, 504]);

async function postWithRetries(path: string, body: unknown, attempts = 5): Promise<any> {
  for (let attempt = 0; attempt < attempts; attempt++) {
    let res: Response | undefined;
    try {
      res = await fetch(`${BASE}${path}`, { method: "POST", headers, body: JSON.stringify(body) });
    } catch {
      res = undefined; // network error: retry
    }
    if (res && !RETRY_STATUS.has(res.status)) {
      if (!res.ok) throw new Error(`${res.status}: ${await res.text()}`);
      return res.json();
    }
    if (attempt === attempts - 1) break;
    const retryAfter = Number(res?.headers.get("retry-after"));
    const delay = retryAfter > 0 ? retryAfter * 1000 : Math.min(60_000, 2 ** attempt * 1000) * (0.5 + Math.random());
    await new Promise((r) => setTimeout(r, delay));
  }
  throw new Error(`${path} failed after ${attempts} attempts`);
}
```

A fetch that returns `200` with an `error` is a *result*. Decide per code with
the table above whether to retry it later; do not loop on it immediately.

## Debugging

Every response and every batch item has a `request_id`. `attempts` shows each
tier with its HTTP status, reason (`challenge_cloudflare`, `http_403`,
`thin_content`, `timeout`, …) and duration. When you ask the operator about a
request, send the `request_id`.