# Cormorant API v1

Base URL: `https://api.cormorant.ch`. All endpoints below are
under `/v1` except health and metrics. The contract is versioned: fields are
only ever **added** within v1; nothing is renamed or removed. Guides, the
interactive reference and `llms.txt` are served at `/docs` (see the
[changelog](/docs/changelog.md) for what changed).

## Authentication

Every `/v1` request needs a project API key (except the public
[service status](#service-status), `/v1/status*`):

```
Authorization: Bearer cmr_...
```

Keys belong to exactly one project, are stored only as HMAC-SHA256 hashes, and
are created/revoked by the operator (Mission Control, or
`python -m app.cli create-key <project> [--role account_admin]`,
`revoke-key <prefix>`). Missing or invalid keys get `401`.

There are two kinds of keys, and the prefix tells them apart:

| Prefix | Role (`X-Cormorant-Key-Role`) | Sees |
|---|---|---|
| `cma_…` | `account_admin` — the **account admin key** | everything of the customer account where an endpoint offers it: all its keys, projects, usage and invoices |
| `cmr_…` | `standard` — an **additional key** | its own usage only |

Both authenticate every `/v1` endpoint the same way. A key's role is fixed when
it is created; to change it, have a new key issued and the old one revoked.
Keys issued before roles existed are standard keys. Every authenticated `/v1`
response carries the header `X-Cormorant-Key-Role: account_admin` or
`X-Cormorant-Key-Role: standard`. Each project has a
request-rate limit (default 200 requests/s, fixed one-second window); excess
requests get `429` with `Retry-After: 1`. A batch call counts as one request.
Every authenticated response carries `RateLimit-Policy` and `RateLimit`
(IETF draft; `"project";q=200;w=1` and, for a key with its own limit,
`"key";q=10;w=1`).

**Per-key limits.** The operator can set, per key, a monthly spend cap (in the
customer's invoice currency; provider cost in USD for projects without a
customer), monthly request quotas per category (`search`, `fetch`,
`fetch_paid`), requests per second and the allowed categories. They are
enforced on every request and batch item: `403 key_category_not_allowed`,
`429 key_quota_exhausted` / `key_budget_exhausted` (`Retry-After` until the
month resets, UTC) and `429 key_rate_limited`. A key without `fetch_paid`
(not allowed, or its quota used up) still fetches, without the paid tiers.
Keys with limits get `X-Cormorant-Key-Budget-Remaining: 12.40 USD` and
`X-Cormorant-Quota-Remaining: fetch=18766, search=900` on every response.
Request counts are exact; spend is the aggregates (5 minutes) plus an
estimate since then.
Keys of a suspended customer get `403` with
`{"detail": {"error": "customer_suspended", "message": "…"}}`.

Every response carries a `request_id` (`req_…`; batch items use
`<job_id>:<index>`). Quote it when debugging — Mission Control can show the
full attempt history for any request id.

`cost_usd` is Cormorant's provider spend caused by *this* request in USD
(Brave, Massive); cache hits and own tiers cost 0. Spend is attributed only to
the project whose request made the paid call; the caches are shared. **Keys of
an external customer get `cost_usd: null`**: customers see what they are
charged — per request in `GET /v1/requests` (`charge`), aggregated in
`GET /v1/usage` — never Cormorant's provider cost. Every search and fetch
result (also batch items) carries `billing_category` (`search`,
`search_fallback`, `search_cached`, `fetch_own`, `fetch_paid`,
`fetch_cached`, `sec`, `sec_cached`): the category, and so the plan meters,
the request counts toward. SEC EDGAR requests (`/v1/sec/…` and fetches of
SEC hosts) are `sec` / `sec_cached`: no provider cost, free unless your plan
prices them, and counted against a key's `fetch` quota. North Data results are `northdata` / `northdata_cached`.

## POST /v1/search

```json
{
  "query": "Nestlé sustainability report 2025",
  "language": "en",        // optional, ISO 639-1 (SearXNG language, Brave search_lang)
  "country": "CH",         // optional, ISO 3166-1 alpha-2 (combined with language for SearXNG)
  "max_results": 20,       // 1..50, default 20
  "freshness": "month",    // optional: day | week | month | year (pd/pw/pm/py accepted)
  "cache_ttl_s": 604800,   // optional: max acceptable cache age; 0 = bypass the cache
  "project_ref": "company:4711"  // optional free text, stored with usage
}
```

Response `200`:

```json
{
  "query": "Nestlé sustainability report 2025",
  "results": [
    {
      "url": "https://www.nestle.com/sustainability/...",
      "title": "Creating Shared Value and Sustainability Report 2025",
      "snippets": ["Our 2025 report covers ...", "Download the full report ..."],
      "engines": ["google", "bing", "brave_api"],
      "published_at": "2026-03-02T00:00:00",
      "rank": 1
    }
  ],
  "provider": "searxng",          // "searxng" | "brave"
  "fallback_reason": null,        // why Brave was used (see below), else null
  "unresponsive_engines": [],     // SearXNG engines that failed for this query
  "cached": false,
  "cost_usd": 0.0,                // null for keys of an external customer
  "billing_category": "search",   // search | search_fallback | search_cached
  "error": null,
  "request_id": "req_6b1f..."
}
```

Order of operations: shared search cache (key = normalised query + language,
country, max_results, freshness; 7 days) → our SearXNG instances (least-loaded
across all hosts, a second instance on transport failure) → Brave Search API.

Brave is used when SearXNG errors or times out (`searxng_error`,
`searxng_timeout`, `no_searxng_instance`), reports unresponsive engines
(`unresponsive_engines`, or `captcha_engines` when any reason mentions a
CAPTCHA), or returns fewer than `MIN_RESULTS` (3) usable results
(`too_few_results`). By default unresponsive engines trigger the fallback only
when fewer than two engines returned results (`SEARCH_FALLBACK_ON_UNRESPONSIVE=
quorum`, `SEARCH_QUORUM_ENGINES=2`); `majority` (as many engines failed as
answered) and `any` are stricter. A Brave rate limit (429) is retried after the
indicated wait; batch items that still fail with it are retried later.

Results are deduplicated by normalised URL (scheme, `www.`, trailing slash and
tracking parameters ignored). **No snippet is lost**: every distinct snippet
and engine seen for a URL is kept, in order of first appearance. When Brave is
used, Brave's results come first, followed by any SearXNG-only results.

If Brave is unavailable or the project budget is exhausted, whatever SearXNG
returned is served (not cached) with `fallback_reason` set. A **genuinely
empty** search — the engines (and Brave, when consulted) answered and found
nothing — returns `results: []` with `error: null`. Only when the providers
could not answer is `error` set (e.g. `budget_exhausted`, `brave_http_429`,
`searxng_timeout`).

## POST /v1/fetch

```json
{
  "url": "https://www.example.com/annual-report-2025.pdf",
  "render": "auto",          // auto | never | always
  "max_bytes": 15728640,     // default 15 MB, max 50 MB (batch items left at the default: PDFs up to 200 MB)
  "accept_pdf": true,
  "cache_ttl_s": 604800,     // optional max acceptable cache age; 0 = bypass
  "min_text_chars": 500,     // below this, HTML main text counts as "thin"
  "allow_archive": true,     // allow an archived copy when the live site blocks us, is gone or 404s
  "include_html": true,      // default true here, false for batch items
  "allow_paid": true,        // false: never use proxy / proxy_render
  "tiers": ["direct", "browser", "archive"],   // optional allowlist of tiers
  "timeout_s": 60,           // optional overall deadline, 5–150 s
  "country": "CH",           // optional Massive geotargeting (paid tiers)
  "user_agent": "Example Research contact@example.org",  // must be on the project allowlist
  "accept_language": "de-CH,de;q=0.9",
  "project_ref": "company:4711"
}
```

Response `200` (also for unsuccessful fetches — check `error`):

```json
{
  "url": "https://www.example.com/annual-report-2025.pdf",
  "final_url": "https://www.example.com/annual-report-2025.pdf",  // null when unknown (Web Render)
  "redirects": [                    // one entry per redirect before final_url
    {"url": "http://example.com/ar25", "status": 301, "location": "https://www.example.com/annual-report-2025.pdf"}
  ],
  "status": 200,
  "content_type": "application/pdf", // what the body is (rendered pages: text/html)
  "origin_content_type": "application/pdf",  // what the origin sent; null when unknown
  "headers": {"content-type": "application/pdf", "last-modified": "Tue, 03 Mar 2026 09:00:00 GMT"},
  "tier": "direct",                 // direct | browser | archive | proxy | proxy_render
  "attempts": [
    {"tier": "direct", "status": 200, "reason": "ok", "ms": 412}
  ],
  "html": null,                     // decoded HTML for HTML pages
  "text": "Annual Report 2025\n\n...",
  "pdf_text_pages": [{"page": 1, "text": "Annual Report 2025"}, {"page": 2, "text": "..."}],
  "pdf_page_count": 148,            // PDFs: number of pages; null otherwise
  "text_truncated": false,          // PDF text stopped at the extraction limits
  "title": "Annual Report 2025",
  "meta": null,                     // article metadata for HTML pages (see below)
  "sha256": "a3f1...",              // of the retrieved bytes
  "bytes": 2483112,
  "cached": false,
  "revalidated": false,             // true: expired cache entry confirmed by the origin (304)
  "retrieved_at": "2026-10-19T08:12:44.120Z",
  "etag": "\"5f2-1a\"",
  "last_modified": "Tue, 03 Mar 2026 09:00:00 GMT",
  "archived_at": null,              // snapshot time (tier = archive) or common_crawl capture time
  "served_from_archive_because": null,  // domain_dead | not_found when the archive stands in
  "body_url": "https://api.cormorant.ch/v1/bodies/a3f1...",  // exact retrieved bytes
  "body_kind": "origin",            // origin | rendered_dom (what sha256 covers)
  "content_kind": "pdf",            // html | pdf | text | json | xml | binary
  "paywall": null,                  // HTML only: none | metered | hard | unknown
  "paywall_evidence": [],           // e.g. ["jsonld_not_free", "vendor_piano"]
  "text_complete": null,            // false when text is only a paywall teaser
  "alternate_of": null,             // set when text comes from a verified copy elsewhere
  "alternate_source": null,         // publisher_feed | guardian_api | open_access | syndication | common_crawl
  "match_score": null,
  "primary_sources": [],            // [{url, kind, text}] for paywalled/unavailable pages
  "cost_usd": 0.0,                  // null for keys of an external customer
  "billing_category": "fetch_own",  // fetch_own | fetch_paid | fetch_cached | sec | sec_cached
  "error": null,
  "request_id": "req_91c2..."
}
```

`text` is exactly the retained text Cormorant stores and serves again on cache
hits (main text via trafilatura, readability fallback, then visible body text;
whitespace normalised). Anchor quotes against it. For PDFs, `text` is the pages
joined with blank lines and `pdf_text_pages` has the per-page text.

**Large PDFs.** `max_bytes` applies to every response of `/v1/fetch`. In batch
fetches whose items leave `max_bytes` unset, a PDF (by
`Content-Type: application/pdf` or the `%PDF-` signature) may be up to
**200 MB**; other content keeps the default 15 MB. An item that sets
`max_bytes` gets exactly that limit, for PDFs too. Such a PDF is fetched by
the `direct` or `proxy` tier (the browser, archive and Web Render tiers keep
`max_bytes`); its exact bytes are available from `body_url`. Downloading it
may take at most 5 minutes in total (or the item's `timeout_s`, when
shorter); a slower download ends with `error: timeout` and an attempt with
reason `pdf_download_timeout`. Text extraction stops after 5,000 pages or
5,000,000 characters of page text: then `text_truncated` is `true`,
`pdf_text_pages` covers the pages read so far (the last one possibly cut)
and `pdf_page_count` still counts every page. To skip PDFs entirely, set
`accept_pdf: false`.

**Non-HTML content.** `content_kind` classifies the response (content type,
sniffed when missing or generic). `json`: `text` is the decoded document
(still valid JSON). `xml` (feeds, sitemaps, XBRL…): `text` is the text content,
one node per line, `title` from the first `<title>`; external entities are
never resolved. `text` (plain, CSV, Markdown): decoded text. `binary`
(spreadsheets, archives, images…): `text` is empty — the bytes are in
`body_url`. All of these are successful fetches (`error: null`).

**Paywalls.** HTML pages carry a `paywall` verdict. Cormorant detects
paywalls and never circumvents them (no crawler User-Agents, cookie clearing,
script blocking or referer tricks).

| `paywall` | Meaning | `text_complete` |
|---|---|---|
| `hard` | `text` is only the teaser: the publisher declares the article not free (`isAccessibleForFree: false`) and the text is cut short, or a subscription call to action follows a teaser together with paywall markup, vendor scripts or truncation | `false` |
| `metered` | a meter ("2 free articles left") or a declared paywall, but the full text came through | `true` |
| `unknown` | weak signals only (a paywall vendor's script, paywall-like markup) | `true` |
| `none` | no paywall signal | `true` |

`paywall_evidence` lists the signals: `jsonld_not_free`, `vendor_<name>`
(piano, poool, zephr, pelcro, laterpay, steady, blendle, …),
`paywall_markup`, `truncation_markup`, `phrase_<lang>`, `metered_<lang>` (en,
de, fr, it) and `length_mismatch` (declared word count far above the
extracted text). A `hard` teaser is a successful fetch (`error: null`)
returned at once — other tiers see the same paywall — and is not cached.
Domains whose articles are repeatedly hard-paywalled skip the paid tiers
(`paywall_skip_paid` attempts). Wayback copies of articles the publisher marks
as not free are not served (`archive_blocked_paywall`); the operator can allow
them per project (`archive_paywalled`).

**Alternative sources.** When the page is a hard-paywall teaser,
disallowed by robots.txt, gone (404/410, `domain_dead`) or blocked on every tier
(`all_tiers_failed`, `domain_unavailable`), Cormorant looks for a verified
open copy elsewhere (`allow_alternates`, default `true`):

| `alternate_source` | How it is found and verified |
|---|---|
| `publisher_feed` | tried first, for paywall teasers and blocked pages only: the article's item in the publisher's own full-text RSS/Atom feed (feeds Cormorant found linked from the publisher's pages or the operator added, and detected to carry whole articles). The item must link to the requested URL (tracking parameters ignored; `match_score` 1.0) or have the same title and contain ≥ 50 % of the teaser's opening word 5-grams; items that are themselves teasers (short, or ending in a subscribe call-to-action) are never used. `final_url` is the feed |
| `guardian_api` | theguardian.com URLs via the Guardian Content API (when the operator configured a key) |
| `open_access` | a DOI in the URL or the page's citation metadata → the best open-access location OpenAlex knows (repository, arXiv, Europe PMC, publisher OA), fetched with the free `direct` tier; `match_score` 1.0 |
| `syndication` | the headline is searched; copies on other sites (wire services first) are fetched with `direct` and accepted only if ≥ 50 % of the teaser's opening word 5-grams appear in them and they are not paywalled themselves; `match_score` is that share |
| `common_crawl` | **batch fetches only** (too slow for synchronous calls), for pages that are gone (404/410, `domain_dead`) or blocked on every tier — never for paywall teasers or robots-disallowed pages: the newest HTTP 200 capture of the exact URL in the last three Common Crawl crawls (CDX index, then the single WARC record by HTTP Range request). Domains known to be paywalled are skipped, and a capture whose markup declares the article not free (`isAccessibleForFree: false`) or that is itself a paywall teaser is not served. `archived_at` is the capture time, `final_url` the captured URL, `html` the captured page; `match_score` 1.0 |

The result then has `tier: "source"`, `alternate_of` (the URL asked for),
`final_url` (the copy), `text_complete: true` and `paywall` still describing
the original page. Without a verified copy the original answer stands (the
flagged teaser or the error) and `attempts` show what was tried.
`primary_sources` lists links from the page to court decisions,
government/regulator pages, press releases, DOIs and PDFs. Searching for
copies may use the Brave fallback (billed like a search). The operator can
turn alternates off per project (`set-option <project> alternates off`).

**Provenance.** `redirects`, `headers` (an allowlist of the origin's final
response headers: content type/length/language/disposition, ETag,
Last-Modified, Cache-Control, Expires, Date, Link, X-Robots-Tag — never
cookies) and `origin_content_type` are kept with cache entries and returned on
hits. For `tier = archive`, `final_url` is the archived copy's URL
(`https://web.archive.org/web/<timestamp>id_/<url>`) and `archived_at` the
snapshot time. Web Render (`proxy_render`) does not report the post-redirect
URL or origin headers: `final_url`, `origin_content_type` are `null` and
`headers` is empty rather than guessed.

**Per-request controls.** `allow_paid=false` removes the paid tiers;
`tiers` restricts the ladder to the listed tiers (an empty intersection is
`error: no_tier_allowed`; cache hits from other tiers are not used).
`timeout_s` is a deadline for the whole fetch across tiers — when it runs out
the result is `error: timeout` with a `request_timeout` attempt (or the best
page so far). `country` geotargets Massive's proxy and Web Render.
`user_agent` replaces our browser User-Agent for `direct`, `browser` and
`proxy` (not Web Render); it must be on the project's allowlist, which the
operator sets (`set-option <project> allowed_user_agents '["…"]'`) — services
such as SEC EDGAR and Wikimedia require a contact User-Agent. Otherwise
`403 user_agent_not_allowed`. `accept_language` sets the Accept-Language
header / browser locale header.

**Cache hits** are used only when they satisfy the request: `render=always`
needs a rendered copy, `render=never` an origin copy; HTML text must reach
`min_text_chars`; a cached body larger than `max_bytes` answers `too_large`
(as a fresh fetch would). Otherwise the page is fetched again. Pages returned
as *best effort* (every tier gave thin text) are never cached.

**Revalidation.** An expired entry (or one older than `cache_ttl_s`) that
would otherwise satisfy the request and has an `ETag` or `Last-Modified` is
revalidated: the `direct` tier sends `If-None-Match` / `If-Modified-Since`.
`304 Not Modified` returns the cached copy with `cached: true`,
`revalidated: true`, a single `direct` attempt with reason `not_modified`,
`cost_usd: 0`, a new `retrieved_at`, and renews the entry's freshness (new
validators from the 304 are kept). Any other answer takes the normal path (a
`200` replaces the entry). `cache_ttl_s: 0` never revalidates. Other tiers
never send conditional requests.

### Article metadata: meta

For HTML pages `meta` describes the article; it is `null` for other content
kinds and on errors. It is kept with cache entries (identical on hits) and
included in batch results.

```json
"meta": {
  "author": "Anna Muster; Beat Beispiel",
  "published_at": "2026-03-19T08:30:00Z",
  "modified_at": "2026-03-19T10:05:12Z",
  "language": "de-CH",
  "canonical_url": "https://www.example.com/news/central-bank-cuts-rates",
  "publisher": "Example Media AG",
  "site_name": "Example News",
  "description": "The central bank cut its policy rate by 0.25 percentage points.",
  "section": "Economy",
  "word_count": 1240
}
```

| Field | Meaning |
|---|---|
| `author` | Author name(s); several are separated by `; ` |
| `published_at` | Publication date, ISO 8601 (format below) |
| `modified_at` | Last modification date, ISO 8601 |
| `language` | Language tag (BCP 47), e.g. `de-CH` |
| `canonical_url` | Canonical URL the page declares, absolute |
| `publisher` | Publishing organisation |
| `site_name` | Name of the site or periodical |
| `description` | The page's own summary (at most 2,000 characters) |
| `section` | Section or category (the first one) |
| `word_count` | Words in `text` — what was retrieved, not what the page declares |

Every field is `null` when the page does not say. Sources, in order of
preference: JSON-LD (the `NewsArticle`, `Article`, `BlogPosting`,
`ScholarlyArticle` or `Report` node, also inside `@graph` or arrays;
malformed JSON-LD is skipped or repaired, never an error), OpenGraph and
`article:*` meta tags, Highwire `citation_*` meta tags (scholarly pages),
`<html lang>` and `<link rel=canonical>` (which wins for `canonical_url`),
then trafilatura's metadata detection. Dates are
`YYYY-MM-DDTHH:MM:SSZ` (converted to UTC) when the page gives a time zone,
`YYYY-MM-DDTHH:MM:SS` when it gives a time without zone, and `YYYY-MM-DD`
(or `YYYY-MM`, `YYYY`) when it gives only a date — no time or zone is
invented.

### Exact bytes: GET /v1/bodies/{sha256}

Every fetch stores the exact bytes it retrieved, content-addressed by
`sha256`, and returns `body_url` (null when object storage is unavailable).
`body_kind` says what the hash covers:

- `origin`: the HTTP entity body the origin (or the archive, for
  `tier = archive`) sent, after content-encoding (gzip/br) is removed — the
  bytes a plain HTTP client would hand you;
- `rendered_dom`: the document serialized by a browser (`browser`,
  `proxy_render`) after scripts ran — use it for text, not as proof of the
  origin's bytes. Non-HTML documents a browser downloads (PDFs) are `origin`.

`GET /v1/bodies/{sha256}` (same Bearer key) streams the bytes with the origin
`Content-Type`, `Content-Length`, `ETag: "<sha256>"`, `X-Body-Kind` and
`Cache-Control: immutable`; `If-None-Match` gives `304`; `HEAD` returns the
headers only. A project can download only bodies it has retrieved itself
(fresh or from the shared cache); anything else is `404`. Bodies are kept
while a cache entry points to them or a project retrieved them within the
last 30 days.

### Tiers and escalation

| Tier | What | Cost |
|---|---|---|
| `direct` | HTTP with Chrome's TLS/HTTP2 fingerprint and headers, redirects, compression | free |
| `browser` | Chromium (new headless) with per-site cookies; waits out JS interstitials | free |
| `archive` | recent Wayback Machine snapshot (≤ 365 days), only when blocked | free |
| `proxy` | Massive residential proxy | per GB |
| `proxy_render` | Massive Web Render | per call |

The ladder starts at `direct` (or at a tier learned for the domain, see
architecture). It escalates on network errors, HTTP 403/429/451/5xx and
bot-challenge pages (Cloudflare, Akamai, DataDome, PerimeterX, Imperva,
Kasada, generic interstitials). Thin main text escalates `direct → browser`
only. `archive` and the paid tiers are used only when our own tiers were
*blocked* — never for a page that is simply short. Definitive answers (404,
410, 401, …) are returned without escalation. No CAPTCHA is solved unless the
operator enabled it for the project (Massive option, see operations).

**Dead domains and missing pages.** A host whose name does not exist
(NXDOMAIN — not a temporary DNS failure such as SERVFAIL or a timeout, which
stay `blocked_destination` with reason `dns_error`) twice within 6 hours is
remembered as dead for 6 hours: fetches answer `error: "domain_dead"` at once,
with a single `{"tier": "dns", "reason": "domain_dead"}` attempt and no tier
work. When a fetch ends in `domain_dead` or a 404/410 and `allow_archive` is
true (and `tiers`, if given, includes `archive`), Cormorant looks for a Wayback
snapshot (≤ 365 days) and, if there is one, returns it instead: `tier:
"archive"`, `archived_at`, and `served_from_archive_because: "domain_dead"` or
`"not_found"`; `attempts` still shows the 404 or the dead host. These stand-in
copies are not cached. The archive's paywall rules apply (no copies for known
paywalled domains or of articles marked not free, `archive_blocked_paywall`).
Without a snapshot the answer stays `domain_dead` or `http_404`/`http_410`
(with an `archive_not_found` attempt), and only then are alternative sources
looked for. Other definitive statuses (401, …) never use the archive.

`render=never` uses `direct → archive → proxy`; `render=always` uses
`browser → proxy_render`.

`attempts[].reason` values include `ok`, `thin_content`, `not_modified`,
`challenge_<vendor>`, `http_<status>`, `timeout`, `tls_error`, `dns_error`,
`dns_nxdomain`, `domain_dead`, `network_*`, `not_configured`, `budget_exhausted`,
`archive_not_found`, `massive_captcha_rejected`, `massive_no_credits`, `paywall` (teaser
of a hard paywall, returned as is), `paywall_skip_paid`, `archive_blocked_paywall`, and
for the alternative-source steps (`tier: source`) `alternate_<source>` (with the copy's
URL in `detail`), `syndication_candidates_<n>`, `open_access_none`,
`open_access_miss`, `guardian_api_miss`, `publisher_feed_miss`,
`publisher_feed_teaser`, `common_crawl_none`, `common_crawl_not_free`,
`common_crawl_paywalled_domain`, `common_crawl_index_error`,
`common_crawl_record_error`, `source_timeout`. Web Render attempts carry
`detail` with the difficulty used (`low`, `medium`, `high`); other attempts
omit `detail`. Busy Web Render answers (`massive_render_503`) are retried
once and appear as two attempts.

### Errors

| `error` | Meaning |
|---|---|
| `invalid_url` | not an http(s) URL |
| `blocked_destination` | private, loopback, link-local or metadata address (also after redirects) |
| `robots_disallowed` | robots.txt disallows the path for `CormorantBot`/`*` |
| `too_large` | body exceeds `max_bytes` (batch PDFs with `max_bytes` unset: 200 MB) |
| `pdf_not_accepted` | PDF with `accept_pdf=false` |
| `http_<status>` | definitive upstream status (e.g. `http_404`) |
| `all_tiers_failed` | every allowed tier failed or was blocked (`status` = last HTTP status) |
| `domain_unavailable` | every tier failed for this domain in 3 consecutive fetches: it is skipped for a while (10 min, then 30 min, 2 h, 6 h); `retry_after_s` says when to try again. Any success ends the skip |
| `domain_dead` | the host name does not exist (NXDOMAIN twice within 6 h); answered at once for 6 h, without trying. With `allow_archive`, a snapshot is returned instead when one exists (`served_from_archive_because: "domain_dead"`) |
| `timeout` | `timeout_s` ran out (see per-request controls), or a large PDF took longer than 5 minutes to download (attempt reason `pdf_download_timeout`) |
| `no_tier_allowed` | `tiers`/`allow_paid` left no tier to try |
| `politeness_timeout` | could not get a per-domain slot in time (transient) |
| `sec_not_configured` | SEC EDGAR URL, but the operator has not configured Cormorant's SEC identity: SEC is disabled |
| `rate_limited` | SEC EDGAR URL: the cluster-wide SEC rate limit (at most 10 requests/s for all clients) had no slot within 10 s; retry after `retry_after_s` (transient) |
| `sec_unavailable` | SEC EDGAR throttled (403/429/503: all SEC requests back off), failed (5xx) or could not be reached; retry after `retry_after_s` (transient) |
| `redirect_not_allowed` | SEC EDGAR URL that redirected to a host outside SEC (not followed) |
| `paid_tier_required` | a host fetched only through paid tiers (North Data) with `allow_paid: false` or a `tiers` list without `proxy`/`proxy_render` |
| `budget_exhausted` | such a host, and the project's budget leaves nothing for the paid tiers |

When no tier reaches `min_text_chars`, the attempt with the most text is
returned (no error).

## Batch: POST /v1/batch/search and POST /v1/batch/fetch

```json
{
  "items": [{"query": "..."}, {"query": "..."}],       // same objects as the single endpoints
  "idempotency_key": "chipiron-run-2026-10-20-search-0001",
  "callback_url": "https://chipiron.example/hooks/cormorant"   // optional
}
```

Response `202` `{"job_id": "…", "created": true}`. Re-posting the same
`idempotency_key` returns the existing job (`200`, `created: false`); the same
key with a different kind or item count is `409`. Up to 50,000 items per job.
Fetch items may set `"include_html": true` to keep the HTML in stored results
(default: text only). Batch fetch items that leave `max_bytes` unset accept
PDFs up to 200 MB (see *Large PDFs* above).

Items are admitted to the queues gradually (fair share across running jobs),
executed with the same logic as the single endpoints, and retried with backoff
only for transient problems. A batch fetch item blocked on our own tiers is
retried once after a cool-down before a paid tier is used.

When the job completes, `callback_url` receives a `POST` with
`{"job_id","kind","status","total","succeeded","failed","finished_at"}` (3
attempts; private addresses are refused). The host is resolved once, checked,
and the connection goes to that checked address (Host header and TLS name
stay the callback's host name), so a DNS answer that changes in between cannot
redirect it; redirects are not followed.

### GET /v1/jobs/{job_id}

```json
{
  "job_id": "1d0c...",
  "kind": "fetch",
  "status": "running",          // queued | running | completed | cancelled
  "counts": {"total": 3000, "pending": 1200, "queued": 380, "done": 1390, "failed": 30, "cancelled": 0},
  "created_at": "...", "updated_at": "...", "finished_at": null,
  "callback_url": null, "callback_status": null,
  "items_url": "/v1/jobs/1d0c...?include=items&offset=0&limit=100"
}
```

Add `?include=items&offset=0&limit=1000` (max 1000) to page through item
results, optionally `&status=failed`:

```json
"items": [
  {"index": 0, "status": "done", "input": {"url": "..."}, "result": { ...fetch response... }, "error": null}
]
```

**Retention:** jobs and their item results are kept for 30 days after the job
finishes, then deleted (`JOB_RETENTION_DAYS`). Download results you need to
keep before that. If a stored result cannot be read (object storage
unavailable), the item has `"result": null` and `"error": "result_unavailable"`;
retry the request later.

### DELETE /v1/jobs/{job_id}

Cancels the job: no new items are admitted (the admission pump runs every
3 s, well within 60 s), pending items become `cancelled`, queued items skip
themselves. Returns the job status. Jobs are only visible to their project.

## SEC EDGAR: /v1/sec/...

Company and filing data from SEC EDGAR. Cormorant sends **its own** declared
identity to SEC (`"<Org name> <contact email>"`, set by the operator) and
keeps all clients together under SEC's fair-access limit: one cluster-wide
rate limit per egress IP (default 8 requests/s, never more than 10), queued
for up to 10 s, and a shared backoff when SEC throttles (403/429/503). Only
the direct tier is used — never the browser, archived copies, paid proxies or
alternative sources. **Do not send your own User-Agent** for SEC: a
`user_agent`/`accept_language` in `POST /v1/fetch` is ignored for SEC hosts.
The same policy applies to `POST /v1/fetch` and batch items for URLs on
`www.sec.gov`, `data.sec.gov` and `efts.sec.gov`.

All endpoints need an API key, count towards the project's rate limit and
are metered as kind `sec` (no provider cost). Answers are HTTP 200 also when
they failed; check `error`:

| `error` | Meaning |
|---|---|
| `not_found` | SEC has no such company, filing or document (also: no filing of the requested forms) |
| `invalid_identifier` | a CIK, accession number, document name, form type, date or query is malformed (`message` says which); nothing was requested from SEC |
| `rate_limited` | no slot in the shared SEC rate limit within 10 s; retry after `retry_after_s` |
| `sec_unavailable` | SEC throttles (all SEC requests back off) or failed; retry after `retry_after_s` |
| `sec_not_configured` | SEC is not enabled on this deployment |
| `timeout`, `too_large`, `robots_disallowed`, … | the fetch errors |

Identifiers: CIKs as a number or 10 digits (`100493`, `0000100493`);
accession numbers with or without dashes (`0000100493-25-000070`,
`000010049325000070`); form types comma-separated (`10-K,20-F,40-F,10-K/A`).
Only the SEC URL patterns behind each endpoint can be requested.

Every answer has `error`, `message`, `retry_after_s`, `cached` (`true` when
all SEC documents came from Cormorant's cache), `retrieved_at` (oldest
document), `source_url` (the main SEC document), `body_url` and `sha256`
(its exact bytes, e.g. the raw submissions JSON) and `request_id`.

**Caching.** Filing documents and indexes (`/Archives/edgar/data/{cik}/{accession}/…`)
are immutable and cached for 365 days (revalidated only when you ask with
`cache_ttl_s`); `company_tickers.json`, submissions and XBRL facts for 12 h,
then revalidated with conditional requests; full-text search for 1 h.

### GET /v1/sec/companies/resolve?ticker=TSN

Also `?cik=100493` or `?name=tyson foods` (exactly one). From
`company_tickers.json` and `company_tickers_exchange.json`:

```json
{"cik": 100493, "cik_padded": "0000100493", "title": "TYSON FOODS, INC.",
 "tickers": ["TSN"], "exchanges": ["NYSE"], "ambiguous": false,
 "candidates": [{"cik": 100493, "cik_padded": "0000100493", "title": "TYSON FOODS, INC.", "tickers": ["TSN"], "exchanges": ["NYSE"]}],
 "error": null, "cached": true, "source_url": "https://www.sec.gov/files/company_tickers.json", "…": "…"}
```

`BRK.B` and `BRK-B` are the same ticker. A name matches exact titles first,
then titles starting with or containing it; when several companies match,
`ambiguous` is `true`, the company fields are `null` and `candidates` lists
up to 10. A CIK without a listed ticker is resolved from its submissions.

### GET /v1/sec/companies/{cik}/submissions

Query: `forms`, `since`, `until` (filing dates, `YYYY-MM-DD`), `limit`
(default 100, max 10,000), `history` (default `true`: also read the pages of
older filings SEC lists under `files`, only those that can hold filings
after `since`). Company fields `cik`, `cik_padded`, `name`, `title`,
`tickers`, `exchanges`, `entity_type`, `sic`, `sic_description`, `category`,
`fiscal_year_end`, `state_of_incorporation`, `ein`, `lei`, `website`,
`phone`, `business_address`, `former_names`; `filings` (newest first, at most
`limit`), `filings_total`, `truncated`, and `files` (`name`, `filing_count`,
`filing_from`, `filing_to`, `url`, `fetched`, `body_url`). Each filing:

| Field | From SEC's column |
|---|---|
| `accession` | `accessionNumber` |
| `form`, `filing_date`, `report_date`, `acceptance_datetime` | `form`, `filingDate`, `reportDate`, `acceptanceDateTime` |
| `act`, `file_number`, `film_number`, `items`, `core_type`, `size` | `act`, `fileNumber`, `filmNumber`, `items`, `core_type`, `size` |
| `is_xbrl`, `is_inline_xbrl` | `isXBRL`, `isInlineXBRL` (as booleans) |
| `primary_document`, `primary_document_description` | `primaryDocument`, `primaryDocDescription` |
| `primary_document_url`, `index_url` | built: the document and the `-index.htm` page (`null` URL when the primary document is in a sub-folder, e.g. ownership forms) |
| `is_amendment` | built: the form ends in `/A` |

### GET /v1/sec/companies/{cik}/filings/latest?forms=10-K,20-F,40-F

The newest filing (by filing date) of `forms` — default `10-K,20-F,40-F`
(annual reports; amendments only when you list them, e.g. `10-K,10-K/A`):
the filing fields above at the top level (`accession`, `form`,
`filing_date`, `report_date`, `primary_document`, `primary_document_url`,
`index_url`, `is_amendment`, …) plus `cik`, `cik_padded`, `name`, `tickers`,
`exchanges`. Older filing pages are searched when the recent list has none;
none at all: `not_found`.

### GET /v1/sec/filings/{cik}/{accession}

The filing index: `form`, `filing_date`, `report_date`, `accepted`,
`company`, `is_amendment`, `index_url`, `folder_url`, `accession`,
`accession_nodash`, `index_source` (`index_htm`, or `index_json` when only the
folder listing has the documents) and `documents`: `sequence`,
`description`, `document` (file name), `type` (`10-K`, `EX-21`, `EX-101.SCH`,
…), `size`, `url`. Sub-folders are not listed.

### POST /v1/sec/documents/fetch

```json
{"cik": 100493, "accession": "0000100493-25-000070", "document": "tsn-20250927.htm",
 "max_bytes": 52428800}
```

Fetches one document of a filing (HTML/iXBRL, PDF, exhibit, XML) and answers
**exactly like `POST /v1/fetch`** (`text`, `pdf_text_pages`, `content_kind`,
`body_url`, `sha256`, `body_kind: origin`, …). Optional: `include_html`
(default `true`), `accept_pdf`, `cache_ttl_s`, `timeout_s`, `project_ref`.
Invalid identifiers answer `error: invalid_identifier` in that shape.

### GET /v1/sec/companies/{cik}/facts

A summary of `data.sec.gov/api/xbrl/companyfacts/CIK##########.json`:
`cik`, `cik_padded`, `name`, `taxonomies` (concepts per taxonomy),
`concepts`, `bytes`; the complete JSON is at `body_url`.

### GET /v1/sec/search?q=…&forms=…&dateRange=…

EDGAR full-text search (filings since 2001). Query: `q` (EDGAR syntax),
`forms`, `dateRange` (`all`, `10y`, `5y`, `1y`, `30d`, or `custom` with
`startdt`/`enddt`), `from` (offset, pages of 100). Answer: `query`, `total`,
`total_relation`, `hits` (`accession`, `ciks`, `display_names`, `form`,
`root_forms`, `file_type`, `file_description`, `file_date`, `period_ending`,
`document`, `url`, `index_url`, `score`).

## North Data

Company profiles, discovery and network graphs from North Data
(`northdata.com` / `northdata.de`). All four endpoints are synchronous (like
`POST /v1/fetch`), authenticated, metered (kind `northdata`, see *Billing
categories*) and answer HTTP 200 with `error` for upstream failures — never
an HTTP 5xx. For many companies use `POST /v1/batch/northdata`.

**Transport.** North Data is fetched only through Massive: the residential
proxy first (Chrome-consistent headers; a sticky exit per `session_key`,
15 minutes, retried once with a fresh exit after a proxy error), then
Massive Web Render when the proxy was blocked. Never from Cormorant's own
addresses: no `direct`, `browser` or `archive` tier and no alternative
sources. This also applies to `POST /v1/fetch` of these hosts. `allow_paid:
false` (or a key without paid fetches) answers `error: paid_tier_required`.
A cluster-wide throttle covers the whole site (`.com` and `.de` together;
suggest, profile and graph): by default at most 2 requests in flight and
1 s between starts. robots.txt is read through the proxy and applies to
profile pages, `suggest.json` and `data.json`.

**Cache.** Profile pages: served for 24 h, then revalidated with a conditional
request (`fetch.revalidated`), kept 7 days. Suggest and graph answers: 24 h.
Challenge pages, thin pages (< 200 characters of text) and pages that do not
parse are never kept. `cache_ttl_s` sets the maximum acceptable age (`0`
bypasses the cache).

| Endpoint | Request | Response |
|---|---|---|
| `POST /v1/northdata/suggest` | `query`, `country?`, `session_key?` | `items[{name, url, type}]` (companies only, absolute URLs), `fetch` |
| `POST /v1/northdata/discover` | `company_name`, `headquarters?`, `country?`, `session_key?`, `min_similarity?` (70), `search_fallback?` (true) | `url`, `query`, `score` (0..1), `name_similarity` (0..100 or null), `discovery_method` (`suggest`/`search`), `candidates[≤10 {url, name, score, query, title}]`, `suggest_fetch` |
| `POST /v1/northdata/profile` | `url` **or** `company_name` (+ `country`, `headquarters`), `session_key?`, `country?`, `include_graph?` (true), `min_similarity?`, `search_fallback?` | `profile` (below), `fetch`, `discovery` (when discovered), `graph_fetch`, `graph_error` |
| `POST /v1/northdata/graph` | `entity_id`, `session_key?`, `country?` | `entity_id`, `company_connections`, `person_connections`, `other_connections`, `fetch` |

Every request also takes `cache_ttl_s`, `allow_paid` (default true) and
`project_ref`. Every response has `op`, `error`, `retry_after_s`, `cost_usd`
(null for external customers), `billing_category` (`northdata` or
`northdata_cached`) and `request_id`. `country` is an ISO code (`CH`) or an
English/German country name (`Switzerland`); unknown names are ignored. It
filters suggest (`countries=`) and geotargets the Massive exit.

```json
{
  "op": "profile",
  "profile": {
    "name": "Novartis Pharma AG", "legal_form": "AG", "register_id": "BR 925942480",
    "court": null, "status": "Active", "address": "Lichtstrasse 35, 4056 Basel, Switzerland",
    "city": "4056 Basel", "country": "Switzerland", "path_country": "CH",
    "entity_id": "…", "subsidiaries": [{"name": "Novartis International AG",
      "relation": "subsidiaries", "url": "/Novartis%20International%20AG",
      "absolute_url": "https://www.northdata.com/Novartis%20International%20AG"}],
    "source_url": "https://www.northdata.com/Novartis%20Pharma%20AG,%20Basel,%20CH/BR%20925942480",
    "discovery_method": "suggest", "discovery_score": 1.0, "name_similarity": 100.0,
    "fetch_tier": "proxy", "parser_version": "3", "provider": "northdata"
  },
  "fetch": {
    "url": "https://www.northdata.com/…", "final_url": "https://www.northdata.com/…",
    "status": 200, "tier": "proxy", "body_url": "/v1/bodies/9f2c…", "sha256": "9f2c…",
    "body_kind": "origin", "retrieved_at": "2026-10-19T12:00:00+00:00", "cached": false,
    "revalidated": false, "attempts": [{"tier": "proxy", "status": 200, "reason": "ok", "ms": 2100}],
    "cost_usd": 0.0004, "parser_version": "3"
  },
  "discovery": {"url": "https://www.northdata.com/…", "query": "Novartis Pharma AG",
    "score": 1.0, "name_similarity": 100.0, "discovery_method": "suggest", "candidates": []},
  "graph_fetch": null, "graph_error": "no_entity_id",
  "error": null, "retry_after_s": null, "cost_usd": 0.0006,
  "billing_category": "northdata", "request_id": "req_…"
}
```

**Profile fields** (`parser_version` `3`; parses HTML and the Markdown Web
Render sometimes returns): `name`, `legal_form` (from the company's own name
or legal-form block), `register_id`, `court`, `status`, `address`, `city`,
`country` (free text, not normalised), `path_country` (ISO code in the URL),
`entity_id`, `lei`, `industry`, `former_names[]`, `former_addresses[]`,
`corporate_purpose`, `additional_info`, `history[{category, date,
description, text, url}]`, `publications[{date, summary, company_name}]`,
`mentions[…]`, `network_graph {company_connections, person_connections,
other_connections}` (with the graph), `officers[{name, role, since, until,
url, absolute_url}]`, `shareholders`/`subsidiaries`/`parent_companies`/`related`
`[{name, relation, url, absolute_url}]`, `financials` (always null),
`source_url`, `discovery_query`, `discovery_score` (0..1),
`discovery_method`, `name_similarity` (0..100), `scraped_at`, `fetch_tier`
(`proxy` or `web_render`), `parser_version`, `provider` (`northdata`).
Graph merge: graph companies not already listed become `related` (relation
`related`); graph persons become `officers` with role `person`; parents and
subsidiaries are never invented.

**Discovery.** Suggest first: company items only; the one with the highest
name similarity (Jaro-Winkler + Levenshtein on names without legal
suffixes, 0..100) wins when it reaches `min_similarity`; `score` = similarity
/ 100; no first-result fallback. Otherwise, with `search_fallback`,
Cormorant's own search with up to four queries (`northdata "{name}" {city
cc}`, `site:northdata.com OR site:northdata.de "{name}" {city or cc}`,
`northdata "{name}" {cc}`, `northdata {name}`), 5 results each, profile URLs
scored 0..1 (name tokens in the path and title, city, country, host, register
in the path); it stops at ≥ 0.75 and rejects < 0.35. Suggest candidates carry
the name similarity (0..100) as `score`, search candidates the URL score.

| `error` | Meaning |
|---|---|
| `not_found` | no profile (404/410, or discovery found nothing) |
| `blocked` | every Massive tier was blocked (challenge, 403/451, content policy); with `retry_after_s` while the site is skipped |
| `rate_limited` | North Data answered 429; retry after `retry_after_s` |
| `all_tiers_failed` | every tier failed for another reason (network, 5xx, not JSON) |
| `parse_failed` | the page is not a company profile (e.g. a search page); not cached |
| `invalid_identifier` | `url` is not a North Data profile URL, or `entity_id` is malformed |
| `paid_tier_required` | `allow_paid: false` or the key has no paid fetches: North Data needs Massive |
| `budget_exhausted` | the project's monthly budget leaves nothing for Massive |
| `robots_disallowed` | robots.txt disallows the URL for `CormorantBot`/`*` |
| `timeout` | the request ran out of time (sync calls stop before the API's wait) |
| `politeness_timeout` | the site throttle stayed busy (transient; batch items retry) |

`graph_error` (profile only): why the graph is missing while the profile is
returned — an error code from the table, or `no_entity_id`.

### Batch: POST /v1/batch/northdata

```json
{
  "items": [
    {"op": "profile", "company_name": "Novartis Pharma AG", "country": "CH", "session_key": "CH0012005267"},
    {"op": "graph", "entity_id": "1234567"}
  ],
  "idempotency_key": "chipiron-run-2026-10-20-northdata-0001",
  "callback_url": "https://chipiron.example/hooks/cormorant"
}
```

`op` is `suggest`, `discover`, `profile` (default) or `graph`, plus the fields
of that endpoint (`422` when the field an operation needs is missing). Items
run like the single endpoints and their results have the same shape; the
site throttle sets the pace (a busy throttle puts an item back in the queue).

## GET /v1/usage?from=YYYY-MM-DD&to=YYYY-MM-DD

Usage of the caller's scope and what it costs. Inclusive dates (UTC), default:
current month to date, at most 400 days. Read from aggregates refreshed every
5 minutes, never from the request log, so it answers in the same time at any
volume.

**Who sees what.** A standard key (`cmr_…`) sees only its own requests. An
account admin key (`cma_…`) sees every key and project of its customer —
revoked keys included, and usage recorded before per-key attribution as
`"key": null`. A project without a customer is its own account. Nothing ever
crosses a customer boundary; a key or project you cannot see is `404`, like
one that does not exist.

**Parameters**

| Parameter | Meaning |
|---|---|
| `from`, `to` | Range of UTC days (inclusive). |
| `group_by` | Comma-separated, combinable: `key`, `project`, `category`, `day`, `hour` (hourly data: last 35 days). |
| `key` | Only these keys (comma-separated prefixes, e.g. `cmr_AbCdEf`). |
| `project` | Only these projects (comma-separated names). |
| `category` | Only these billing categories: `search`, `search_fallback`, `search_cached`, `fetch_own`, `fetch_paid`, `fetch_cached`, `company`, `company_cached`, `sec`, `sec_cached`, `northdata`, `northdata_cached`. |
| `format` | `csv` (same as `Accept: text/csv`). |

```json
{
  "project": "acme-crawler", "key": "cma_XyZaBc", "role": "account_admin",
  "scope": {"customer": "Acme", "projects": ["acme-crawler", "acme-search"], "keys": "all"},
  "from": "2026-10-01", "to": "2026-10-19",
  "as_of": "2026-10-19T11:05:02+00:00", "final": false, "currency": "USD",
  "totals": {"requests": 9120, "succeeded": 8800, "failed": 320, "success_rate": 0.9649,
             "cached": 1200, "cache_hit_rate": 0.1316,
             "errors": {"http_404": 210, "all_tiers_failed": 110},
             "latency_ms_p50": 900, "latency_ms_p95": 7400, "bytes": 812345678,
             "paid_requests": 90, "paid_share": 0.0113, "charge": "15.96",
             "billable": {"fetch_own": {"quantity": "8710", "included": "1000",
                                        "billable": "7710", "amount": "15.42"},
                          "fetch_paid": {"quantity": "90", "included": "0",
                                         "billable": "90", "amount": "0.36"},
                          "paid_gb": {"quantity": null, "included": null,
                                      "billable": null, "amount": "0.18"}}},
  "search": {"requests": 1200, "cached": 300, "cache_hit_rate": 0.25, "succeeded": 1198,
             "success_rate": 0.998, "bytes": 0, "paid_bytes": 0,
             "latency_ms_p50": 1100, "latency_ms_p95": 3900, "fallbacks": 60,
             "fallback_rate": 0.0667, "providers": {"searxng": {"requests": 840, ...}, "brave": {...}}},
  "fetch": {"requests": 7920, "cached": 900, "tiers": {"direct": {...}, "proxy": {...}},
            "errors": {"http_404": 210}, ...},
  "northdata": {},  // North Data requests (kind northdata), same fields; {} when none
  "group_by": ["key"],
  "groups": [{"key": "cmr_AbCdEf", "key_label": "crawler", "key_role": "standard",
              "key_status": "active", "project": "acme-crawler", "requests": 9000, ...,
              "charge": "15.70", "billable": {...}}],
  "keys": [{"prefix": "cmr_AbCdEf", "label": "crawler", "role": "standard",
            "project": "acme-crawler", "active": true, "requests": 9000, "cached": 1150,
            "succeeded": 8690, "search_requests": 1100, "fetch_requests": 7900, "company_requests": 0,
            "northdata_requests": 0,
            "paid_bytes": 9123456, "charge": "15.70"}],
  "account": [{"period": "2026-10", "from": "2026-10-01", "to": "2026-10-19", "currency": "USD",
               "free": true, "usage_charges": "15.96", "account_level_amount": "-15.96",
               "account_lines": [{"kind": "free", "description": "Free plan: 100 % discount",
                                  "amount": "-15.96"}],
               "list_total": "15.96", "net_total": "0.00"}]
}
```

- **Metrics** (in `totals` and every group): `requests` (cache hits
  included), `succeeded`, `failed`, `cached`, `errors` (failures by error
  code, top 50), `latency_ms_p50`/`latency_ms_p95` of uncached requests
  (estimated from a latency histogram with bounds 100, 250, 500 ms, 1, 2, 5,
  10, 20, 30, 60 and 120 s — exact to the bucket), `bytes`, `paid_requests`
  and `paid_share` (fetches that needed the paid premium tiers, of all
  uncached fetches).
- **Charges** (projects with a customer): `charge` is what the group costs at
  your plan's prices in your invoice currency, as a decimal string; `billable`
  lists each price-plan meter with the measured `quantity`, the `included`
  units it used (its pro-rata share of the plan's included units of the
  month), `billable` and `amount`. Meters billed on provider cost show only the
  amount. The period is rated exactly like the invoice and each usage line is
  split to the cent, first per key, then per day and category: **any grouping
  adds up to the same total**, and the keys add up to the usage lines of the
  estimate (`GET /v1/account`) or invoice. Base fee, minimum, contract
  discount and the free-plan credit are account-level lines: account admin
  keys see them per period in `account`. Free accounts see list-price charges
  and the 100 % `free` line.
- **Provider cost** — `cost_usd` and the project `budget` (Cormorant's spend
  with Brave and Massive) appear only for projects without an external
  customer (internal use). Customers see what they pay, never what Cormorant
  pays.
- **Freshness**: `as_of` is the time up to which every request is included;
  `final: true` (top level, and per group with `group_by=day`) means the days
  are closed and will not change (a day becomes final two hours after it
  ends).
- `search` and `fetch` keep the per-kind blocks of earlier versions (now for
  the caller's scope), with `providers`/`tiers` breakdowns. With `key` in
  `group_by`, `keys` keeps the earlier per-key list as well.

**CSV.** `Accept: text/csv` (or `format=csv`) returns one row per group — or
one row of totals without `group_by` — with the dimensions, the metrics,
`charge`, the meters as `<meter>_quantity`, `<meter>_included`,
`<meter>_billable`, `<meter>_amount` and `errors` as `code=n;code=n`.

```sh
curl -sS "$CORMORANT_URL/v1/usage?from=2026-10-01&to=2026-10-31&group_by=key,day" \
  -H "Authorization: Bearer $CORMORANT_API_KEY" -H "Accept: text/csv" -o usage.csv
```

## GET /v1/usage/timeseries

The same metrics per `granularity=hour` (last 35 days; default range: today)
or `granularity=day` (default: month to date), one point per bucket up to now
— buckets without usage are zero, so the points can be charted directly.
`group_by` (`key`, `project`, `category`, combinable) gives one series per
value; `key`, `project` and `category` filter as for `GET /v1/usage`. Same
scope rules, same charges (an hour's charge is its share of the day's), CSV
with `Accept: text/csv`.

```json
{"granularity": "hour", "from": "2026-10-19", "to": "2026-10-19", "as_of": "2026-10-19T11:05:02+00:00",
 "final": false, "currency": "USD", "group_by": ["key"],
 "series": [{"key": "cmr_AbCdEf", "key_label": "crawler", "key_role": "standard",
             "key_status": "active", "project": "acme-crawler",
             "points": [{"t": "2026-10-19T00:00:00+00:00", "final": false, "requests": 410,
                         "succeeded": 402, "failed": 8, "latency_ms_p95": 6100,
                         "paid_share": 0.01, "charge": "0.82", ...}, ...]}]}
```

## GET /v1/usage/domains

The busiest fetched domains of the caller's keys over the range (kept 90
days): `requests`, `succeeded`, `success_rate`, `paid_requests` and
`paid_share` per domain, busiest first (`limit`, default 50, at most 500).
`group_by=key` returns a top list per key. Built from each key's 100 busiest
domains per day; usage recorded before per-key attribution is not included.

```json
{"from": "2026-10-01", "to": "2026-10-19",
 "domains": [{"domain": "www.example.com", "requests": 1200, "succeeded": 1150,
              "success_rate": 0.9583, "paid_requests": 40, "paid_share": 0.0333}]}
```

## GET /v1/requests

The request ledger, newest first and in real time: one row per search, fetch
or batch item of the caller's scope (a standard key: its own requests; an
account admin key: the whole customer). Kept as long as the request log (90
days; default range: the last 30 days).

| Parameter | Meaning |
|---|---|
| `from`, `to` | Range of UTC days. |
| `key`, `project`, `category` | As for `GET /v1/usage`. |
| `kind` | `search` or `fetch`. |
| `outcome` | `succeeded` or `failed`. |
| `error`, `domain`, `project_ref`, `job_id`, `cached` | Exact matches. |
| `limit` | Rows per page, 1–1000 (default 100). |
| `cursor` | `next_cursor` of the previous page. |

```json
{"from": "2026-09-20", "to": "2026-10-19", "as_of": "2026-10-19T11:05:02+00:00",
 "currency": "USD", "limit": 100, "next_cursor": "MjAyNi0xMC0xOVQxMTowMjoxMyswMDowMHw5ODc2",
 "requests": [{"request_id": "req_6b1f2c0e9a7d4b5c8e3f1a2b3c4d5e6f",
               "time": "2026-10-19T11:02:13+00:00", "key": "cmr_AbCdEf", "key_role": "standard",
               "project": "acme-crawler", "project_ref": "company:4711", "job_id": null,
               "kind": "fetch", "domain": "www.example.com", "tier": "direct", "status": 200,
               "ok": true, "error": null, "latency_ms": 840, "cached": false, "bytes": 48211,
               "paid": false, "category": "fetch_own", "charge": "0.001842"}]}
```

Page with `cursor` until `next_cursor` is `null`; the next page is also in
the `Link: <…>; rel="next"` header (and `X-Next-Cursor`). Pages are keyset
pages, so they stay fast and consistent however deep you go, while new
requests arrive on top. `charge` is the request's share of its key's
charges for that day and category (their average price, 6 decimals); it is
`null` until the request is aggregated (within 5 minutes) and settles when
the day is final. `GET /v1/requests/{request_id}` returns one request with
`reason` and the tiers tried (`attempts`); another key's (standard keys) or
another customer's request is `404`. CSV with `Accept: text/csv` (one page;
the next cursor is in `X-Next-Cursor`).

## GET /v1/quota

Every limit that applies to the caller this month, each with `limit`,
`used`, `reserved`, `remaining`, `resets_at`, `projected_end_of_period`
(linear extrapolation to the end of the month) and `status` (`ok`,
`warning` from 80 %, `exhausted`): the key's spend cap
(`monthly_spend`, money as decimal strings), request quotas
(`monthly_requests.<category>`) and rate limit; the project's rate limit
(and provider budget, for projects without an external customer). An account
admin key gets every active key of the account and, for a customer, the plan's
included units (`included.<meter>`).

```json
{"key": "cmr_AbCdEf", "role": "standard", "period": "2026-10",
 "resets_at": "2026-11-01T00:00:00+00:00", "as_of": "2026-10-19T11:05:02+00:00",
 "usage_refreshed_at": "2026-10-19T11:05:00+00:00",
 "keys": [{"key": "cmr_AbCdEf", "key_label": "crawler", "key_role": "standard",
           "project": "acme-crawler", "is_self": true, "allowed_categories": null,
           "usage": {"requests": {"search": 120, "fetch": 81234, "fetch_paid": 412},
                     "spend": "38.12", "currency": "USD"},
           "limits": [{"name": "monthly_requests.fetch", "unit": "requests", "limit": 100000,
                       "used": 81234, "reserved": 0, "remaining": 18766,
                       "resets_at": "2026-11-01T00:00:00+00:00",
                       "projected_end_of_period": 132550, "status": "warning"},
                      {"name": "monthly_spend", "unit": "USD", "limit": "50.00", "used": "37.90",
                       "reserved": "0.22", "remaining": "11.88", "status": "ok", ...}]}],
 "project": {"project": "acme-crawler",
             "limits": [{"name": "rate_limit", "unit": "requests/s", "limit": 200, "used": 3, ...}]}}
```

`reserved` is the spend estimated since the last aggregation (or, for the
provider budget, spend held for calls in flight). `events` lists the
thresholds crossed this month in your scope (see *Alerts* below).

## Alerts: POST, GET, DELETE /v1/alerts

A webhook receives an event whenever something in its scope crosses **50, 80
or 100 %** of a limit or allowance in a month (each threshold fires once per
month): a key's spend cap or request quota, a project's provider budget
(projects without an external customer) or the plan's included units of the
account. The same events are listed in `GET /v1/quota` (`events`).

```json
POST /v1/alerts
{"url": "https://hooks.example.com/cormorant", "scope": "customer", "thresholds": [80, 100]}

201 {"id": 7, "url": "https://hooks.example.com/cormorant", "scope": "customer", "key": null,
     "project": null, "thresholds": [80, 100], "created_at": "2026-10-19T11:00:00+00:00",
     "last_delivery_at": null, "last_status": null, "secret": "whsec_9f2c…"}
```

- `scope`: `key` (one key; `key` = its prefix, default the calling key),
  `project` (`project` = name; the project and its keys) or `customer` (the
  whole account). A standard key can only watch itself (`403
  account_admin_required` otherwise); an account admin key anything in its
  account. Default: `key` for standard keys, `customer` for admin keys.
- `thresholds`: any of `50`, `80`, `100` (default all).
- `url` must be `https` and resolve to a public address; at most 20 webhooks
  per account. `secret` is returned **only once**.
- `GET /v1/alerts` lists the webhooks you can see (never secrets) with
  `last_delivery_at` and `last_status`; `DELETE /v1/alerts/{id}` removes one.

**Deliveries** are `POST <url>` with the event as JSON:

```json
{"id": "evt_42", "created_at": "2026-10-19T11:05:03+00:00", "type": "usage.threshold",
 "scope": "key", "customer": "Acme", "project": "acme-crawler", "key": "cmr_AbCdEf",
 "key_role": "standard",
 "limit": {"name": "monthly_spend", "unit": "USD", "limit": "50.00", "used": "40.12",
           "threshold": 80, "period": "2026-10", "resets_at": "2026-11-01T00:00:00+00:00"}}
```

Headers: `X-Cormorant-Event: usage.threshold`, `X-Cormorant-Delivery: dlv_…`
and `X-Cormorant-Signature: t=1760871903,v1=5257a8…`, where `v1` is the hex
HMAC-SHA256 of `"<t>.<raw body>"` with your `secret`. Verify it in constant
time and reject a `t` older than a few minutes:

```python
import hashlib, hmac, time


def verify(secret: str, body: bytes, header: str) -> bool:
    parts = dict(p.split("=", 1) for p in header.split(","))
    mac = hmac.new(secret.encode(), f"{parts['t']}.".encode() + body, hashlib.sha256)
    fresh = abs(time.time() - int(parts["t"])) < 300
    return fresh and hmac.compare_digest(mac.hexdigest(), parts["v1"])
```

Any `2xx` answer counts as delivered; other answers and network errors are
retried after 1, 5, 15, 60, 180, 360 and 720 minutes (8 attempts). Redirects
are not followed, and each delivery connects to the address that passed the
check (no DNS rebinding). Events are evaluated with the usage aggregates (every 5
minutes). Use `id` to ignore duplicates.

## GET /v1/keys and GET /v1/keys/self

The keys the caller may see — by prefix, label and role, never the key itself:
every key of the customer's projects for an account admin key (revoked keys
included), only itself for a standard key. The prefix always shows the role:
`cma_…` is an account admin key, `cmr_…` a standard key.

```json
{"role": "account_admin",
 "keys": [{"prefix": "cma_XyZaBc", "label": "ops", "role": "account_admin",
           "project": "acme-crawler", "status": "active",
           "created_at": "2026-09-01T08:00:00+00:00", "revoked_at": null,
           "last_used_at": "2026-10-19T11:02:00+00:00", "is_self": true},
          {"prefix": "cmr_AbCdEf", "label": "crawler", "role": "standard", "status": "revoked", ...}]}
```

Each key also has `limits` (`monthly_spend_cap`, `currency`, `monthly_requests`,
`rate_limit_rps`, `allowed_categories`; `null` without limits).
`GET /v1/keys/self` returns the calling key in the same shape plus its use
this month (`usage`) and quota state (`quota`, the entries of `GET /v1/quota`).

## Account and invoices

Keys belong to a project; a project may belong to a **customer** (the billed
account). These endpoints show the key's own project usage plus the plan,
current-period estimate and issued invoices of that customer — never another
customer's data, never drafts. Money is a decimal **string** in the invoice
currency (USD, CHF or EUR). Usage is aggregated every 5 minutes (`usage_refreshed_at`).
See [billing.md](/docs/billing.md) for pricing rules.

### GET /v1/account

```json
{
  "project": {"id": 7, "name": "acme-crawler"},
  "period": {"period": "2026-10", "from": "2026-10-01", "to": "2026-10-31"},
  "usage_refreshed_at": "2026-10-19T11:00:04+00:00",
  "project_usage": {"fetch_own": {"requests": 9120, "succeeded": 8800, "bytes": 812345678,
                    "bytes_ok": 800000000, "paid_bytes": 0, "paid_bytes_ok": 0,
                    "cost_usd": "0", "cost_ok_usd": "0"}, "search": {...}},
  // cost_usd / cost_ok_usd: null for projects of an external customer
  "customer": {"name": "Acme", "company_name": "Acme Research AG", "currency": "USD",
               "status": "active", "free": true},
  "customer_usage": {...},
  "plan": {"id": 1, "code": "standard-usd", "version": 1, "name": "Standard (USD)",
           "currency": "USD", "free": false},
  "estimate": {"period": "2026-10", "currency": "USD",
               "lines": [{"kind": "usage", "component": "fetch_own",
                          "description": "Page fetches (1,000 included)", "quantity": "8800",
                          "unit": "fetch", "unit_price": null, "amount": "15.60",
                          "detail": {"included": "1000", "billable": "7800",
                                     "tiers": [{"from": "0", "to": "100000", "quantity": "7800",
                                                "unit_price": "0.002", "amount": "15.60"}]}},
                         {"kind": "free", "description": "Free plan: 100 % discount", "amount": "-15.60", ...}],
               "subtotal": "15.60", "discount_total": "15.60", "net_total": "0.00",
               "vat_rate": "8.1", "vat_amount": "0.00", "total": "0.00", "free": true,
               "list_total": "15.60"}
}
```

`keys` breaks the period down per API key. An account admin key (`cma_…`)
sees every key of the customer's projects (or of this project without a
customer); an additional key (`cmr_…`) sees only itself. Keys appear by
`prefix`, `label` and `role`, never ids or secrets, with its usage by category and `amount`, its share of
the usage charges at list price. Each usage line of the estimate is split
between keys in proportion to their share of that meter (to the cent), so the
amounts add up to the usage lines; base fee, minimum, discounts and the free
credit stay on the account. A row with `"prefix": null` is usage recorded
before per-key attribution.

```json
"keys": [{"prefix": "cma_XyZaBc", "label": "billing", "role": "account_admin", "project": "acme-crawler",
          "active": true, "requests": 9120, "succeeded": 8800,
          "usage": {"fetch_own": {"requests": 9120, ...}}, "amount": "15.60"}]
```

Projects without a customer get `"customer": null, "plan": null, "estimate": null`
(and `amount: null` per key).
Categories: `search`, `search_fallback`, `search_cached`, `fetch_own`,
`fetch_paid`, `fetch_cached`, `company`, `company_cached`, `sec`, `sec_cached`, `northdata`,
`northdata_cached`.

### GET /v1/account/invoices

`{"invoices": [{"number": "CORM-2026-0001", "status": "issued|paid|void", "period": "2026-09",
"period_start", "period_end", "currency", "plan_label", "free", "subtotal", "discount_total",
"net_total", "vat_rate", "vat_amount", "total", "vat_note", "issued_at", "due_at", "paid_at",
"voided_at", "pdf_url": "/v1/account/invoices/CORM-2026-0001/pdf"}]}`

### GET /v1/account/invoices/{number} and /pdf

The invoice with its lines (and tier breakdown), or the PDF. Another
customer's number (or a draft) is `404`.

## Company data

The company register: one source-attributed record (`company/v1`) per
company Cormorant has been asked about, kept and versioned. Sources are
official registers, GLEIF and other free, legitimate sources
(`GET /v1/companies/sources`). Every data call is metered as **`company`**
(billing category and meter `company`; one request per call or batch item).
`GET /v1/companies/sources`, `GET /v1/schemas/company` and watch management
are not metered.

| Endpoint | Purpose |
|---|---|
| `GET /v1/companies/search?q=&country=&identifier=&status=&legal_form=&limit=&cursor=&live=` | Register matches (trigram/full-text over current, former and trading names; exact on `identifier`) plus live hits from the sources covering `country` on the first page. `match_score` 0–1, `in_register`, `ref` |
| `GET /v1/companies/{id}` | The record. `id` = `cmp_…` or `scheme:value`. `max_age`, `refresh`, `include=people,ownership,filings,financials,events,claims`, `sources=` |
| `GET /v1/companies/{id}/history?field=` | Claims timeline: every value each source asserted, with `observed_at`, `valid_from`, `valid_to`, `current` |
| `GET /v1/companies/{id}/{events,people,ownership,relationships,filings,financials,sources}` | Sub-resources with paging (`limit`, `cursor`); `people` accepts `include_former` |
| `POST /v1/companies/resolve` | `{name, country, city, identifiers}` → `match` (confident only) and ranked `candidates` |
| `POST /v1/companies/batch`, `POST /v1/companies/batch/csv` | Job of kind `company`: `get`, `refresh` or `resolve` per item; JSON or CSV in; results via `GET /v1/jobs/{job_id}` or `GET /v1/companies/batch/{job_id}.csv` |
| `GET /v1/companies/changes?since=&cursor=&country=` | Change feed of the register (company and changed top-level fields), oldest first |
| `POST/GET /v1/companies/watches`, `DELETE /v1/companies/watches/{id}` | Signed `company.changed` webhooks for a list of companies |
| `GET /v1/companies/sources` | Sources: countries, schemes, capabilities, precedence, licence URL, freshness classes, live health |
| `GET /v1/schemas/company` | JSON Schema (2020-12) of `company/v1` |

**Identifiers** are written `scheme:value` and normalised (spaces, dots and
case do not matter; check digits are verified): `ch_uid`
(`CHE-101.329.561`), `ch_chid`, `ch_ehraid`, `lei`, `us_cik`, `uk_crn`,
`de_register` (`München:HRB:6684`), `fr_siren`, `fr_siret`, `no_orgnr`,
`dk_cvr`, `fi_ytunnus`, `ee_regcode`, `cz_ico`, `pl_krs`, `be_kbo`, `au_abn`,
`au_acn`, `nz_nzbn`, `jp_corporate_number`, `br_cnpj`, `sg_uen`, `ca_corp`,
`vat` (`DE129273398`, `CHE101329561`), `isin`, `ticker` (`XNAS:AAPL`),
`wikidata`. An invalid identifier is `422 invalid_identifier`.

**Refresh on request.** A request answers from the register. Each source's
data about the company is re-fetched first when it is older than the
source's freshness class for that field group (e.g. registry core 24 h,
people 7 days) or than `max_age` (seconds); `refresh=true` re-fetches
everything. The answer's `refresh` lists per source and group what was
`refreshed`, `fresh`, `failed` (error class; the register still answers),
`not_found` or `in_progress`. A company not yet in the register is looked
up in the sources that know the identifier's scheme and added; unknown
everywhere is `404 company_not_found`. Companies nobody asks for are never
re-scraped; official change feeds (e.g. GLEIF's daily updates) keep the ones
in the register current.

**Provenance.** Every value is a claim of a source; the record shows the
resolved value of each field, `provenance` (source, `observed_at`,
`confirmed_at`, `sha256` of the raw document, `conflict`), `quality`
(`completeness`, `conflicts` where sources disagree) and `sources[]` (raw
documents with `body_url` — downloadable with `GET /v1/bodies/{sha256}` once
you have received the record). `include=claims` adds every current claim.
Precedence when sources disagree: official register > official gazette >
GLEIF > regulatory filings > crosswalks > commercial sources. A merged
record's `cmp_` id keeps working and answers with the surviving record and
`redirected_from`.

```sh
curl -s "https://api.cormorant.ch/v1/companies/ch_uid:CHE-101.329.561?include=ownership" \
  -H "Authorization: Bearer $CORMORANT_API_KEY"
```

```json
{
  "schema_version": "company/v1",
  "id": "cmp_01JA2B3C4D5E6F7G8H9J0KMNPQ",
  "identifiers": [{"scheme": "ch_uid", "value": "CHE-101.329.561", "issuer": "Federal Statistical Office (BFS)", "status": "active", "parts": null, "sources": ["gleif"]},
                  {"scheme": "lei", "value": "BFM8T61CT2L1QCEMIK50", "issuer": "GLEIF", "status": "active", "parts": null, "sources": ["gleif"]}],
  "names": {"legal": "UBS AG", "previous": [{"name": "SBC AG", "lang": "en"}], "trading": [], "translations": []},
  "legal_form": {"local_code": "AG", "local_name": "Company limited by shares", "elf_code": "MVII", "category": "corporation"},
  "status": {"value": "active", "local_value": "ACTIVE"},
  "jurisdiction": {"country": "CH", "subdivision": null, "register": {"authority": "RA000549", "court": null, "entity_id": "CHE-101.329.561"}},
  "addresses": [{"type": "registered", "lines": ["Bahnhofstrasse 45"], "street": "Bahnhofstrasse 45", "house_number": null, "postal_code": "8001", "city": "Zurich", "region": "CH-ZH", "country": "CH"}],
  "dates": {"incorporated": "1978-02-28", "registered": null, "deregistered": null, "fiscal_year_end": null},
  "ownership": [{"type": "direct_parent", "holder": {"name": "UBS Group AG", "identifiers": [{"scheme": "lei", "value": "549300SZJ9VS8SGXAN81"}], "country": "CH"}, "source": "gleif"}],
  "provenance": {"names.legal": {"source": "gleif", "observed_at": "2026-10-19T08:00:00+00:00", "confirmed_at": "2026-10-19T08:00:00+00:00", "sha256": "9f86…", "conflict": false}},
  "sources": [{"source": "gleif", "url": "https://api.gleif.org/…/BFM8T61CT2L1QCEMIK50", "retrieved_at": "2026-10-19T08:00:00+00:00", "sha256": "9f86…", "body_url": "/v1/bodies/9f86…", "content_type": "application/vnd.api+json", "licence": "https://www.gleif.org/en/meta/lei-data-terms-of-use", "parser_version": "gleif/1"}],
  "quality": {"completeness": 0.778, "conflicts": []},
  "refresh": [{"source": "gleif", "group": "core", "status": "fresh"}, {"source": "gleif", "group": "ownership", "status": "refreshed", "at": "2026-10-19T08:00:01+00:00"}],
  "redirected_from": null,
  "updated_at": "2026-10-19T08:00:01+00:00"
}
```

**Batch.** `POST /v1/companies/batch` takes `{action, items: [{ref} | {name,
country, city, identifiers}], include, max_age, idempotency_key,
callback_url}`; `POST /v1/companies/batch/csv` takes a CSV body (header
`ref,name,country,city,identifiers`, identifiers separated by `;`) with the
same options as query parameters. Item results: `{company_id, record}` for
`get`/`refresh` (the record as above), `{match, candidates, company_id}` for
`resolve`, or `{error}` (`company_not_found`, `invalid_identifier`).

**Change feed and watches.** `GET /v1/companies/changes` returns `{items:
[{seq, company_id, kind: created|updated|merged|split, fields, country,
changed_at}], next_cursor}`; start with `since` (ISO 8601) and continue with
`cursor`. `POST /v1/companies/watches` with `{companies: [ids or
scheme:value], url}` (HTTPS, public address) returns the watch with its
`secret` once (at most 20 watches per project, 1000 companies each). Changes
of watched companies are POSTed every minute as `{event: "company.changed",
id, watch_id, changes}` with `X-Cormorant-Event`, `X-Cormorant-Delivery` and
`X-Cormorant-Signature: t=<unix>,v1=<hex>` (HMAC-SHA256 of `"<t>.<raw
body>"`, as for usage alerts). Any 2xx is delivered; otherwise retried after
1, 5, 15, 60 and 180 minutes, then hourly, without skipping changes. A
watched record merged into another is followed.

**Errors**: `404 company_not_found`, `422 invalid_identifier`,
`422 invalid_parameter` (e.g. unknown `include`), `422 invalid_item` (batch);
otherwise as for every endpoint (`401`, `403`, `429`).

## Service status

Public status of the service for your monitoring, and the page at
[`/status`](/status). **No authentication** (an `Authorization` header is
accepted and ignored), CORS-enabled, refreshed at most every 15 s and shared
by all callers (`Cache-Control: public, max-age=15`, `ETag` /
`If-None-Match` → `304`). Limited to 120 requests per minute per client IP
(`429` with `Retry-After`). The status never contains host names, addresses,
instance URLs, project names, keys or internal error strings — only component
ids, statuses, reason codes and aggregate numbers.

The status endpoints answer **`200` whatever the service status**: the
status is in the body. Alert on the body (`status` is not `operational`), and
treat a timeout, a connection error or any other HTTP code as "Cormorant is
unreachable" — a status endpoint that answered `503` during an outage could
not be told apart from a broken proxy or network in between.

| Endpoint | Returns |
|---|---|
| `GET /v1/status` | overall `status`, every component with `reasons` and `metrics`, incidents and maintenance |
| `GET /v1/status/summary` | only `status`, `description`, `updated_at` (for uptime monitors) |
| `GET /v1/status/components/{id}` | one component plus the incidents and maintenance naming it (`404` for an id not in this deployment) |
| `GET /v1/status/history?hours=24` | worst status per 5 minutes (per hour beyond 24 h, up to 168 h) and uptime per component; overall as `_overall` |

```json
{
  "status": "degraded",
  "description": "Degraded performance",
  "updated_at": "2026-10-19T09:41:15+00:00",
  "window_s": 600,
  "components": [
    {"id": "api", "name": "API", "description": "…", "status": "operational", "reasons": [], "metrics": {}, "group": null},
    {"id": "fetch", "name": "Fetch", "description": "…", "status": "degraded", "reasons": ["low_success_rate"], "metrics": {}, "group": null},
    {"id": "company_gleif", "name": "Company data: GLEIF", "description": "…", "status": "operational", "reasons": [], "metrics": {}, "group": "company_data"}
  ],
  "incidents": [
    {"id": "inc_3f9a1c0b7e", "title": "Elevated fetch failures", "status": "identified",
     "impact": "degraded", "components": ["fetch"], "created_at": "…", "updated_at": "…",
     "resolved_at": null, "updates": [{"at": "…", "status": "identified", "message": "…"}]}
  ],
  "maintenance": []
}
```

**Statuses**, best to worst: `operational`, `maintenance`, `degraded`,
`partial_outage`, `major_outage`. The overall `status` is the worst component
status, except that a major outage of a component other than `api`,
`search`, `fetch` or `storage` counts as a partial outage, and it is
`maintenance` when the only components not operational are under announced
maintenance. An unresolved incident shows its components at least at its
`impact`. Components not part of the deployment are not listed (`archive`
when archived copies are disabled, `paid_tiers` when no paid tier is
configured, `sec` while no SEC identity is configured) — do not alert on a
missing component.

| Component | Metrics (rates 0–1 over the last `window_s` = 600 s; `null` = too few requests) | Not operational when |
|---|---|---|
| `api` | — | database or cache unreachable (major) |
| `search` | `success_rate`, `requests`, `engines_healthy_share`, `fallback_available`, `fallback_share`, `fallback_success_rate`, `workers_alive`, `queue_lag_s` | success rate of uncached searches < 0.95 / 0.80 / 0.50 (degraded / partial / major); healthy engines < 50 % (degraded) or none (partial with the paid fallback, major without); no worker; queue lag |
| `fetch` | `success_rate`, `requests`, `latency_p95_ms`, `workers_alive`, `sync_queue_backlog`, `sync_queue_lag_s`, `batch_queue_backlog` | success rate of uncached fetches < 0.85 / 0.60 / 0.30 (4xx answers of the site, robots.txt, invalid or blocked URLs, too large, PDF not accepted and deferred are not failures); p95 > 60 s (degraded); no worker (major); queue lag |
| `browser_rendering` | `success_rate`, `attempts`, `workers_alive`, `sync_queue_backlog`, `sync_queue_lag_s`, `batch_queue_backlog` | browser attempts failing for infrastructure reasons: success < 0.80 / 0.50 / 0.10; no browser worker (major) |
| `archive` | `success_rate`, `attempts` | archive unreachable or erroring: success < 0.80 / 0.50 / 0.10 |
| `paid_tiers` | `proxy_success_rate`, `proxy_attempts`, `rendering_success_rate`, `rendering_attempts` (configured tiers only) | provider errors: success < 0.80 / 0.50 / 0.10 |
| `sec` | `success_rate`, `requests`, `throttle_wait_p95_ms` (p95 wait for a slot in the cluster-wide SEC rate limiter), `backoff_active`, `rate_limit_rps` | SEC backing off after 403/429/503 (degraded, `sec_backoff`); success rate of uncached SEC requests < 0.85 / 0.60 / 0.30 (4xx, invalid identifiers and not found are not failures); p95 wait > 5 s (degraded, `throttled`) |
| `batch_jobs` | `scheduler_last_run_s`, `workers_alive`, `running_jobs`, `oldest_waiting_job_s`, `queue_backlog` | scheduler last ran > 30 s (degraded) / > 120 s ago (partial); no batch worker (major); a job waiting to start > 15 min (degraded) / > 60 min (partial) |
| `storage` | `database_ok`, `cache_ok`, `object_storage_ok` (`null`: not configured) | database or cache unreachable (major); object storage unreachable (degraded) |
| `company_<source>` (group `company_data`, one per active company data source, e.g. `company_gleif`) | `success_rate`, `requests` (last hour of calls to the source; `null` below 10), `latency_p95_ms`, `parse_failure_rate`, `canary_ok`, `backoff_active`, `schema_drift` (fields that used to be filled and now come back empty), `change_feed_lag_s` | success < 0.85 / 0.50 / 0.20; canaries (known companies checked every 15 minutes) failing (partial); backoff after repeated failures, schema drift, parse failures > 10 % (degraded). Counts at most as `degraded` for the overall status |

Queue lag (`queue_lag_s`, `sync_queue_lag_s`: how long the oldest synchronous
request has waited for a worker) above 15 / 60 / 120 s is degraded / partial
/ major. Rates computed on fewer than 20 requests (10 tier attempts) are
reported but do not change a status. New metric keys may be added.

**Reason codes**: `no_workers`, `queue_lag`, `low_success_rate`,
`high_latency`, `engines_reduced`, `engines_unavailable`, `tier_errors`,
`scheduler_stalled`, `jobs_waiting`, `database_unreachable`,
`cache_unreachable`, `object_storage_unreachable`, `sec_backoff`,
`throttled`, `incident`, `maintenance`; company data sources also `canary_failed`, `source_backoff`,
`schema_drift`, `parse_failures`.

**Incidents and maintenance** are posted by the operator. `incidents` lists
unresolved incidents (`investigating`, `identified`, `monitoring`) and those
`resolved` within 7 days, newest first, each with its timeline (`updates`,
newest first). `maintenance` lists `scheduled` and `in_progress` windows and
those `completed` within 7 days, soonest first; an empty `components` list
means the whole service.

**History**: `GET /v1/status/history` returns, per component, `statuses`
(oldest first; `null` where nothing was recorded) and `uptime` over the
period, where a `partial_outage` bucket counts as 30 % downtime and a
`major_outage` bucket as 100 %. Kept 7 days.

## Health and metrics

- `GET /health` → `200 {"status":"ok"}` (process is up). For monitoring
  the service as a whole use [`GET /v1/status`](#service-status).
- `GET /ready` → `200` when PostgreSQL and Redis answer and at least one SearXNG
  instance is registered, else `503`; body lists the checks, never secrets.
- `GET /metrics` → Prometheus text (private network only): requests by kind,
  provider and outcome, cost, bytes, paid bytes, search fallbacks, latency
  histogram, queue lengths, SearXNG instance count — aggregated across all hosts.

## Examples

```sh
export CORMORANT_URL=https://api.cormorant.ch   CORMORANT_API_KEY=cmr_...
curl -s "$CORMORANT_URL/v1/search" -H "Authorization: Bearer $CORMORANT_API_KEY" \
  -H 'Content-Type: application/json' -d '{"query":"Holcim annual report 2025","country":"CH"}'

curl -s "$CORMORANT_URL/v1/fetch" -H "Authorization: Bearer $CORMORANT_API_KEY" \
  -H 'Content-Type: application/json' -d '{"url":"https://www.holcim.com/investors"}'

curl -s "$CORMORANT_URL/v1/batch/fetch" -H "Authorization: Bearer $CORMORANT_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"idempotency_key":"demo-1","items":[{"url":"https://example.com"},{"url":"https://example.org"}]}'
```

```python
import httpx

client = httpx.Client(
    base_url=CORMORANT_URL, headers={"Authorization": f"Bearer {KEY}"}, timeout=180
)
page = client.post("/v1/fetch", json={"url": "https://www.holcim.com/investors"}).json()
if page["error"] is None:
    text = page["text"]
```

Sync calls wait up to 150 s for the result (then `504` with the request id);
use the batch endpoints for bulk work.