# North Data via Cormorant

Cormorant fetches, parses and caches North Data (`northdata.com` /
`northdata.de`) company profiles for you: discovery of the profile URL from a
company name, the parsed profile with its register data, history, related
companies and officers, and North Data's network graph. You send JSON and
get structured JSON back, with the provenance of every page and the exact
bytes behind it.

This is the scraping path (the public pages); it does not use North Data's
paid API.

## Four endpoints

| You have | Call | You get |
|---|---|---|
| a name, want candidates | `POST /v1/northdata/suggest` | North Data's autocomplete, companies only |
| a name (+ country, headquarters) | `POST /v1/northdata/discover` | the best profile URL with its score |
| a URL, or a name to discover | `POST /v1/northdata/profile` | the parsed profile (+ network graph) |
| an `entity_id` | `POST /v1/northdata/graph` | the network graph alone |

Most clients only need **profile**: give it the company name, country and
headquarters and it discovers the URL, fetches the page, parses it and merges
the graph in one call.

```sh
curl -s https://api.cormorant.ch/v1/northdata/profile \
  -H "Authorization: Bearer $CORMORANT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"company_name": "Novartis Pharma AG", "country": "CH",
       "headquarters": "Basel, Switzerland", "session_key": "CH0012005267"}'
```

```python
import httpx

client = httpx.Client(
    base_url="https://api.cormorant.ch",
    headers={"Authorization": f"Bearer {key}"},
    timeout=180,
)
r = client.post(
    "/v1/northdata/profile",
    json={
        "url": "https://www.northdata.com/Unilever+plc",
        "session_key": "GB00B10RZP78",
    },
)
body = r.json()
if body["error"]:
    print("no profile:", body["error"], body["retry_after_s"])
else:
    profile = body["profile"]
    print(profile["name"], profile["register_id"], profile["city"])
    print("page bytes:", body["fetch"]["body_url"])
```

Suggest, discover and graph work the same way:

```sh
curl -s https://api.cormorant.ch/v1/northdata/suggest \
  -H "Authorization: Bearer $CORMORANT_API_KEY" -H "Content-Type: application/json" \
  -d '{"query": "Novartis", "country": "CH"}'
# {"op": "suggest", "query": "Novartis", "country": "CH",
#  "items": [{"name": "Novartis Pharma AG", "url": "https://www.northdata.com/…", "type": "company"}, …],
#  "fetch": {…}, "error": null, …}

curl -s https://api.cormorant.ch/v1/northdata/discover \
  -H "Authorization: Bearer $CORMORANT_API_KEY" -H "Content-Type: application/json" \
  -d '{"company_name": "Novartis Pharma AG", "headquarters": "Basel", "country": "CH"}'

curl -s https://api.cormorant.ch/v1/northdata/graph \
  -H "Authorization: Bearer $CORMORANT_API_KEY" -H "Content-Type: application/json" \
  -d '{"entity_id": "1234567", "session_key": "CH0012005267"}'
```

## session_key and country

- **`session_key`** keeps one residential exit for the company: suggest,
  profile and graph requests with the same key go out through the same Massive
  exit for 15 minutes ("flex" mode: single errors do not rotate it). Use a
  stable identifier of the company you are researching, such as its ISIN.
  After a proxy error the request is retried once on a fresh exit
  (`{session_key}-{8 hex}`).
- **`country`** is the company's or research market's country: ISO code
  (`CH`) or an English or German name (`Switzerland`, `Schweiz`). It filters
  suggest results and geotargets the exit. Unknown names are ignored, never
  cut to two letters.

## How Cormorant fetches North Data

- **Only through Massive, never from Cormorant's own addresses.** The
  residential proxy goes first, with the headers of one consistent desktop
  Chrome (the same User-Agent, client hints and language on pages and on the
  site's JSON calls). Massive Web Render follows when the proxy was
  blocked (a challenge, 403/429/451/5xx, a refusal) or the page was thin.
  North Data's fraud prevention refuses most residential exits, so after a
  refusal Cormorant skips the proxy for an hour and goes straight to Web
  Render (about 1 in 10 requests still tries the proxy, and a success ends
  the skip). The network graph is requested exactly as North Data's own page
  requests it (a `GET` of `data.json` with the page's `extra` token), so Web
  Render can fetch it as well. There
  is no direct, browser or archive tier. Every request therefore uses a paid
  tier:
  `allow_paid: false` answers `paid_tier_required`, an exhausted budget
  `budget_exhausted`. The same policy applies when you fetch these hosts with
  `POST /v1/fetch`.
- **One throttle for the whole site.** `.com` and `.de` count as one site.
  Across the whole cluster, at most 2 requests are in flight and starts are at
  least 1 s apart, for suggest, profile and graph alike. A 429 or a challenge
  answered to Web Render slows the site down further for a while (a refused
  residential exit does not: that limit is per exit, not for the cluster). Synchronous calls wait up to 60 s
  for a slot; batch items go back to the queue instead.
- **Bot protection** (Cloudflare, Akamai, DataDome, PerimeterX, Imperva, AWS
  WAF, Sucuri, captcha and "unusual traffic" pages) is detected and never
  served or cached as a profile. Thin pages (under 200 characters of text) and
  pages that do not parse as a profile (`parse_failed`) are not cached either.
- **Domain memory.** When every tier failed for the site in 3 consecutive
  requests, the site is skipped for a while (`blocked` with `retry_after_s`).
  Cormorant never learns a cheaper start tier for North Data.

### Caching

| What | Served from the cache for | Then |
|---|---|---|
| Profile pages | 24 h | revalidated with a conditional request through the proxy (`fetch.revalidated: true` when unchanged); entries expire after 7 days |
| `suggest.json` answers | 24 h | fetched again |
| `data.json` (graph) answers | 24 h | fetched again |

`cache_ttl_s` sets the maximum age you accept (`0` bypasses the cache). The
cache is shared: a profile one project fetched costs the next project
nothing. Cache hits answer `cached: true` and `billing_category:
northdata_cached`.

### robots.txt

Cormorant checks North Data's robots.txt for `CormorantBot` (and `*`)
before every profile page **and** before `suggest.json` and `data.json`. The
file is read through the residential proxy, like everything else, and cached
for 24 h. A disallowed URL answers `robots_disallowed` without fetching.

We made that call for the JSON endpoints on purpose. They are not
documented APIs but the requests North Data's own pages make, and robots.txt
is the site's statement about automated access to any path. So Cormorant
follows it for them too, instead of exempting them as "API calls". At the
time of writing, North Data's robots.txt blocks only a list of named crawlers
(AI and archive bots) and none of the paths Cormorant uses.

## Errors and retries

Upstream failures are HTTP 200 with `error` set, never an HTTP 5xx. Errors
worth retrying are `rate_limited` and `blocked` (both after `retry_after_s`),
`timeout`, `politeness_timeout` and `all_tiers_failed` (later, with backoff).
`not_found`, `parse_failed` and `invalid_identifier` are final for that input.
`paid_tier_required` and `budget_exhausted` are about your request or
project. A profile whose graph failed is still returned, with `graph_error`
set.

## Many companies: batch

For more than a handful of companies, submit one job:

```json
{
  "items": [
    {"op": "profile", "company_name": "Novartis Pharma AG", "country": "CH", "session_key": "CH0012005267"},
    {"op": "profile", "url": "https://www.northdata.com/Unilever+plc", "session_key": "GB00B10RZP78"}
  ],
  "idempotency_key": "research-run-2026-10-20-northdata",
  "callback_url": "https://example.org/hooks/cormorant"
}
```

`POST /v1/batch/northdata` returns a `job_id`. Poll `GET /v1/jobs/{job_id}`
or wait for the callback, then page through `?include=items`. Each item's
`result` has the shape of the matching single endpoint. The site throttle sets
the pace, about one profile per second across the cluster, so plan large runs
accordingly.

## Costs

Every uncached North Data request goes through Massive: the residential
proxy is billed by traffic, Web Render by request. A profile call can make up
to three requests (suggest, page, graph). `cost_usd` (for projects without a
customer account) and `GET /v1/usage` (the `northdata` block, the
`northdata` / `northdata_cached` categories) show what it cost; see
[Costs & billing](/docs/billing.md).

## What a profile contains, and checking the match

- **Officers** come from the page's management sections and its *Persons*
  timeline (full name, role, start and end date, e.g. Roland Busch, CEO since
  2021-03-15), then from the network graph (role `person`). North Data's free
  pages show a selection of persons; complete lists (for example every
  authorised signatory) are part of its paid service and not available here.
- **Related companies**: the page's subsidiary/parent/shareholder lists,
  merger, acquisition and control partners from its M&A timeline (relation
  `merger`, `acquisition`, `control`), and the graph's companies (`related`).
- **Network**: `network_graph` is North Data's graph; when that cannot be
  fetched (`graph_error`), it is built from the page's persons and related
  companies and says `source: "page"`.
- **Is it the right company?** Check `status` and `identifiers` before you
  use a discovered profile. A name match can be a branch or a foreign
  registration: "Novartis Pharma AG, Basel" on North Data is (also) the
  company's registration in Norway's Brønnøysund register (`identifiers`:
  "The Brønnøysund Register Centre 925942480", EUID `NOFOR.925942480`),
  terminated in 2024 (`status: "Terminated"`). Its `register_id`
  `BR 925942480` is that Norwegian number, not a Swiss UID.
- **Discovery says so itself**: pass `country` (and `headquarters`). Entries
  registered in that country are preferred, active over terminated, then the
  headquarters city; if none is found, Cormorant tries more suggest queries
  and the search before giving up, and a foreign or terminated result comes
  back with `discovery.warnings` (`country_mismatch`, `country_unverified`,
  `terminated`, `liquidation`) and `discovery.country`, the register country
  read from the page (`profile.register_country`). Treat any warning as "not
  verified"; `discovery.rejected` lists pages fetched and set aside for a
  better candidate.

## Reference

Company profiles, discovery and network graphs from North Data
(`northdata.com` / `northdata.de`). All four endpoints are synchronous (like
`POST /v1/fetch`), authenticated, metered (kind `northdata`, see *Billing
categories*) and answer HTTP 200 with `error` for upstream failures — never
an HTTP 5xx. For many companies use `POST /v1/batch/northdata`.

**Transport.** North Data is fetched only through Massive: the residential
proxy first (Chrome-consistent headers; a sticky exit per `session_key`,
15 minutes, retried once with a fresh exit after a proxy error), then
Massive Web Render when the proxy was blocked. North Data's fraud prevention
usually refuses residential exits (429, 403 for `data.json`): after a refusal
the proxy is skipped for an hour (`NORTHDATA_PROXY_SKIP_S`; 10 % of requests
still try it) and requests go straight to Web Render (attempt reason
`proxy_skipped_refused`). The network graph is requested like North Data's
own page does: `GET /data.json?domain=company&id=…&type=graph&extra=<token>`,
so Web Render can serve it too. Never from Cormorant's own
addresses: no `direct`, `browser` or `archive` tier and no alternative
sources. This also applies to `POST /v1/fetch` of these hosts. `allow_paid:
false` (or a key without paid fetches) answers `error: paid_tier_required`.
A cluster-wide throttle covers the whole site (`.com` and `.de` together;
suggest, profile and graph): by default at most 2 requests in flight and
1 s between starts. robots.txt is read through the proxy and applies to
profile pages, `suggest.json` and `data.json`.

**Cache.** Profile pages: served for 24 h, then revalidated with a conditional
request (`fetch.revalidated`), kept 7 days. Suggest and graph answers: 24 h.
Challenge pages, thin pages (< 200 characters of text) and pages that do not
parse are never kept. `cache_ttl_s` sets the maximum acceptable age (`0`
bypasses the cache).

| Endpoint | Request | Response |
|---|---|---|
| `POST /v1/northdata/suggest` | `query`, `country?`, `session_key?` | `items[{name, url, type}]` (companies only, absolute URLs), `fetch` |
| `POST /v1/northdata/discover` | `company_name`, `headquarters?`, `country?`, `session_key?`, `min_similarity?` (70), `search_fallback?` (true) | `url`, `query`, `score` (0..1), `name_similarity` (0..100 or null), `discovery_method` (`suggest`/`search`), `candidates[≤10 {url, name, score, query, title, country}]`, `country`, `warnings[]`, `suggest_fetch` |
| `POST /v1/northdata/profile` | `url` **or** `company_name` (+ `country`, `headquarters`), `session_key?`, `country?`, `include_graph?` (true), `min_similarity?`, `search_fallback?` | `profile` (below), `fetch`, `discovery` (when discovered: as `discover`, plus `rejected[{url, warnings}]`), `graph_fetch`, `graph_error` |
| `POST /v1/northdata/graph` | `entity_id`, `session_key?`, `country?` | `entity_id`, `company_connections`, `person_connections`, `other_connections`, `fetch` |

Every request also takes `cache_ttl_s`, `allow_paid` (default true) and
`project_ref`. Every response has `op`, `error`, `retry_after_s`, `cost_usd`
(null for external customers), `billing_category` (`northdata` or
`northdata_cached`) and `request_id`. `country` is an ISO code (`CH`) or an
English/German country name (`Switzerland`); unknown names are ignored. It
filters suggest (`countries=`) and geotargets the Massive exit.

**Discovery and `country`.** Candidates are ranked by register country
(`country` equals the requested one; read from the register in the URL —
`CHE-…` Switzerland, `HRB`/Amtsgericht Germany, `BR` Norway's Brønnøysund … —
before the address, because North Data lists foreign registrations under the
company's address), then active over terminated, then the `headquarters`
city, then name similarity (at least `min_similarity`). When suggest has no
entry in the requested country (or only a terminated one), further suggest
queries are tried (the name without its legal form, then name + city), then
the search fallback with its country-specific queries. If the best entry is
still foreign or terminated it is returned **with `warnings`** rather than
silently: `country_mismatch` (register country differs), `country_unverified`
(unknown), `terminated`, `liquidation`; empty for a clean match. `profile`
checks the fetched page's register (EUID, register entries): when it is
foreign or terminated, up to `NORTHDATA_DISCOVERY_ALTERNATES` (1) other
candidates in the requested country are fetched, and the first clean one is
used; the set-aside pages are listed in `discovery.rejected`. Example: for
`Novartis Pharma AG`, `CH` the only suggestion is the terminated Norwegian
entry `…/BR 925942480` (EUID `NOFOR.925942480`): it comes back with
`discovery.country: "NO"` and `warnings: ["country_mismatch", "terminated"]`.

```json
{
  "op": "profile",
  "profile": {
    "name": "Novartis Pharma AG", "legal_form": "AG", "register_id": "BR 925942480",
    "court": null, "status": "Active", "address": "Lichtstrasse 35, 4056 Basel, Switzerland",
    "city": "4056 Basel", "country": "Switzerland", "path_country": "CH",
    "entity_id": "…", "subsidiaries": [{"name": "Novartis International AG",
      "relation": "subsidiaries", "url": "/Novartis%20International%20AG",
      "absolute_url": "https://www.northdata.com/Novartis%20International%20AG"}],
    "source_url": "https://www.northdata.com/Novartis%20Pharma%20AG,%20Basel,%20CH/BR%20925942480",
    "discovery_method": "suggest", "discovery_score": 1.0, "name_similarity": 100.0,
    "fetch_tier": "web_render", "parser_version": "4", "provider": "northdata"
  },
  "fetch": {
    "url": "https://www.northdata.com/…", "final_url": "https://www.northdata.com/…",
    "status": 200, "tier": "proxy", "body_url": "/v1/bodies/9f2c…", "sha256": "9f2c…",
    "body_kind": "origin", "retrieved_at": "2026-10-19T12:00:00+00:00", "cached": false,
    "revalidated": false, "attempts": [{"tier": "proxy", "status": 200, "reason": "ok", "ms": 2100}],
    "cost_usd": 0.0004, "parser_version": "4"
  },
  "discovery": {"url": "https://www.northdata.com/…", "query": "Novartis Pharma AG",
    "score": 1.0, "name_similarity": 100.0, "discovery_method": "suggest", "candidates": [],
    "country": "NO", "warnings": ["country_mismatch"], "rejected": []},
  "graph_fetch": null, "graph_error": "no_entity_id",
  "error": null, "retry_after_s": null, "cost_usd": 0.0006,
  "billing_category": "northdata", "request_id": "req_…"
}
```

**Profile fields** (`parser_version` `4`; parses HTML and the Markdown Web
Render sometimes returns): `name`, `legal_form` (from the company's own name
or legal-form block), `register_id` (check `identifiers` for which register it
belongs to), `court` (German register court as shown, e.g. `Amtsgericht
München` or `District Court of Munich`), `status` (`Terminated` /
`Liquidation` when North Data marks it so, else `Active` for a register
profile), `identifiers[{label, source, value, current}]` (the Identification
block: register entries, LEI, EUID, …), `address`, `city`,
`country` (free text, not normalised), `register_country` (ISO code of the
register: EUID, else the register entry, else the URL), `path_country` (ISO
code in the URL's address part),
`entity_id`, `lei`, `industry`, `former_names[]`, `former_addresses[]`,
`corporate_purpose`, `additional_info`, `history[{category, date,
description, text, url}]`, `publications[{date, summary, company_name}]`,
`mentions[…]`, `network_graph {company_connections, person_connections,
other_connections, source}` (`source`: `data.json`, or `page` when the graph
could not be fetched and the network is built from the page's persons and
merger/acquisition data), `officers[{name, role, since, until, url,
absolute_url}]` (from the page's management sections and its Persons
timeline, e.g. `Roland Busch`, `CEO`, since `2021-03-15`, then graph persons),
`shareholders`/`subsidiaries`/`parent_companies`/`related` (`related` also
holds merger/acquisition/control partners of the page's M&A timeline)
`[{name, relation, url, absolute_url}]`, `financials` (always null),
`source_url`, `discovery_query`, `discovery_score` (0..1),
`discovery_method`, `name_similarity` (0..100), `scraped_at`, `fetch_tier`
(`proxy` or `web_render`), `parser_version`, `provider` (`northdata`).
Graph merge: graph companies not already listed become `related` (relation
`related`); graph persons become `officers` with role `person`; parents and
subsidiaries are never invented.

**Discovery.** Suggest first: company items only; the one with the highest
name similarity (Jaro-Winkler + Levenshtein on names without legal
suffixes, 0..100) wins when it reaches `min_similarity`; `score` = similarity
/ 100; no first-result fallback. Otherwise, with `search_fallback`,
Cormorant's own search with up to four queries (`northdata "{name}" {city
cc}`, `site:northdata.com OR site:northdata.de "{name}" {city or cc}`,
`northdata "{name}" {cc}`, `northdata {name}`), 5 results each, profile URLs
scored 0..1 (name tokens in the path and title, city, country, host, register
in the path); it stops at ≥ 0.75 and rejects < 0.35. Suggest candidates carry
the name similarity (0..100) as `score`, search candidates the URL score.

| `error` | Meaning |
|---|---|
| `not_found` | no profile (404/410, or discovery found nothing) |
| `blocked` | every Massive tier was blocked (challenge, 403/451, content policy); with `retry_after_s` while the site is skipped |
| `rate_limited` | North Data answered 429; retry after `retry_after_s` |
| `all_tiers_failed` | every tier failed for another reason (network, 5xx, not JSON) |
| `parse_failed` | the page is not a company profile (e.g. a search page); not cached |
| `invalid_identifier` | `url` is not a North Data profile URL, or `entity_id` is malformed |
| `paid_tier_required` | `allow_paid: false` or the key has no paid fetches: North Data needs Massive |
| `budget_exhausted` | the project's monthly budget leaves nothing for Massive |
| `robots_disallowed` | robots.txt disallows the URL for `CormorantBot`/`*` |
| `timeout` | the request ran out of time (sync calls stop before the API's wait) |
| `politeness_timeout` | the site throttle stayed busy (transient; batch items retry) |

`graph_error` (profile only): why the graph is missing while the profile is
returned — an error code from the table, or `no_entity_id`.

### Batch: POST /v1/batch/northdata

```json
{
  "items": [
    {"op": "profile", "company_name": "Novartis Pharma AG", "country": "CH", "session_key": "CH0012005267"},
    {"op": "graph", "entity_id": "1234567"}
  ],
  "idempotency_key": "chipiron-run-2026-10-20-northdata-0001",
  "callback_url": "https://chipiron.example/hooks/cormorant"
}
```

`op` is `suggest`, `discover`, `profile` (default) or `graph`, plus the fields
of that endpoint (`422` when the field an operation needs is missing). Items
run like the single endpoints and their results have the same shape; the
site throttle sets the pace (a busy throttle puts an item back in the queue).
## Coming from Chipiron's own scraper

The profile follows Chipiron's `NorthdataProfile`, with these differences:

| Chipiron | Cormorant |
|---|---|
| `discovery_method: "brave"` | `"search"` (Cormorant's own search, Brave only as its fallback) |
| entity `url` relative only | `url` as on the page, plus `absolute_url` |
| officers `{name, role}` (`role` = "officers"/"person") | `{name, role, since, until, url, absolute_url}`; `role` is the stated role when the page has one |
| `fetch_tier` `direct`/`browser`/`proxy`/`web_render` | only `proxy` or `web_render` (never our own addresses) |
| `parser_version: "2"` | `"4"` |
| `status` only when the page has a status field | `Terminated` / `Liquidation` from North Data's ✝︎ marker or the register history, else `Active` |
| — | `identifiers` (the Identification block, with each register's name) |
| `network_graph` only from `data.json` | also built from the page (`source: "page"`) when the graph cannot be fetched |
| — | new: `path_country`, `lei`, `industry` |
| identity acceptance against the issuer | stays in Chipiron: use `name`, `country`, `city`, `register_country`, `register_id`, `name_similarity`, `discovery_score` and `discovery.warnings` |