# Monitoring Cormorant's status

Cormorant publishes the health of its subsystems — search, fetch, browser
rendering, archive, paid tiers, batch jobs and storage — so you can watch it
from your own monitoring and tell "our pipeline is broken" from "Cormorant is
having a problem". People can read the same information at
[`/status`](/status); machines read `GET /v1/status`.

No API key is needed, and the answer is shared by everyone and refreshed at
most every 15 seconds, so polling it costs you nothing against your project's
rate limit.

## Quick start

```sh
# Tiny body for uptime monitors
curl -s https://api.cormorant.ch/v1/status/summary
# {"status":"operational","description":"All systems operational","updated_at":"…"}

# Everything: components, reasons, metrics, incidents, maintenance
curl -s https://api.cormorant.ch/v1/status
```

## What to poll, and how often

- **Uptime monitors** (Uptime Kuma, Better Stack, Pingdom, a cron job):
  `GET /v1/status/summary` and a keyword check for `"status":"operational"`.
- **Your own dashboards and alerting**: `GET /v1/status`, then look at the
  components your integration depends on (`search`, `fetch`, `batch_jobs`, …).
  Company data sources are listed one per source in the group `company_data`
  (`company_gleif`, …); a degraded source only means fresher data from it may
  be delayed — the company register still answers.
- **Interval**: every **30–60 seconds**. Polling more often only returns the
  same snapshot (`Cache-Control: public, max-age=15`); send `If-None-Match`
  with the last `ETag` to get a `304` when nothing changed. More than 120
  requests per minute from one IP address get `429`.

## Why the status endpoints always answer 200

The HTTP code says whether *Cormorant answered*; the body says *how it is
doing*. A status endpoint that returned `503` during an outage could not be
told apart from a broken proxy, a DNS problem or your own network. So:

| You see | It means | Alert |
|---|---|---|
| `200` and `status: operational` | all good | no |
| `200` and `status: degraded` | slower or more failures than usual; requests mostly work | warning |
| `200` and `status: partial_outage` / `major_outage` | part or all of the service is failing | page someone if you depend on it |
| `200` and `status: maintenance` | announced maintenance in progress (see `maintenance`) | no, or informational |
| timeout, connection error, any non-200 | Cormorant unreachable (or your network) | yes, after 2–3 consecutive checks |

## Recommended alert rules

1. **Unreachable**: 3 consecutive failed checks (timeout 10 s) → critical.
2. **Outage of what you use**: a component you depend on is
   `partial_outage` or `major_outage` in 2 consecutive checks → critical.
3. **Degradation**: a component you use is `degraded` for 10 minutes →
   warning. Short blips are normal: status is computed from the last 10
   minutes of traffic.
4. **Incidents**: a new entry in `incidents` with `status` other than
   `resolved` → notify your team with its `title` and latest `updates`.
5. Do **not** alert on a component that is missing: components not part of
   the deployment are not listed. Use the stable `id`, never `name`.

`reasons` says why a component is not operational (`no_workers`,
`queue_lag`, `low_success_rate`, …) and `metrics` gives the numbers behind
it, e.g. the fetch success rate or the age of the oldest waiting request.

## Example: check from Python

```python
import httpx

r = httpx.get("https://api.cormorant.ch/v1/status", timeout=10)
r.raise_for_status()  # non-200: Cormorant unreachable
status = r.json()
mine = {c["id"]: c for c in status["components"] if c["id"] in ("search", "fetch")}
for c in mine.values():
    if c["status"] in ("partial_outage", "major_outage"):
        print(f"ALERT {c['name']}: {c['status']} ({', '.join(c['reasons'])})")
for incident in status["incidents"]:
    if incident["status"] in ("investigating", "identified", "monitoring"):
        print(f"Incident: {incident['title']} — {incident['updates'][0]['message']}")
```

## Incidents, maintenance and history

The operator posts incidents (with an impact and a timeline of updates) and
announces maintenance windows; both appear in `GET /v1/status` and on the
status page. During an announced window the affected components show
`maintenance` rather than an outage. `GET /v1/status/history?hours=24` gives
the worst status per 5 minutes for the last 24 hours (hourly up to 7 days)
and the uptime per component — useful for SLA reports.

## Reference

Public status of the service for your monitoring, and the page at
[`/status`](/status). **No authentication** (an `Authorization` header is
accepted and ignored), CORS-enabled, refreshed at most every 15 s and shared
by all callers (`Cache-Control: public, max-age=15`, `ETag` /
`If-None-Match` → `304`). Limited to 120 requests per minute per client IP
(`429` with `Retry-After`). The status never contains host names, addresses,
instance URLs, project names, keys or internal error strings — only component
ids, statuses, reason codes and aggregate numbers.

The status endpoints answer **`200` whatever the service status**: the
status is in the body. Alert on the body (`status` is not `operational`), and
treat a timeout, a connection error or any other HTTP code as "Cormorant is
unreachable" — a status endpoint that answered `503` during an outage could
not be told apart from a broken proxy or network in between.

| Endpoint | Returns |
|---|---|
| `GET /v1/status` | overall `status`, every component with `reasons` and `metrics`, incidents and maintenance |
| `GET /v1/status/summary` | only `status`, `description`, `updated_at` (for uptime monitors) |
| `GET /v1/status/components/{id}` | one component plus the incidents and maintenance naming it (`404` for an id not in this deployment) |
| `GET /v1/status/history?hours=24` | worst status per 5 minutes (per hour beyond 24 h, up to 168 h) and uptime per component; overall as `_overall` |

```json
{
  "status": "degraded",
  "description": "Degraded performance",
  "updated_at": "2026-10-19T09:41:15+00:00",
  "window_s": 600,
  "components": [
    {"id": "api", "name": "API", "description": "…", "status": "operational", "reasons": [], "metrics": {}, "group": null},
    {"id": "fetch", "name": "Fetch", "description": "…", "status": "degraded", "reasons": ["low_success_rate"], "metrics": {}, "group": null},
    {"id": "company_gleif", "name": "Company data: GLEIF", "description": "…", "status": "operational", "reasons": [], "metrics": {}, "group": "company_data"}
  ],
  "incidents": [
    {"id": "inc_3f9a1c0b7e", "title": "Elevated fetch failures", "status": "identified",
     "impact": "degraded", "components": ["fetch"], "created_at": "…", "updated_at": "…",
     "resolved_at": null, "updates": [{"at": "…", "status": "identified", "message": "…"}]}
  ],
  "maintenance": []
}
```

**Statuses**, best to worst: `operational`, `maintenance`, `degraded`,
`partial_outage`, `major_outage`. The overall `status` is the worst component
status, except that a major outage of a component other than `api`,
`search`, `fetch` or `storage` counts as a partial outage, and it is
`maintenance` when the only components not operational are under announced
maintenance. An unresolved incident shows its components at least at its
`impact`. Components not part of the deployment are not listed (`archive`
when archived copies are disabled, `paid_tiers` when no paid tier is
configured, `sec` while no SEC identity is configured) — do not alert on a
missing component.

| Component | Metrics (rates 0–1 over the last `window_s` = 600 s; `null` = too few requests) | Not operational when |
|---|---|---|
| `api` | — | database or cache unreachable (major) |
| `search` | `success_rate`, `requests`, `engines_healthy_share`, `fallback_available`, `fallback_share`, `fallback_success_rate`, `workers_alive`, `queue_lag_s` | success rate of uncached searches < 0.95 / 0.80 / 0.50 (degraded / partial / major); healthy engines < 50 % (degraded) or none (partial with the paid fallback, major without); no worker; queue lag |
| `fetch` | `success_rate`, `requests`, `latency_p95_ms`, `workers_alive`, `sync_queue_backlog`, `sync_queue_lag_s`, `batch_queue_backlog` | success rate of uncached fetches < 0.85 / 0.60 / 0.30 (4xx answers of the site, robots.txt, invalid or blocked URLs, too large, PDF not accepted and deferred are not failures); p95 > 60 s (degraded); no worker (major); queue lag |
| `browser_rendering` | `success_rate`, `attempts`, `workers_alive`, `sync_queue_backlog`, `sync_queue_lag_s`, `batch_queue_backlog` | browser attempts failing for infrastructure reasons: success < 0.80 / 0.50 / 0.10; no browser worker (major) |
| `archive` | `success_rate`, `attempts` | archive unreachable or erroring: success < 0.80 / 0.50 / 0.10 |
| `paid_tiers` | `proxy_success_rate`, `proxy_attempts`, `rendering_success_rate`, `rendering_attempts` (configured tiers only) | provider errors: success < 0.80 / 0.50 / 0.10 |
| `sec` | `success_rate`, `requests`, `throttle_wait_p95_ms` (p95 wait for a slot in the cluster-wide SEC rate limiter), `backoff_active`, `rate_limit_rps` | SEC backing off after 403/429/503 (degraded, `sec_backoff`); success rate of uncached SEC requests < 0.85 / 0.60 / 0.30 (4xx, invalid identifiers and not found are not failures); p95 wait > 5 s (degraded, `throttled`) |
| `batch_jobs` | `scheduler_last_run_s`, `workers_alive`, `running_jobs`, `oldest_waiting_job_s`, `queue_backlog` | scheduler last ran > 30 s (degraded) / > 120 s ago (partial); no batch worker (major); a job waiting to start > 15 min (degraded) / > 60 min (partial) |
| `storage` | `database_ok`, `cache_ok`, `object_storage_ok` (`null`: not configured) | database or cache unreachable (major); object storage unreachable (degraded) |
| `company_<source>` (group `company_data`, one per active company data source, e.g. `company_gleif`) | `success_rate`, `requests` (last hour of calls to the source; `null` below 10), `latency_p95_ms`, `parse_failure_rate`, `canary_ok`, `backoff_active`, `schema_drift` (fields that used to be filled and now come back empty), `change_feed_lag_s` | success < 0.85 / 0.50 / 0.20; canaries (known companies checked every 15 minutes) failing (partial); backoff after repeated failures, schema drift, parse failures > 10 % (degraded). Counts at most as `degraded` for the overall status |

Queue lag (`queue_lag_s`, `sync_queue_lag_s`: how long the oldest synchronous
request has waited for a worker) above 15 / 60 / 120 s is degraded / partial
/ major. Rates computed on fewer than 20 requests (10 tier attempts) are
reported but do not change a status. New metric keys may be added.

**Reason codes**: `no_workers`, `queue_lag`, `low_success_rate`,
`high_latency`, `engines_reduced`, `engines_unavailable`, `tier_errors`,
`scheduler_stalled`, `jobs_waiting`, `database_unreachable`,
`cache_unreachable`, `object_storage_unreachable`, `sec_backoff`,
`throttled`, `incident`, `maintenance`; company data sources also `canary_failed`, `source_backoff`,
`schema_drift`, `parse_failures`.

**Incidents and maintenance** are posted by the operator. `incidents` lists
unresolved incidents (`investigating`, `identified`, `monitoring`) and those
`resolved` within 7 days, newest first, each with its timeline (`updates`,
newest first). `maintenance` lists `scheduled` and `in_progress` windows and
those `completed` within 7 days, soonest first; an empty `components` list
means the whole service.

**History**: `GET /v1/status/history` returns, per component, `statuses`
(oldest first; `null` where nothing was recorded) and `uptime` over the
period, where a `partial_outage` bucket counts as 30 % downtime and a
`major_outage` bucket as 100 %. Kept 7 days.