# Limits & politeness

Cormorant is a good citizen of the web on your behalf. That shapes how fast
you can go — mostly per *domain*, not per project.

## Hard limits

| Limit | Value | What happens |
|---|---|---|
| Requests per project | 200/s by default (operator can set per project) | `429` with `Retry-After: 1`. A batch call counts as one request. |
| Per-key limits | set by the operator per key (see below) | `429 key_quota_exhausted`, `key_budget_exhausted`, `key_rate_limited`; `403 key_category_not_allowed` |
| Synchronous wait | 150 s (or `timeout_s` + 15 s) | `504 {"detail": {"error": "timeout", …}}` |
| `timeout_s` | 5–150 s | validation error outside the range |
| Items per batch job | 50,000 | `413` |
| Request body | 64 MB | `413` — split big batches |
| Fetch body size (`max_bytes`) | default 15 MB, at most 50 MB | `error: "too_large"` |
| PDFs in batch fetches | up to 200 MB when the item leaves `max_bytes` unset (an explicit `max_bytes` applies to PDFs too) | `error: "too_large"` |
| Large PDF download time | 5 minutes in total (or `timeout_s`, when shorter) | `error: "timeout"` |
| PDF text | first 5,000 pages / 5,000,000 characters | `text_truncated: true`; the full PDF is at `body_url` |
| Redirects | at most 10 followed per fetch | the fetch fails |
| Search `max_results` | 1–50 | validation error |
| Job page size (`limit`) | 1–1,000 | validation error |
| Retention | jobs and results: 30 days after the job finishes; bodies: at least 30 days after your last retrieval | `404` afterwards |

## Per-key limits

The operator can give each API key its own limits, enforced on every request
(ask for them, e.g. to give a client or a notebook a bounded key):

| Limit | What happens |
|---|---|
| Monthly spend cap (your invoice currency) | `429 key_budget_exhausted` until the month resets |
| Monthly requests per category: `search`, `fetch`, `fetch_paid` | `search`/`fetch`: `429 key_quota_exhausted`; `fetch_paid` used up: fetches still run, without the paid tiers |
| Requests per second | `429 key_rate_limited` with `Retry-After: 1` |
| Allowed categories (e.g. search only) | `403 key_category_not_allowed`; without `fetch_paid`, fetches run without the paid tiers |

Batch jobs are checked when created and again for each item when it runs:
items beyond a quota fail with the same error code and are not billed.
Request counts are exact; the spend cap compares the charges of the
aggregates (refreshed every 5 minutes) plus an estimate of the requests since
then, so a key can overshoot its cap by a few minutes of traffic.

Every authenticated response tells you where you stand:

```
RateLimit-Policy: "project";q=200;w=1, "key";q=10;w=1
RateLimit: "project";r=187;t=1, "key";r=9;t=1
X-Cormorant-Key-Budget-Remaining: 12.40 USD
X-Cormorant-Quota-Remaining: fetch=18766, search=900
```

`RateLimit-Policy`/`RateLimit` follow the IETF draft *RateLimit header
fields for HTTP* (`q` limit, `w` window in seconds, `r` remaining, `t`
seconds to reset). The two `X-Cormorant-…` headers appear only when the key
has a spend cap or quotas. `GET /v1/quota` has the full picture, and
`POST /v1/alerts` registers a signed webhook for 50 / 80 / 100 % of a limit.

## robots.txt

Cormorant checks robots.txt for the user-agent token `CormorantBot` (then
`*`) before fetching and answers `error: "robots_disallowed"` when a path is
off limits. robots.txt files are cached for a day; a missing robots.txt (4xx)
allows everything.

## Per-domain politeness

To avoid hammering any single site, Cormorant limits concurrency **per domain
and egress IP, across all of its hosts**:

- at most 2 requests in flight to the same domain from the same egress,
- at least 250 ms between request starts,
- after a `429` or a bot challenge the interval doubles (up to 30 s) and
  decays again after about 10 minutes.

What this means for you:

- **Many URLs on many domains go fast.** A batch of 10,000 URLs spread over
  2,000 sites runs in parallel across all workers.
- **Many URLs on one domain go at the domain's polite rate** — a few requests
  per second at best, slower if the site pushes back. 5,000 pages from one
  site take tens of minutes however you send them.
- **A domain that blocks every tier is skipped for a while.** After three
  consecutive fetches where every tier failed, fetches for that domain answer
  `domain_unavailable` with `retry_after_s` (10 minutes at first, up to 6
  hours) instead of spending time and money on it.
- **Batches handle this for you.** Admission is domain-aware: a domain that
  dominates your job drains at its polite rate while the other domains
  proceed, and items waiting for a busy domain do not block workers.
- **Synchronous calls wait** up to 45 s for a slot on a busy domain and then
  return `error: "politeness_timeout"`. Hitting one domain with many parallel
  sync calls is the slowest way to use Cormorant.

## Search throughput

Each SearXNG instance takes about 15 queries per second (a query takes ~4 s
because engines are queried upstream); bursts queue instead of failing. The
Brave fallback is capped cluster-wide. For hundreds of queries, use
`POST /v1/batch/search`.

## Practical advice

- Prefer one batch over many parallel sync calls.
- Do not run your own retry loop on `politeness_timeout` or `all_tiers_failed`
  in tight succession — see [Errors & retries](/docs/errors.md).
- Re-fetching the same URL within its cache lifetime is free and instant;
  `cache_ttl_s: 0` forces a real request — use it only when you need it.
- If a site needs a contact User-Agent (SEC EDGAR, Wikimedia), ask the
  operator to add yours to the project allowlist and send it with
  `user_agent`.