CormorantDocs
Menu

Limits & politeness

Cormorant is a good citizen of the web on your behalf. That shapes how fast you can go — mostly per domain, not per project.

Hard limits#

Limit Value What happens
Requests per project 200/s by default (operator can set per project) 429 with Retry-After: 1. A batch call counts as one request.
Per-key limits set by the operator per key (see below) 429 key_quota_exhausted, key_budget_exhausted, key_rate_limited; 403 key_category_not_allowed
Synchronous wait 150 s (or timeout_s + 15 s) 504 {"detail": {"error": "timeout", …}}
timeout_s 5–150 s validation error outside the range
Items per batch job 50,000 413
Request body 64 MB 413 — split big batches
Fetch body size (max_bytes) default 15 MB, at most 50 MB error: "too_large"
PDFs in batch fetches up to 200 MB when the item leaves max_bytes unset (an explicit max_bytes applies to PDFs too) error: "too_large"
Large PDF download time 5 minutes in total (or timeout_s, when shorter) error: "timeout"
PDF text first 5,000 pages / 5,000,000 characters text_truncated: true; the full PDF is at body_url
Redirects at most 10 followed per fetch the fetch fails
Search max_results 1–50 validation error
Job page size (limit) 1–1,000 validation error
Retention jobs and results: 30 days after the job finishes; bodies: at least 30 days after your last retrieval 404 afterwards

Per-key limits#

The operator can give each API key its own limits, enforced on every request (ask for them, e.g. to give a client or a notebook a bounded key):

Limit What happens
Monthly spend cap (your invoice currency) 429 key_budget_exhausted until the month resets
Monthly requests per category: search, fetch, fetch_paid search/fetch: 429 key_quota_exhausted; fetch_paid used up: fetches still run, without the paid tiers
Requests per second 429 key_rate_limited with Retry-After: 1
Allowed categories (e.g. search only) 403 key_category_not_allowed; without fetch_paid, fetches run without the paid tiers

Batch jobs are checked when created and again for each item when it runs: items beyond a quota fail with the same error code and are not billed. Request counts are exact; the spend cap compares the charges of the aggregates (refreshed every 5 minutes) plus an estimate of the requests since then, so a key can overshoot its cap by a few minutes of traffic.

Every authenticated response tells you where you stand:

RateLimit-Policy: "project";q=200;w=1, "key";q=10;w=1
RateLimit: "project";r=187;t=1, "key";r=9;t=1
X-Cormorant-Key-Budget-Remaining: 12.40 USD
X-Cormorant-Quota-Remaining: fetch=18766, search=900

RateLimit-Policy/RateLimit follow the IETF draft RateLimit header fields for HTTP (q limit, w window in seconds, r remaining, t seconds to reset). The two X-Cormorant-… headers appear only when the key has a spend cap or quotas. GET /v1/quota has the full picture, and POST /v1/alerts registers a signed webhook for 50 / 80 / 100 % of a limit.

robots.txt#

Cormorant checks robots.txt for the user-agent token CormorantBot (then *) before fetching and answers error: "robots_disallowed" when a path is off limits. robots.txt files are cached for a day; a missing robots.txt (4xx) allows everything.

Per-domain politeness#

To avoid hammering any single site, Cormorant limits concurrency per domain and egress IP, across all of its hosts:

  • at most 2 requests in flight to the same domain from the same egress,
  • at least 250 ms between request starts,
  • after a 429 or a bot challenge the interval doubles (up to 30 s) and decays again after about 10 minutes.

What this means for you:

  • Many URLs on many domains go fast. A batch of 10,000 URLs spread over 2,000 sites runs in parallel across all workers.
  • Many URLs on one domain go at the domain's polite rate — a few requests per second at best, slower if the site pushes back. 5,000 pages from one site take tens of minutes however you send them.
  • A domain that blocks every tier is skipped for a while. After three consecutive fetches where every tier failed, fetches for that domain answer domain_unavailable with retry_after_s (10 minutes at first, up to 6 hours) instead of spending time and money on it.
  • Batches handle this for you. Admission is domain-aware: a domain that dominates your job drains at its polite rate while the other domains proceed, and items waiting for a busy domain do not block workers.
  • Synchronous calls wait up to 45 s for a slot on a busy domain and then return error: "politeness_timeout". Hitting one domain with many parallel sync calls is the slowest way to use Cormorant.

Search throughput#

Each SearXNG instance takes about 15 queries per second (a query takes ~4 s because engines are queried upstream); bursts queue instead of failing. The Brave fallback is capped cluster-wide. For hundreds of queries, use POST /v1/batch/search.

Practical advice#

  • Prefer one batch over many parallel sync calls.
  • Do not run your own retry loop on politeness_timeout or all_tiers_failed in tight succession — see Errors & retries.
  • Re-fetching the same URL within its cache lifetime is free and instant; cache_ttl_s: 0 forces a real request — use it only when you need it.
  • If a site needs a contact User-Agent (SEC EDGAR, Wikimedia), ask the operator to add yours to the project allowlist and send it with user_agent.

View as Markdown