API reference
Rate limits and holds
Requests per minute, tokens per minute, concurrency, and how allowance is held before a request runs.
On this page
Every allocation carries three rate limits from its agreement, and every request holds allowance before it runs. Both are checked centrally under a lock on the allocation, so parallel workers can't jointly go over.
The three limits
| Limit | What's counted | Over the limit |
|---|---|---|
requests_per_minute | Requests started in the trailing 60 seconds, including ones that later failed | 429 rate_limited, Retry-After: 60 |
tokens_per_minute | Input plus output tokens booked in the trailing 60 seconds | 429 rate_limited, Retry-After: 60 |
max_concurrency | Requests in flight at once | 429 rate_limited, Retry-After: 2 |
Read your limits from . They're the numbers in your agreement, and the supplier can't change them without a new agreement.
tokens_per_minute counts usage once it's booked, so a burst of long requests can start together and only count against the limit as they finish.
Admission order
Before a request reaches the provider, the gateway checks, in order:
- The allocation is in its window and
active. Areadyallocation becomesactivehere if its window has opened. - Requests per minute.
- Concurrency.
- Tokens per minute.
- Enough allowance to hold the request.
The first check that fails is the error you get.
How allowance is held
Each request places a hold on your allowance before it runs:
- Input: an estimate from the prompt length, about one token per three characters of message content plus a small fixed overhead. It's deliberately generous.
- Output: the request's
max_tokens(ormax_completion_tokens), 1,024 if you didn't set one.
If max_tokens is more than your remaining output allowance, it's lowered to fit, so a single request can't overspend the agreement. If no output allowance is left, or the input estimate is more than the input left, the request fails with 402 allocation_exhausted.
While a request runs, its hold counts against remaining. That's why remaining can drop and come back up while requests are in flight.
When usage arrives
When the provider reports token counts, Cairva releases the hold and books the real counts in one transaction. Each provider response is booked exactly once, keyed on the allocation and the provider's request id, so the same response can't be booked twice.
When the full input or output allowance has been used, the allocation becomes depleted, its agreement completes, and later requests get 402 allocation_depleted.
When usage doesn't arrive
If the provider completes a request but doesn't return token counts, Cairva doesn't guess. The request is marked usage_pending, its hold stays in place, and the response says "usage_source": "pending" with usage: null. Cairva operations then book the counts from provider metering with written evidence. The booked event has source: "reconciliation" and finality: "confirmed", and the hold is released.
A request that's still marked in flight after 10 minutes, for example because a process died mid-stream, is moved to usage_pending and reconciled the same way. It stops counting toward concurrency at that point.
When a request fails
If the provider errors, the hold is released right away and the request books nothing. Failed requests still count toward requests_per_minute.