Skip to Content

Limits and estimates

Usage limits, rate limits, service capacity and provider quotas apply independently. A request must pass each applicable check. Upstream quota cannot override the customer’s Modem limit.

Usage allowances

A usage limit caps the amount a customer can use in a billing period. Usage rules match model identifiers or model patterns and measure a supported unit. A matching rule is required; otherwise the gateway returns model_not_entitled. Requests are counted at admission. Provider-reported measures such as tokens are reconciled when generation finishes, so concurrent work can cross a soft allowance ceiling. A rule is not a guarantee that an upstream provider will accept the request.

See Configure a product and customer usage. Subscription relay rules currently use request and token measures; they are not provider credit caps.

Rate windows

Rate limits use fixed, epoch-aligned minute, hour or day windows. They are not rolling lookbacks or continuously refilling buckets. Counters are isolated by customer, product, rule and measure. When several rules block a request, the gateway computes the latest reset needed to make it fit.

A rate_limited refusal includes Retry-After only when the request could fit every blocking rule in an empty bucket. A request larger than a rule’s entire capacity cannot be fixed by waiting; the header is omitted and SDK retry is disabled. A useful delay longer than 60 seconds can still be returned with automatic SDK retry disabled. Your UI can display that time without blindly retrying generation.

Measures not known at admission can begin with a zero reservation and settle later. This allows measured totals to cross the limit with in-flight work. Counter-store failure normally stops admission with 503 metering_unavailable; it must not be presented as a customer’s exhausted provider subscription.

Estimated cost

Regular-product Chat Completions and Completions rules can measure costMicros: one million units is one US dollar. The gateway needs an applicable model price band. A matching cost rule on an unsupported API or model without applicable prices produces 503 product_not_configured. Providers intentionally configured without prices count at zero; that is different from a missing applicable price.

Estimated cost is settled from measured token usage rather than reserved as an exact dollar amount before inference. A cost allowance can therefore be exceeded by in-flight generation. It is not a prepaid provider balance or strict spending guarantee.

Allowance calculations can use conservative price bands when input/output or cache-write details are missing. Reference estimates shown in reports require their own usage detail and may be partial or unavailable. These two calculations need not match. Allowance totals round upward to whole microdollars; reference calculations can retain fractional microdollars.

For subscriptions, API-parity cost estimates describe what comparable API usage might cost. They do not represent a charge from ChatGPT, Grok or Claude, and they must not be subtracted from a provider credit balance as if they did.

Concurrency and duration

Concurrency limits govern simultaneous requests, independently of rate and allowance rules. A refusal is 429 too_many_concurrent_requests, normally with a one-second retry hint. A stream holds its slot until completion or cancellation. Relay account capacity is shared by the underlying account across links; creating another relay does not create more account capacity.

A finite request duration can end work independently of quota. Hosted capacity and deadlines depend on deployment configuration; do not assume a source default is your service entitlement. See Operational recovery for interruption and retry decisions.

Last updated on