Budgets and rate limits

Budgets cap how much is spent or how many tokens are used in a day or month. Rate limits cap how fast requests or tokens flow. Both are enforced by the model gateway on every request, before anything is sent to a provider, and both hold across any number of server replicas.

Budgets

Open Models > Budgets, or use POST /api/v1/models/budgets (models:write).

Gateway keys and authorizer usage plans can carry their own budgets on top; see Gateway keys and workload identity.

How enforcement works

1. Reserve. Before a request is sent, the gateway reserves its worst case on every applicable budget in one database transaction, with the counters locked in a fixed order. A request that would cross a hard limit is refused with 429 budget_exceeded. Concurrent requests can never reserve more than a hard limit.
2. Worst case. Input tokens are estimated at one token per three bytes of prompt text plus fixed overheads (1,600 tokens per image). Output tokens are the request's max_tokens, capped by the model's max_output_tokens; when the client sends none, the gateway sets it to the model's maximum. Requests for several completions reserve once per completion.
3. Settle. After the response, the reservation is replaced with the usage the provider reported. Reported usage is never capped: an overrun is charged, logged and audited. If the provider reports no usage, the estimate is charged.
4. Unfinished requests. A reservation not settled in time is settled at its reserved amount by a sweep (NI_AUTH_MODELS_RESERVATION_HOLD, default 30m). Charging the worst case is the fail-closed choice.

Crossing a warning threshold or a soft limit adds X-Keystone-Budget-Warning to the response, and the first crossing in a period emits a models.budget.warning event.

Spend on a person's own provider key is not counted against spend budgets, because it is not the organisation's money; tokens are counted. Cache hits reserve a token and a micro-dollar, so a used-up hard budget refuses them too.

GET /api/v1/models/budgets returns each budget with its live usage (models:view). Settled reservations are kept for NI_AUTH_MODELS_RESERVATION_RETENTION (default 90 days); budget counters, usage events and the audit log keep the totals.

Model Access budgets are the only AI budgets Govern enforces. The older cost explorer AI budgets were imported into them and their routes answer 410.

Rate limits

Open Models > Rate limits, or use POST /api/v1/models/rate-limits (models:admin).

curl -sS -X POST "$KEYSTONE_URL/api/v1/models/rate-limits" \
  -H "Authorization: Bearer $ADMIN_TOKEN" -H 'Content-Type: application/json' \
  -d '{"name": "Interactive users", "subject_kind": "user", "metric": "tokens",
       "window_seconds": 60, "limit": 200000}'

An organisation has at most 200 rules.

How enforcement works

• A request is checked against every enabled rule that applies; all must admit it.
• A rule on a model key counts every request sent to that model however it was named: directly, as a fallback, as a route target or from its cached answer. Naming a route or another model never avoids it. A model whose rule refuses is skipped like a model whose circuit is open; the request is refused with 429 rate_limited and Retry-After when no candidate can serve it.
• requests cost 1 per request that reaches dispatch. tokens cost the request's estimate before dispatch and are corrected to actual usage afterwards.
• Counters use a sliding window and are checked and charged in one PostgreSQL transaction with rows locked, so two requests can never both take the last of a limit on any number of replicas.
• If the counter store cannot be read or locked in time, the request is refused with 503 service_unavailable. Rate limiting fails closed.
• Responses carry X-RateLimit-Limit, X-RateLimit-Remaining and X-RateLimit-Reset for the rule with the least room left.

Tag rules are attribution, not enforcement

The client decides whether to send a tag in X-Keystone-Tags, so a request that leaves it out is not counted by the tag's rule. Use tag rules to shape the traffic of cooperating clients, such as a batch job. For a limit that must hold, use a user, api_key, project, team or org rule.

Routes

Changing a rule's window or model resets its counters; disabling it stops counting at once. Every change is audited (models.rate_limit.create, .update, .delete).

Plan limits on the hosted service

On the hosted service, plan limits also apply (for example the daily gateway tokens). A self-hosted installation has no plan limits.