Routing and caching make the gateway more reliable and cheaper. None of them widens access: every model a route or fallback reaches is decided again for the caller, and a cached answer is served only to someone who could have made the same request.
Retries, timeouts and the circuit breaker
Set per organisation on Models > Gateway endpoints, or with GET/PUT /api/v1/models/gateway-settings (models:view to read, models:admin to change). PUT replaces every field.
| Setting | Default | Range | Meaning |
|---|---|---|---|
retry_attempts | 2 | 0 to 5 | Retries of a transient failure on the same model |
retry_base_ms | 250 | 10 to 10000 | Base of the exponential backoff with full jitter |
retry_max_ms | 4000 | base to 60000 | Cap of one backoff wait |
upstream_timeout_ms | 60000 | 1000 to 600000 | Wait for the provider's response headers; a stream that started is never cut |
breaker_failures | 5 | 0 to 100 | Consecutive failures that open a model's circuit; 0 disables the breaker |
breaker_cooldown_seconds | 30 | 5 to 3600 | How long an open circuit skips the model |
| Failure | Same model again | Next candidate |
|---|---|---|
| 429, 408, 5xx, timeout, unreachable | Yes, up to retry_attempts | After the retries |
| 401, 403 (rejected key), 404 (unknown model), refused by the outbound guard | No | Yes |
| Any other 4xx (the caller's request) | No | No, returned to the caller |
| Budget or plan refusal | No | No |
A provider's Retry-After replaces the backoff when it is not longer than retry_max_ms; a longer one moves to the next candidate at once. A client can lower retries for one request with X-Keystone-Retries, never raise them. Nothing is retried once a byte of the answer reached the caller. Every attempt is its own budget reservation and usage event.
The circuit breaker is per server process, organisation and model. After the threshold the model is skipped for the cooldown, then one probe request is let through. When every candidate is skipped this way the caller gets 503 circuit_open with Retry-After. Circuit state starts empty after a restart.
Fallback chains
Each catalog model can have a fallback chain of up to 20 models (PUT /api/v1/models/catalog/{id}/fallbacks). When the upstream fails before any byte reached the client, the next model is tried. Each fallback is decided again (assignment, classification, region, budget), so a fallback can never reach a model the caller may not use.
Routes
A route is a virtual model name over 1 to 20 catalog models. Clients name it wherever they name a model, and GET /v1/models lists the routes the caller can use. Route keys and model keys share one namespace.
Open Models > Routes and strategies, or POST /api/v1/models/routes (models:write):
{
"route_key": "assistant",
"display_name": "Assistant",
"strategy": "consistent_hash",
"hash_key": "user",
"targets": [{"model_id": "<gpt>", "weight": 3}, {"model_id": "<claude>", "weight": 1}],
"conditions": [
{"match": {"classifications": ["restricted"]}, "targets": [{"model_id": "<on-prem llama>"}]},
{"match": {"metadata": {"tier": "batch"}, "max_tokens_at_least": 4000}, "strategy": "least_cost",
"targets": [{"model_id": "<haiku>"}, {"model_id": "<mini>"}]}
]
}| Strategy | Order of the targets |
|---|---|
fallback | As configured |
weighted | Weighted random first pick, the rest as fallbacks (load balancing, canaries, A/B) |
least_latency | Unmeasured targets first, then by moving average time to response headers |
least_cost | By catalog input plus output price |
round_robin | In turn, by weight |
least_busy | Fewest requests in flight first |
consistent_hash | Weighted rendezvous hashing on hash_key: user (default), api_key or metadata:<name>; one key stays on one target while it is usable |
Latency, in-flight counts and round-robin counters are per server process.
Conditions
A route can carry up to 20 ordered conditions, each with its own targets and strategy. The first condition whose match fits the request decides; a request no condition matches goes to the route's own targets. A match can test:
| Field | Fits when |
|---|---|
uses | The request's use is listed |
classifications | The request's classification is listed |
project_ids | The request names a listed project |
team_ids | The caller belongs to a listed team |
metadata | Every listed pair equals a value of X-Keystone-Metadata |
max_tokens_at_least, max_tokens_at_most | The requested output limit lies in the range |
A condition never widens access. When the matching condition has no target the caller may use, the request is refused; it does not fall through to the route's own targets. A route ignores its targets' own fallback chains: the route is the chain. A disabled route refuses with model_not_allowed.
GET /api/v1/models/routes/{id}/effective shows how a route resolves. Route changes are audited (models.route.create, .update, .delete).
Response cache
The cache is off by default. It needs the deployment's key management service: without it, turning the cache on is refused (503 models.cache.unavailable).
Turn it on in Models > Response cache or with the gateway settings:
| Setting | Default | Meaning |
|---|---|---|
cache_mode | off | off, exact (identical requests) or semantic (identical requests, then a semantically equal last message) |
cache_scope | user | user: each person or key has their own entries, per project. project: members of a project share. org: the whole organisation shares. |
cache_ttl_seconds | 3600 | 60 to 2592000 |
cache_max_classification | internal | Requests of a stricter classification never read or write the cache |
cache_semantic_threshold | 0.95 | 0.8 to 0.999 |
cache_embedding_model_id | Required for semantic: an enabled catalog model with the embedding use |
Rules that keep the cache safe:
semanticworks withuserscope only.- The request key is the request after DLP, so a cached answer was produced for exactly what policy let through.
- A hit is served only while the model that produced it is still one the caller may use.
- Answers paid with a person's own provider key are stored only under
userscope. - Entries are sealed with AES-256-GCM under a data key wrapped by the organisation's key
models-gateway-cache. - A hit still counts on rate limits and budgets, is audited with the cost it saved, and is recorded as a zero-cost usage event.
- Changing any rule's output guardrails deletes the organisation's cached answers.
- An organisation's cache holds at most 20,000 entries and 512 MiB; older entries are evicted first.
Clients can skip the cache for one request with X-Keystone-Cache-Control: no-cache (do not read) or no-store (do not write). GET /api/v1/models/cache/stats (models:view) reports entries, hits and savings; POST /api/v1/models/cache/flush (models:admin) deletes every entry, or those of one model or route with {"model": "<key>"}.
Advanced routing in the AI Gateway service
The AI Gateway service adds reliability settings (GET/PUT /api/v1/models/gateway-reliability), hedged requests, traffic mirroring and canaries on AI Gateway > Routing and reliability. These are recorded as working in the source; see the console page for the options your deployment offers.