Routing

Model aliases

A request may name a team alias such as org/fast instead of a model id. The alias is resolved first, before Directives, certification or planning look at the request, so everything below applies to the model it names. A preset alias can also fill request defaults and name its own fallback models. A deprecated alias runs until its date with an ib-alias-deprecated response header, then runs as its replacement (Retiring an alias). An alias names a model: Dispatch never picks the model for it. See Model aliases.

Allowed-model lists may name a whole vendor with vendor/*; a route is planned only if its model is allowed at every level (Allowing a vendor), and fallback models are checked the same way.

Providers and profiles

Many models are served by more than one provider: anthropic/claude-sonnet-5, for example, by Anthropic, Amazon Bedrock and Google Vertex AI, among others. Only providers that passed certification for a model are used, and Dispatch tries them in a fixed order. Directives apply first: to keep a model off a provider, leave the provider out of allowedProviders in Directives, and it is never used. deniedProviders does the same and wins over the allowed list. requireZdr narrows the list to zero-data-retention providers in the same way, and requireNoTraining to providers listed as not training on prompts; both follow the current upstream listing, rechecked every hour, and each request also asks the upstream gateway to refuse a provider that doesn't qualify. A provider skipped by a Directive is listed on the receipt with the reason (policy_denied, no_zdr or no_training).

A request also goes only to providers that support what it asks for: tools, structured output, vision, and the web search tool version it sends. Models lists each model's providers and prices, and the project's Dispatch page shows which models and providers served its recent traffic.

Tool-call and JSON reliability

Providers of the same open model don't all handle tools and JSON schemas equally well. A provider can return a tool call as plain text, arguments that aren't valid JSON, or JSON that ignores the schema you sent, and still bill for the request. So certified routes are also probed with fixed tool-call and structured-output tasks, and graded on what your code would parse.

When a request sends tools or a json_schema response format, Dispatch tries a model's providers that pass these probes most often first. A provider that fails at least half of its recent probes is skipped, with unreliable_tools or unreliable_structured_output on the receipt, as long as another provider of the model passes more. If none does, Dispatch keeps them all, since then the model behaves that way wherever it runs. Requests without tools or a schema, including json_object, are routed as before.

Weight precision

Some providers serve open models with quantized weights, such as fp8 or fp4, which can be faster and cheaper but less accurate. Set routing.minPrecision in Directives to "fp8" to skip fp4 providers, or to "bf16" to skip fp8 ones too. Precision is as each provider lists it; providers that list none, such as a model's own maker, are kept. Skipped providers show below_min_precision on the receipt. The highest precision any level sets applies.

Provider failover

If a provider fails before it starts answering, with a connection error, a rate limit, or a server error, Dispatch tries the next provider for the same model. A model with a single provider gets one retry on it after a short wait. A request makes at most three attempts; set routing.maxApplicationAttempts in Directives to allow fewer. Providers that keep failing are paused for a short cooldown so other requests skip them.

Some failures are not retried, because another attempt would fail the same way or could bill twice: a request the provider rejected as invalid, and a provider that stopped answering after the request reached it. If every attempt fails, the response is the last attempt's error, such as 429 upstream_rate_limited or 503 no_healthy_route (see Errors). Attempts that may have billed keep their hold until the provider's usage is known.

Every attempt is listed on the request's receipt in Logs.

Model fallback

Falling back to a different model is off by default. Turn it on in Directives with routing.allowModelFallback and name the models that may stand in, in order, with routing.fallbackModels. A team sets whether fallback is allowed; a project can turn it off or choose its own models. A preset alias with fallbackModels uses its own list for its requests, with fallback on unless a level turned it off. Each fallback model must be allowed by your Directives and must accept the original request unchanged: same tools, same output constraints, enough context.

The last attempt of a request is kept for the first fallback model that can take it, so with the default of three attempts the requested model is tried on up to two providers first. The hold covers every planned attempt at its own model's price. When a fallback model answers, the response's model field names it, the response carries an ib-fallback-model header, and the receipt lists each attempt. A request under /v1/messages can only fall back to a model that endpoint serves.

Streaming

Once a stream has started, it can't move to another provider. If it fails partway, the stream sends an error event with the code stream_interrupted and ends without a [DONE] marker. Treat a stream without one as incomplete. A stream that sends nothing for 30 seconds is ended the same way.

Mixed-model sessions

Opt a key in and each turn is classified by task, so easy turns can run on a cheaper model that still meets your quality floor. Start the key in Shadow mode: it is free, needs no code changes, and shows projected savings before any request changes. Switch the same key to Suggest and your app accepts or ignores each suggestion. See Mixed-model sessions.