Mixed-model sessions

Why opt in

Most conversations mix easy turns and hard ones. A summary, a translation or a field extraction rarely needs the model you picked for planning and code. Mixed-model sessions classify each turn and pick the cheapest model in your pool that scores above your quality floor for that kind of task.

Start in Shadow mode. It needs no code changes, never changes which model answers, and costs you nothing: the classifier runs after the response on our account. The Savings page then shows what each turn would have cost on the suggested model.

Collecting those savings takes development work. In Suggest mode nothing switches on its own: your app has to read each suggestion and send it back to accept it. Changing the mode in the console doesn't do that for you. See From Shadow to Suggest for what to change.

How it works

For each turn from an opted-in key:

1. A small classifier (TypeSafe Jev) reads the system prompt, tool names, the latest user message, and as many earlier turns as fit in 2,000 tokens. It returns one task class and a confidence.
2. Below your minimum confidence, or for unknown, nothing is suggested.
3. Otherwise, the candidates are the models in your pool that are certified, allowed by Directives, support what the request needs (tools, structured output, web search), and score at or above the quality floor for that task.
4. The cheapest candidate is suggested, or your manual pick for that task. Nothing is suggested if it would not save more than the classifier costs.

Task classes: extraction, summarization, rewriting, translation, general_qa, coding, complex_reasoning, and unknown (small talk, acknowledgements, or a mix).

Turn it on

1. Open Dispatch → Mixed-model sessions for the project.
2. Under Keys, set the key to Shadow and save. Each key has its own mode (Off, Shadow or Suggest); keys that are Off ignore everything on this page.
3. Send traffic as usual, ideally with a session id (next section).
4. Review Savings. To collect the savings, change your app to send suggestions back, deploy it, and only then set the same key to Suggest. See From Shadow to Suggest.

Use Edit pool to choose which models a turn may move to, and Model per task to pin a model for one task class. The Open test panel button runs a turn through the classifier without changing your settings.

Group turns into sessions

A session id ties the turns of one conversation together. Pass the same id on every turn, either as ib.session in the body or as an ib-session header. Use whichever your client makes easier: a header needs no body changes, so it works with SDKs that reject unknown fields. Ids are 1 to 128 characters of letters, digits, _, ., : and -. An invalid header is ignored; an invalid ib.session returns 400.

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.inferencebureau.com/v1",
  apiKey: process.env.IB_API_KEY,
});

// One id per conversation. A header works with any client;
// the ib.session body field does the same thing.
const res = await client.chat.completions.create(
  { model: "anthropic/claude-sonnet-5", messages },
  { headers: { "ib-session": conversationId } },
);

With a session id:

• Logs → Sessions shows each conversation, which model served each turn, and why it switched.
• In Suggest mode, a turn sent without a token declines the session's open suggestion, so stale suggestions don't pile up.
• The response echoes it back in the ib-session header.

Sessions are optional. Without one, every turn is still classified, and turns just aren't grouped.

Shadow mode

Every turn is served by the model you requested. After the response, the classifier runs on our account and records what it would have suggested. Nothing is added to the response and nothing is charged. Results appear under Shadow results on the project's Savings page: what Suggest would have done on each turn, and the estimated saving.

Shadow works on chat completions and Messages, streamed or not. Turns in the middle of a tool cycle are skipped. Content is never sent to the classifier when your Directives disallow Jev. When they require zero data retention, it goes only to Jev's ZDR provider.

From Shadow to Suggest

Shadow tells you what you would save. Suggest is how you actually save it, and it needs a code change in the place where your app sends turns. Switching the mode without that change saves nothing: every turn keeps running on the requested model, and with inline confirmation you start paying to classify each one.

1. Choose a confirmation style. Inline returns a suggestion on each response for the next turn and adds no latency. Pre-flight asks POST /v1/suggest before the turn, so the switch applies to that turn and a user can approve it first.
2. Send a session id on every turn, if you don't already. It's optional, but it groups turns in Sessions and stops stale suggestions from piling up.
3. Read the suggestion. Inline: the ib-suggestion response header, which is the only place it appears on streams. Pre-flight: suggestion in the /v1/suggest response. Most SDKs hide response headers by default. The OpenAI SDKs need .withResponse() or with_raw_response.
4. Keep it with the conversation, and send it back as ib.suggestion on the next turn (inline) or this turn (pre-flight). Keep requesting your usual model; we do the swap. Typed SDKs may need extra_body or a type override to send the ib object.
5. Check what changes for you. Accepted turns are answered by a different model. On Messages, earlier thinking blocks are dropped while Suggest is on (see What never changes).
6. Deploy first, then switch the key. While the key is in Shadow no tokens are issued, so the new code just sends nothing extra. Once you set the key to Suggest, it starts accepting suggestions. The same key does both; you never need a second one.
7. Verify. Responses to turns that sent a token carry ib-suggestion-status: accepted, and switched turns appear in Dispatch, Sessions and Savings. If every status is ignored_…, the reason is in the value (see Reference).

Not ready to change the client? Stay in Shadow. It keeps measuring at no cost.

Suggest mode

In Suggest mode the classifier result comes back to your app as a suggestion: a short-lived token naming a cheaper model. Nothing switches on its own. Your app accepts a suggestion by sending the token back with the turn, and Inference Bureau then serves that turn on the suggested model. The request still names your usual model; you never send the suggested model yourself.

Choose how suggestions reach you under Confirmation:

Inline confirmation

The turn is served by the requested model while the classifier runs alongside it. The response carries a suggestion for the next turn: in the ib-suggestion header, and on non-streamed responses also in an ib object in the body. Send that token as ib.suggestion on the next turn to accept it.

let pending: string | null = null;

async function turn(messages: OpenAI.ChatCompletionMessageParam[]) {
  const { data, response } = await client.chat.completions
    .create({
      // Always the model you would normally use. An accepted
      // suggestion swaps it on our side.
      model: "anthropic/claude-sonnet-5",
      messages,
      // @ts-expect-error Inference Bureau extension, removed before dispatch
      ib: { session: conversationId, ...(pending ? { suggestion: pending } : {}) },
    })
    .withResponse();

  // Headers carry the suggestion on streamed and non-streamed responses.
  pending = response.headers.get("ib-suggestion");
  return data;
}

On a non-streamed response the body includes the full suggestion:

"ib": {
  "suggestion": {
    "suggestion": "sug_8x2k…",
    "model": "openai/gpt-5-mini",
    "taskClass": "summarization",
    "confidence": 0.91,
    "reason": "cheapest_passing",
    "estimate": { "baselineUsd": "0.004120", "suggestedUsd": "0.000610", "classifierUsd": "0.000040" },
    "expiresIn": 900
  }
}

Streams send bytes through untouched, so on a stream the suggestion is in headers only. Read them from the response before consuming the body.

Pre-flight confirmation

Before sending a turn, post the same body to POST /v1/suggest. If it returns a suggestion, send the turn with that token to run it on the suggested model; send it without the token to decline. This is the place to ask a user to confirm, since you know the model before the turn runs. Chat completions, Messages and Responses bodies all work, and a suggestion only names a model the turn's endpoint serves.

const turnBody = { model: "anthropic/claude-sonnet-5", messages };

const s = await fetch("https://api.inferencebureau.com/v1/suggest", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.IB_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ ...turnBody, ib: { session: conversationId } }),
}).then((r) => r.json());

// s.suggestion is null when the requested model is already the best choice.
const res = await client.chat.completions.create({
  ...turnBody,
  // @ts-expect-error Inference Bureau extension
  ib: { session: conversationId, ...(s.suggestion ? { suggestion: s.suggestion } : {}) },
});

Suggestion tokens

A token is accepted when it is from the same project, still open, unexpired, and issued for the model the turn requests. Each token works once. A token that can't be used never fails the request: the turn runs on the requested model and the ib-suggestion-status header says why. To decline explicitly, send the token with "decision": "decline"; that also records whether a person or a rule made the call (confirmed_by), which shows up in Sessions.

Tool cycles

While a tool cycle is open (the last message is a tool result, or the assistant's last message called tools) the model is pinned: no classifier call, no charge, and any token sent is left unused for later. The response carries ib-pinned: tool_cycle. Switching resumes on the next user turn, so a tool loop always finishes on the model that started it.

Task hints

If your app already knows what a turn is, for example a dedicated summarize button, send ib.task with the task class. The classifier is skipped and the hinted turn costs nothing extra. Hints only choose the task class: the pool, floor and Directives still decide the model.

Pointing an alias at a suggestion

When your app sends a team alias such as org/fast, suggestions are made for the model it names, and an accepted one changes the model for that turn only. If the same cheaper model keeps winning, you can make it the alias's model and skip the suggestion step:

• The project's Savings page lists an alias when, in the last 7 days, at least 5 accepted turns sent as it ran on the same suggested model and saved money overall, net of the classifier. Say org/fast names anthropic/claude-sonnet-5 and its turns keep accepting anthropic/claude-haiku-4.5.
• Owners and admins see Point org/fast at anthropic/claude-haiku-4.5, then confirm. Nothing changes an alias on its own.
• The change is an ordinary Directives edit at the level that defines the alias (team, project, environment or key): checked against the version the page showed, so a change made meanwhile is refused rather than overwritten, recorded in the audit log, and applied to new requests. A preset keeps its defaults and fallbacks (minus the new model).
• Aliases being retired (deprecatedAt) and aliases whose model changed since those turns aren't offered.

What it costs

The classifier sees at most 2,000 tokens, so a charge is a fraction of a cent. Suggestions already account for it: a cheaper model is only suggested when it saves more than the classifier costs. See Pricing for current prices.

What never changes

• Directives still decide which models and providers may serve a turn.
• A suggested model must support everything the request asks for: tools, structured output, and web search tools.
• The classifier only returns a task label. Text inside a prompt can't add a model to the pool, lower the floor, or pick a model.
• One suggestion per turn, at most. The requested model is always the fallback.
• On Messages, thinking blocks from earlier turns are dropped before dispatch while Suggest mode is on, because a model won't accept another provider's signed thinking. The open tool loop keeps them.

Reference

The ib object, accepted on chat completions, Messages, Responses and /v1/suggest, and removed before the request reaches a provider:

Response headers:

ib-suggestion-status values: accepted, declined, ignored_expired, ignored_model_mismatch (the token was issued for a different requested model), ignored_already_accepted / ignored_already_declined, ignored_unknown, ignored_tool_cycle, ignored_key_not_opted_in (the key is Off), ignored_shadow_mode (the key is in Shadow).

The most common reasons a suggestion is or isn't made, in reason and ib-suggestion-reason: