Why tool calls fail on some providers of the same open model

Open models are served by many providers at once. The weights are the same, but the serving stack is not: each provider has its own chat template, tool-call parser, constrained decoding (or none) and sometimes quantized weights. Those differences rarely matter for plain chat. They matter a lot when your code has to parse the answer.

A request can succeed, bill you, and still be useless to your code. The provider might write the tool call as text instead of returning a structured call, return arguments that aren't valid JSON, or send JSON that ignores the schema you asked for. Your application sees a 200 and then fails somewhere downstream.

How the probes work

Every certified route of a model that more than one provider serves is probed with six fixed tasks: three tool calls and three strict json_schema responses. The tasks are synthetic and never use customer content.

• Tool calls: look up the weather in Paris in celsius, create a 30-minute calendar event with two attendees, and convert 125.5 USD to EUR. Each must come back as a structured call to the right function with the right arguments.
• Structured output: extract a contact into a schema, classify a product review with an enum and a list of aspects, and return exactly the first five primes.

Each answer is graded on what your code would parse. These count as failures:

• a tool call written as text, no tool call, or the wrong function
• arguments or content that aren't valid JSON
• wrong values, or extra keys the schema doesn't allow
• a provider rejecting the tools or schema it says it supports

Rate limits, server errors, timeouts and answers cut off by max_tokens aren't counted, since they say nothing about how a provider handles tools.

What the probes found

The first full sweep ran on Oct 19, 2026, with three runs of each task per route.

Tool calls are mostly solved: only five routes fell below half. Strict JSON schemas are not. Over half of the routes failed most structured-output tasks, and only 101 routes passed every task of both kinds. The most common failures were content that wasn't valid JSON and extra keys outside the schema, such as an answer or analysis field the model added on its own.

The spread within a single model is the point. For zai/glm-5.2, four providers passed every JSON task while ten others passed one or none, including the model's own maker. For deepseek/deepseek-v4-pro, four of six providers passed everything. For openai/gpt-oss-120b, three providers passed every task and three passed one in three.

These numbers come from one sweep and will move as providers update their serving stacks, which is why routing reads the most recent results rather than a fixed list.

How routing uses the results

When a request sends tools or a json_schema response format, Dispatch orders the model's providers by how often they pass these probes, best first. A provider that fails at least half of its recent probes is skipped, as long as another provider of the same model does better. If every provider falls below half, they're all kept, since then the model behaves that way wherever it runs.

A few things never change:

• The model is always the one you asked for. Reliability only chooses among its providers.
• Your Directives, price ceilings and capability requirements still apply first.
• Requests without tools or a schema, including json_object, are routed as before.

Seeing it on your requests

When a provider is skipped, the request's receipt in Logs names it with unreliable_tools or unreliable_structured_output. If you also care about quantization, set routing.minPrecision in Directives to keep requests off providers that serve fp4 or fp8 weights; those show below_min_precision instead.