B2B AI structured outputs in 2026: JSON Schema, tool calls, or humans?

Most production AI features do not fail because a model forgot what JSON looks like. They fail because a team treated a correctly shaped object as if it were a correct business decision.

A response can parse cleanly, match every required field, and still name the wrong customer, invent a value that was not in the source, select an action the user is not allowed to take, or quietly omit the one fact that should have sent the case to a person. That is why structured output is useful, but not magical.

My practical answer for a B2B product in 2026 is:

• Use plain text when a person is the only consumer and no downstream code needs a contract.
• Use provider-constrained structured output when application code needs a predictable result for a read-only or recommendation step.
• Use a typed tool call when the model is proposing an operation that your application must execute, authorize, and record.
• Always validate in application code for syntax, schema, meaning, evidence, and policy. Provider constraints handle shape, not your business truth.
• Repair, retry, or ask a human according to the consequence of being wrong. Never let a generic retry loop become an accidental authorization system.

That is the whole decision in miniature. The rest of the article makes the boundaries concrete: what each layer can promise, where current provider features differ, how to handle failures, and how to run a pilot before a malformed or merely plausible object reaches a system of record.

This is not a claim that one model or SDK is always better. It is a way to decide what your product needs to prove before it accepts an AI-produced value or action.

The short answer: shape is not meaning

Think of an AI response as moving through four gates.

Structured output mostly improves the first two gates. It can make a response easier to parse and easier to hand to a type checker. It does not know whether the ticket really belongs to the account in the object, whether the invoice total agrees with the source, or whether the person who asked is allowed to approve a refund.

Consider a support triage response like this:

{
  "category": "refund",
  "priority": "low",
  "account_id": "acct_4821",
  "next_step": "draft_reply",
  "needs_review": false
}

Every field can have the right type. The object can pass a strict schema. It can still be wrong if the message came from acct_9120, if the customer actually reported a security incident, or if the account has a policy that requires review for every refund. A schema is a contract about representation. Your application still owns the contract about reality.

That distinction is the best reason to publish an article about structured outputs at all. The feature is valuable enough to use, but easy enough to misunderstand that the surrounding checks matter more than the impressive word “strict.”

What structured output actually controls

JSON Schema is a standard vocabulary for describing and validating JSON documents. Model providers use it as an interface for constrained generation, but they do not all implement the same complete dialect. OpenAI’s current documentation distinguishes JSON mode from Structured Outputs: both can produce valid JSON, while only Structured Outputs is intended to adhere to a supplied schema. The same documentation calls out refusals, incomplete responses, unsupported schema features, nesting limits, and required object settings.

Anthropic’s current platform documentation describes two related capabilities: JSON outputs for the shape of Claude’s response and strict tool use for the parameters in a tool call. It also documents grammar compilation, first-request latency, caching, schema limitations, refusals, and edge cases such as enum capitalization. Google’s Gemini documentation makes the semantic boundary especially explicit: structured output can produce syntactically correct JSON, but the application should still validate the values.

The durable lesson is not “provider A is strict and provider B is loose.” It is that every provider feature has a capability contract. Record the exact model, endpoint, schema subset, refusal behavior, tool combination, streaming behavior, and SDK parser that your workflow relies on.

Notice what is missing from the table: “the model sounded confident.” Confidence language can be useful context, but it is not an independent check. A field called confidence does not turn a guess into a fact. If the downstream consequence matters, measure the outcome and preserve the evidence that made the decision possible.

The four lanes: text, structured result, tool call, or human

Teams often ask whether they should use JSON or tools as if those were competing formatting preferences. They solve different problems.

Lane one: plain text for a human-facing answer

If the only consumer is a person, plain text may be the most honest contract. A support assistant that explains a policy, a research note that compares options, or a draft that a person will rewrite does not need to pretend that every sentence is an API object.

Use a stronger structure when the interface needs it—headings, bullets, citations, or a few named fields—but do not add a complex schema merely because the word “agent” appears in the product brief. A schema can make the application harder to change without making the answer more useful.

Lane two: structured output for a result

Use a structured result when the model is returning an interpretation that another part of the application will inspect without immediately causing a side effect. Classification, extraction, routing recommendations, UI cards, and a decision brief are common examples.

The application should be able to say, “this object is well-shaped, the evidence is present, the values pass domain checks, and it is safe to show or queue for review.” It should also be able to say, “this object is well-shaped but not accepted.” That second outcome is not a failure of the schema. It is the point of having a business validator.

Lane three: a tool call for a proposed operation

Use a tool call when the model needs to ask the application to do something. OpenAI’s function-calling guide draws this boundary directly: use function calling to connect the model to tools, functions, data, and systems; use structured response formats when you want to structure the model’s response.

Anthropic describes tool use in similar terms. The model emits a structured request, your code or the platform executes the operation, and the result returns to the loop. The model does not execute a client-side tool merely by naming it.

That separation is important. A tool call can be perfectly typed and still be unauthorized. The execution layer must verify identity, tenant, target, arguments, scope, approval state, rate limit, and whether the operation has already happened. The tool schema describes what the call looks like. It does not grant permission to make the call.

Lane four: human review or abstention

Human review is not an apology for an immature AI system. It is a product choice for work where the cost of a wrong value, wrong target, or silent omission is larger than the cost of a pause.

Make the review decision specific. Show the proposed action, the exact arguments, the source evidence, the account or tenant, the policy result, and what will happen after approval. If an argument changes, the old approval should not silently carry forward. If a reviewer cannot tell what the action will do, the interface is not a meaningful approval boundary.

Design one canonical business contract before choosing a provider

Do not begin with a provider’s schema helper. Begin with the business object you want your product to understand.

For a lead triage workflow, the canonical contract might contain:

• decision: qualified, not_qualified, needs_review, or unknown;
• reason_code: a short, versioned domain value rather than a paragraph that must be parsed later;
• evidence: source record IDs or quoted spans that a reviewer can locate;
• missing_information: what prevents a stronger decision;
• tenant_id: supplied and checked by application context, not invented by the model;
• requires_review: a routing signal that is checked against policy, not a permission grant;
• contract_version: the application contract that accepted the object.

The model may fill some of those fields. The application should fill or verify others. In particular, do not ask the model to choose its own tenant boundary, permission scope, or authoritative record identifier when your application already knows them.

Keep the canonical contract separate from provider syntax. One provider may require every object to set additionalProperties to false. Another may have different limits for arrays, unions, or nested objects. A third may support a helper that validates the returned object locally but cannot express all the same constraints in the provider request. Generate or adapt the provider schema from the canonical contract, then validate the returned value against the canonical contract again.

canonical business result
  - decision
  - evidence references
  - missing information
  - abstain or review reason
  - contract version

provider adapter
  - supported schema subset
  - tool or response format
  - refusal and incomplete-output mapping
  - model and endpoint identity

application acceptance
  - parse and schema validation
  - semantic and source checks
  - identity and policy check
  - accept, repair, abstain, or review

This may feel like extra ceremony for a small prototype. It is usually less work than discovering six months later that your prompts, database fields, and billing rules all assume different meanings for unknown.

When provider-constrained JSON is the right first move

Provider-constrained output is a strong first move when the model is interpreting messy input into a bounded, mostly read-only result.

Good examples include:

• classifying a support ticket into a small set of queues;
• extracting fields from an email before a person verifies them;
• returning a UI card with a known set of sections;
• turning a free-form request into a draft plan with explicit unknowns;
• producing a recommendation that another application step will inspect.

Use a small schema. Each field should have one job. Prefer an explicit unknown or needs_review value to forcing the model to select the closest known category. Include descriptions for ambiguous fields. Define whether an empty array means “nothing found,” “not checked,” or “the model failed,” because those are different states.

Keep a refusal and incomplete-response branch. OpenAI’s guide notes that a refusal or a max-token interruption can produce a response that does not match the requested schema. Anthropic’s documentation describes similar invalid-output situations and notes that schema compilation has its own latency and caching behavior. A parser that only handles the successful object is not production handling; it is a demo path.

Measure the whole result:

• Was the response parseable?
• Did it match the accepted provider and application schema?
• Did the values agree with the source or system of record?
• Did the model abstain when the source was insufficient?
• How much reviewer correction was needed?
• Did the structured path add noticeable latency or token cost?
• Did a model or provider change alter the accepted outcome?

If the answer is “we only measured whether JSON.parse succeeded,” you have measured the envelope, not the job.

When a tool call is the better contract

A tool call is the better contract when the next step is an operation rather than an answer. The model can propose create_draft_invoice, lookup_customer, or schedule_follow_up. Your application decides whether that function is available, whether the arguments are allowed, and what actually happens.

Use the narrowest tool that represents the real job. A tool named run_any_sql has a large and difficult-to-review capability surface. A tool named find_open_invoices can constrain inputs, queries, tenant scope, and output shape more directly. A tool named send_message should make the destination, sender, content, approval, and idempotency key explicit.

Separate these three records:

1. Proposal: the model’s structured request, including the tool name, arguments, and source context.
2. Policy decision: the application’s answer about identity, scope, approval, rate, and target.
3. Effect: what the tool actually changed or returned, with a provider or system reference.

A tool call is not a shortcut around validation. Strict tool use can reduce malformed arguments, but your application still needs to check that a valid argument refers to a real object in the right tenant and that the current identity may act on it.

For an external write, bind the policy decision to the exact proposal. If the model changes the recipient, amount, record, or permission scope after approval, require a new check. If a timeout occurs after an external system may have accepted the request, do not blindly retry. Look up the idempotency key or reconcile the system of record first.

The earlier B2B AI agent identity guide covers the difference among user delegation, workload identity, and an agent or task principal. The principle here is narrower: structured arguments make an action inspectable, while identity and policy decide whether it is allowed.

Application validation is mandatory

Validation is not one check. It is a stack.

Google’s current documentation says to validate values in the application because syntactically correct JSON may still be semantically wrong. That is a compact version of the whole architecture.

Public implementation signals show why this matters. A current LangChain issue describes a raw JSON-schema dictionary path where a schema-violating value can pass through without the validation error needed to trigger a retry. A Vercel AI issue describes an intermittent provider path that emits an internal XML-like tool-call dialect instead of the expected JSON, sometimes repeating on retries. These reports are not universal benchmarks, but they are exactly the kind of integration edge case a production contract test should expose.

Do not hide validation inside a helper you have never tested. Feed the helper a deliberately wrong type, an unknown enum, an omitted field, a stale record, a cross-tenant identifier, and a refusal. Confirm that the correct failure reaches the branch that is supposed to handle it.

Repair and retry: bounded recovery, not an infinite hope loop

There are at least three different things engineers call a retry:

1. Transport retry: the provider request failed or timed out before a usable response. This needs backoff and a safe repeat policy.
2. Shape repair: the response arrived but could not be parsed or validated. A second model call may be reasonable for a side-effect-free task.
3. Task retry: the whole workflow starts again after an error. This is dangerous when a previous tool call may have partially succeeded.

Keep those counters separate. A provider retry can be invisible to the product user. A semantic repair should leave a trace. A task retry should reconcile prior effects before it attempts another write.

A bounded repair loop can look like this:

1. Preserve the original raw response and the validation error.
2. State the one or two fields that failed instead of asking for a brand-new answer with a vague apology.
3. Keep the same task ID and contract version.
4. Do not add new authority, tools, or data merely because the first output was invalid.
5. Stop after a small, explicit budget and route to needs_review or an abstention state.

Repair is a poor fit when the problem is missing evidence, a refusal, an authorization denial, a changed system-of-record value, or an external write whose outcome is unknown. Repeating the same request does not create evidence or permission.

The AI agent background jobs guide covers idempotency, partial completion, reconnects, and durable state in more detail. The important connection is that output handling is part of job handling. If a valid-looking object can start a side effect, the retry design must know whether that side effect happened.

The business truth problem

The most expensive failures are often not malformed. They are plausible.

Support triage

A model classifies a ticket as billing_question, returns a valid queue name, and suggests a draft reply. The category is well-shaped. The ticket may still contain a hidden account takeover signal that the classifier missed. The system should check for known risk indicators, preserve the source record, and route uncertainty to a person rather than treating the enum as proof.

Document intake

A model extracts an invoice total as a number with two decimal places. The number may belong to the subtotal, not the total; it may omit a page-level fee; or it may be copied from a related purchase order. The B2B document AI guide makes the same broader point: structured JSON is not provenance. Preserve the original, page or region evidence, validation result, and review status.

CRM routing

A model returns region: "Midwest" and owner_id: "user_17". Both strings can be valid. The region may be inferred from an outdated address, and the owner may not have access to the customer’s segment. The application should look up authoritative fields, apply routing policy, and treat the model result as a proposal.

External action

A model proposes a valid send_email call. The address may have been copied from untrusted content, the message may include confidential context, or a retry may send it twice. The typed arguments help a reviewer inspect the proposal. They do not remove the need for destination allowlists, identity checks, approval, redaction, and idempotency.

These examples all share one pattern: the schema describes a shape, while the product owns the truth conditions. Write those truth conditions down before choosing a model feature.

Multiple providers: standard vocabulary, non-standard behavior

JSON Schema gives teams a shared vocabulary, which is valuable. It does not mean a schema can be copied into every provider and behave identically.

OpenAI documents a supported subset with keyword and size limits. Anthropic documents a constrained grammar path with its own limitations and caching behavior. Google documents a JSON Schema subset and separately warns that values need application validation. SDKs add another layer: one helper may parse into a typed object, another may return raw content, and a framework may choose a tool-based strategy behind the scenes.

Public practitioner discussion points at the same portability problem. One current thread describes testing constraints across providers and recommends generating provider-specific schemas from a canonical semantic model. That is a useful implementation signal, not a universal benchmark. The correct response is not to declare portability impossible. It is to make it testable.

Keep a capability matrix with at least these rows:

• schema dialect and unsupported keywords;
• object, array, union, enum, and nullable behavior;
• tool calls combined with a structured final response;
• refusal and max-token behavior;
• streaming and partial-object behavior;
• schema compilation or first-request latency;
• SDK parsing and error types;
• model, endpoint, and adapter identity;
• observed behavior on representative, adversarial, and recovery cases.

Then replay the same cases. A provider-neutral application contract is worthwhile only if the application can see and handle the differences rather than quietly normalizing them into a false sense of equivalence.

Schema and tool design: small is a reliability feature

A large schema can look thorough while making the model’s job and the application’s test matrix harder. Research on structured output and tool-schema compilation treats schema size and representation as real deployment concerns, not only style preferences.

Start with the smallest object that lets the next application step make a good decision. A support classifier probably does not need the entire customer profile, a prose essay, three redundant confidence fields, and a copy of the input. A tool probably does not need to accept arbitrary query text if a bounded filter can do the job.

Useful design habits:

• Give fields names that describe the decision, not the prompt wording.
• Use enums for small closed sets, with an explicit unknown or review value.
• Keep free-form explanation separate from machine action fields.
• Include evidence references instead of asking the model to restate an entire source.
• Version the contract when meaning changes, not only when a TypeScript type changes.
• Make read-only tools and write tools visibly different.
• Use descriptions and examples for ambiguous fields, then test whether they change outcomes.
• Do not put secrets, access tokens, or tenant policy into a model-generated field.

Schema design is part of product design. If reviewers need to understand why a result was accepted, a short reason code plus source references may be better than a long explanation field that no one can reliably compare.

Which contract fits common B2B jobs?

The smallest useful architecture is often hybrid. A model produces a structured recommendation. Rules verify known constraints. A person reviews exceptional cases. A tool executes only after identity and policy checks. The model does not need to own every stage for the feature to be valuable.

Build, buy, or postpone?

You do not need a general AI gateway on day one. You do need an explicit place where the application decides whether an output is accepted.

A purchased platform can reduce repeated plumbing. It cannot make an undefined business truth well-defined. Before selecting a gateway, ask whether you can export the canonical schema, validation failures, raw response metadata, provider route, contract version, and review outcome. Portability includes your evidence, not only your API key.

A 30-day pilot that can settle the choice

Pick one repeated job with one downstream consequence. “Return structured answers” is not a job. “Classify inbound support tickets into five queues, preserve evidence, and abstain when identity or urgency is unclear” is a job.

Days 1–5: define the contract and baseline

• Write the input, accepted result, unknown state, abstain state, and definition of done.
• Mark each step as read, recommend, reversible write, or external commitment.
• Collect normal, messy, empty, contradictory, adversarial, and recovery cases.
• Run the current human or deterministic path so you know what “better” means.
• Write the canonical contract before writing the provider adapter.

Days 6–12: test the simplest model path

Start with plain text or a small structured result. Measure parse failures, schema failures, semantic corrections, abstentions, reviewer minutes, latency, and cost. Do not add a tool call because the model’s answer is interesting. Add it only when the next step genuinely needs a system operation.

Days 13–19: test constraints and failures

Turn on provider-constrained output or strict tool parameters. Then test unsupported schema features, long and nested objects, refusals, max-token truncation, empty values, unknown enums, provider errors, streaming or partial responses, and a deliberately wrong but schema-valid answer.

If you support more than one provider, replay the same cases through each path. Record the provider, model, endpoint, SDK, schema version, request mode, raw failure class, and accepted result. A test that only asserts “object returned” is too weak.

Days 20–26: add repair, review, and recovery

Implement a bounded repair path for side-effect-free failures. Create a review state for missing evidence or semantic ambiguity. Add idempotency and reconciliation before any write. Have a reviewer approve the exact proposal and then change one argument to verify that the old approval cannot be reused.

Days 27–30: choose the contract stack

Compare paths on the same rows:

1. parse success and schema acceptance;
2. semantic acceptance against a reviewed sample;
3. abstention quality and missing-evidence behavior;
4. review minutes and correction types;
5. latency, token use, and retry cost;
6. tool-call correctness, if tools are involved;
7. duplicate or partial side effects;
8. provider portability and contract drift;
9. auditability and recovery effort.

Pick the smallest stack that improves the whole job. If provider-constrained output adds shape without reducing downstream correction, keep the simpler path. If a tool call makes the action boundary visible and enforceable, use it. If semantic uncertainty remains expensive, keep a person in the loop. If you do not know what success means yet, postpone the automation and improve the baseline.

What is likely to change next

This section is inference, not a reported fact or a guarantee.

Provider APIs will probably keep moving toward more explicit structured-output and tool-use contracts. SDKs will likely make typed schemas, adapters, and repair hooks easier to write. Benchmarks will get better at separating schema conformance from answer correctness. Tool and schema compilation will matter more as agent context budgets become crowded.

That progress will not eliminate the business-truth problem. Providers can constrain a model to emit a date, but they cannot know whether the date belongs to the right contract unless the application supplies the authoritative context and checks the result. They can constrain a tool argument, but they cannot decide whether the current identity may change that record. They can report a structured refusal, but they cannot define your acceptable abstention policy.

The durable product asset is therefore not a clever prompt or a provider-specific helper. It is the combination of a versioned business contract, a provider capability record, semantic checks, a review policy, and evidence of what happened. The model and syntax may change. That decision boundary should remain understandable.

FAQ

Are structured outputs guaranteed to be correct?

No. They can make the response more likely to match a supported structural schema, but that is not proof that the values are true, current, complete, authorized, or useful. Validate the result against source data, domain rules, system-of-record state, and the consequence of using it.

What is the difference between JSON mode and structured output?

JSON mode generally targets valid JSON. Structured output adds a schema constraint where the provider and model support it. The exact supported schema subset, refusal behavior, limits, and SDK parsing still vary. Keep application-side validation even when the provider describes the result as schema-conformant.

Is a tool call the same as structured output?

No. A structured result describes an answer or recommendation. A tool call asks an application or platform to perform an operation. Tool arguments can be schema-valid and still unauthorized, stale, or pointed at the wrong tenant. Treat execution as a separate policy and audit boundary.

Do I still need to validate provider output?

Yes. Validate parsing, schema, semantic values, evidence, system-of-record state, identity, and policy. A provider constraint reduces one class of failure; it does not know your business rules or whether an external side effect is allowed.

Should I retry invalid structured output?

Sometimes, for a bounded, side-effect-free task where the failure is local and the validation error can be made precise. Do not retry refusals, missing evidence, authorization failures, or writes with an unknown outcome as if they were formatting errors. Preserve the original response and stop after an explicit budget.

How complex should my schema be?

As small as the next application step allows. Use explicit fields, narrow enums, an unknown state, evidence references, and a version. Split one giant object into stages when different fields have different sources, permissions, or review policies. A large schema is not automatically a thorough schema.

Do I need a separate schema for every provider?

Keep one canonical business contract, then generate or adapt provider-specific schemas when their supported subsets differ. Test the exact provider path. A common JSON Schema vocabulary helps with portability, but it does not prove identical decoding, tool behavior, refusals, or SDK errors.

When should a human review the result?

Review when the consequence is high, the source evidence is incomplete or conflicting, the result changes a system of record, or the action is difficult to reverse. Make the review show the exact proposal, target, evidence, identity, and policy result. Human review is strongest when it is specific and auditable rather than a generic approval button.

The practical next step

Choose one workflow and write down three things: the smallest object the next step needs, the evidence that would prove each important field, and the consequence if the value is wrong. Then decide which lane belongs at each boundary: text, structured result, tool call, or person.

If you have a real workflow, provider choice, output contract, or downstream action that feels harder than the demo suggested, bring the messy version. BishopTech can help turn it into a scoped contract, validation path, evaluation set, and first build that keeps the uncertain parts visible.

Start a scoped BishopTech consultation

Sources and further reading

The links below are dated references for the provider behavior, standards, research, and public implementation signals discussed here. Official documentation describes each provider’s own contract. Research preprints are identified with their study limits. Public issues and discussions are qualitative signals, not universal benchmarks or demand evidence.

1. OpenAI — Structured model outputs ↗ — current API documentation accessed September 21, 2026; JSON mode versus schema adherence, refusals, incomplete responses, and supported-schema limits.
2. OpenAI — Function calling ↗ — current API documentation accessed September 21, 2026; tool and function boundary versus structured response formats.
3. Anthropic — Structured outputs ↗ — current platform documentation accessed September 21, 2026; JSON outputs, strict tool use, grammar compilation, limitations, refusals, and caching.
4. Anthropic — How tool use works ↗ — current platform documentation accessed September 21, 2026; structured tool requests, execution loop, stop reasons, and when tools are unnecessary.
5. Google AI for Developers — Structured outputs ↗ — current Gemini API documentation accessed September 21, 2026; JSON Schema subset and the instruction to validate values in the application.
6. JSON Schema — Specification ↗ — specification index accessed September 21, 2026; standards vocabulary, published drafts, and dialect context.
7. The Structured Output Benchmark ↗ — arXiv, April 28, 2026; separates structured-output capability from source-modality quality and evaluates against schemas and ground truth.
8. TSCG: Deterministic Tool-Schema Compilation for Agentic LLM Deployments ↗ — arXiv, May 4, 2026; studies tool-schema representation and context cost.
9. LangChain issue #38714 ↗ — opened July 8, 2026; public signal about a raw JSON-schema validation path and retry behavior.
10. Vercel AI issue #16926 ↗ — opened July 8, 2026; public signal about intermittent provider-adapter tool-call formatting and retries.
11. Reddit r/LLMDevs — Provider-specific JSON Schema constraint testing ↗ — May 12, 2026 as displayed on the public thread; qualitative portability and adversarial-testing signal.
12. Reddit r/n8n — Structured output parser production failures ↗ — 2026 public thread; qualitative signal about semantic failures, logging, and bounded repair.