[weiqingy]

Linked issue: #912

Builds on #1128, now merged. Rebased onto it, so the diff here is this change alone.

Purpose of change

Nothing changes for a caller yet: no production code asks what this adds. It is the second step of wiring an agent output_schema through to the providers that can enforce it, and it removes a drift hazard the first step exposed.

Whether a native schema reaches a provider depends on the schema form, the model, and per-provider conditions (Azure's api version floor, Anthropic's caller supplied output config, Gemini's bound tools). Each connection decided this inside its own request path, where a caller cannot see it. A separate answer that restated those conditions would drift silently.

Runtime flow

Both connection bases gain one request-level query, supportsNativeStructuredOutput(schema, tools, modelParams) in Java and supports_native_structured_output(output_schema, tools, model_kwargs) in Python. It returns NativeStructuredOutputSupport:

• INFEASIBLE: no request this connection builds can carry the schema.
• FEASIBLE: it can carry it, but the effective model is not known to honor it.
• NATIVE_RECOMMENDED: it can carry it and the effective model is known to honor it.

The default is INFEASIBLE. Every connection with a native branch answers from the same private helpers its request path uses: effective-model resolution, the capability check modelSupportsNativeStructuredOutput (_model_supports_native_structured_output in Python) and the feasibility check.

StructuredOutputStrategy.resolvesToNative takes the three-state answer: PROMPT never goes native, AUTO only on NATIVE_RECOMMENDED, and NATIVE on either feasible answer while raising on INFEASIBLE.

Key decisions

Three answers, not a boolean. AUTO needs to know whether the model will honor the schema; NATIVE ignores that and must fail only when the schema cannot be carried. The schema reaches a connection only on a final call after the tool rounds, so a failure left to the request builder would surface late and be retried.

Effective model, capability and feasibility are provider-internal. They are private helpers, except the capability check on the OpenAI completions connection, which stays protected so vLLM can replace the allowlist alone.

Behavioral Semantics

Interaction decisions

Connection	INFEASIBLE when	FEASIBLE vs NATIVE_RECOMMENDED decided by
OpenAI, Bedrock, Tongyi	schema form not translatable	model allowlist
vLLM	as OpenAI	its own allowlist override
Azure	schema form, or api version below the floor	allowlist on the model backing the deployment
Anthropic	schema form, or a caller supplied output config	model allowlist
Gemini	schema form, or tools bound	model allowlist
Ollama, watsonx	schema form	never FEASIBLE: any model is treated as capable

Behavioral contracts

1. An override answers from the same logic its request path uses: NATIVE_RECOMMENDED exactly when the branch carries the schema, INFEASIBLE when no model would make it, FEASIBLE when only the model stops it.
2. It accepts a missing schema, missing tools and missing parameters without raising, and reads the parameters without consuming them.
3. An answer other than INFEASIBLE is not a promise the call succeeds. A connection may still raise once its branch has decided to apply, where the caller supplied a conflicting format, or where the schema cannot be rendered.
4. resolvesToNative: PROMPT false, AUTO true only on NATIVE_RECOMMENDED, NATIVE true unless INFEASIBLE, which raises (IllegalArgumentException / ValueError). A null answer raises (NullPointerException / TypeError).

Failure behavior

No new failure path, and one narrow change on invalid input. A branch checks feasibility before capability, so an infeasible request no longer evaluates the capability check. With a non-string model, the Python OpenAI, Anthropic and vLLM connections raised there and now send the request without a schema.

Tests

Contract	Tests
1	per connection: an agreement test asserting the exact answer against what the request path carried, and capability tests for capable, unknown, empty and missing models
2	the base-class tests in both languages
3	the conflicting-format raises, pinned by existing tests
4	the strategy tests, all nine combinations in both languages

Coverage by risk. The failure that matters is an answer that stops matching the branch. In every native connection in both languages, a mutant answering FEASIBLE where the answer is INFEASIBLE fails the suite, as does AUTO accepting FEASIBLE. In Java, dropping the default-model fallback, judging Azure's deployment name instead of its backing model, and skipping the capability check also fail.

Suite results: Java api 424, chat-models common 6, openai 126, anthropic 101, bedrock 89, gemini 70, ollama 18, watsonx 51. Python api 452 passed, 12 skipped; chat models 325 passed.

Not verified. No live provider was called.

API

New NativeStructuredOutputSupport enum and one query on each chat-model connection base, protected in Java and public in Python, plus overrides. The capability and effective-model methods already on main leave the base class and become provider-internal helpers; neither is in a release. No user-facing API changes, and no behavior change for an existing caller beyond the invalid-model case above.

Documentation

• doc-needed
• doc-not-needed
• doc-included

Was this patch authored or co-authored using generative AI tooling?

• Yes
• No

Generated-by: Claude Code 2.1.272 (Claude Opus 5), Claude Code 2.1.292 (Claude Opus 5.5)

[wenjin272]

Thanks for the thorough Java/Python implementation and tests. I left two API-design comments: one on consolidating the connection-level predicates, and one non-blocking follow-up for flat RowTypeInfo native support.

[wenjin272]

Do we need two connection-level predicates here? Although model capability and request/schema feasibility are independent dimensions internally, the framework ultimately needs one answer: whether native structured output can be applied to this request. Could canApplyNativeStructuredOutput resolve the effective model internally and incorporate capability, replacing supportsNativeStructuredOutput as a connection-level hook? Providers could still keep model-specific allowlists as private helpers. I do not see a production caller that needs to consume the two dimensions independently, so exposing both seems to add API and composition complexity without a caller requirement.

[weiqingy]

You're right that nothing consumes the two independently today, and the combining happens inside each connection rather than in the framework: canApplyNativeStructuredOutput(...) && supportsNativeStructuredOutput(...) at OpenAICompletionsConnection.java:303-304, and the same pair in the Anthropic, Bedrock, Gemini, Ollama, Azure and watsonx connections.

The caller that would read capability on its own is StructuredOutputStrategy.resolvesToNative (StructuredOutputStrategy.java:65-75), and it is not wired to anything yet. The setup parses and stores the strategy without reading it (BaseChatModelSetup.java:71-74), and it still calls the three-argument chat (:200), so no schema reaches a connection through the framework at all on this branch. The independence is a designed contract here rather than observed behavior.

I would still keep them separate, and the reason that already shows up in code is provider inheritance. VLLMChatModelConnection extends OpenAICompletionsConnection and overrides capability only (VLLMChatModelConnection.java:80-83), because the inherited allowlist rejects served models such as Qwen/Qwen2.5-7B-Instruct, while it inherits feasibility unchanged. Merge the two and super stops being usable, since the parent's answer has already applied the allowlist the subclass exists to escape. vLLM would have to restate the parent's feasibility rule locally: one line in Java, and in Python also a reach into a module-private helper in the parent module (vllm_chat_model.py:77-85).

Your point holds for the other providers though. Ollama and watsonx answer capability with an unconditional true (OllamaChatModelConnection.java:203, WatsonxChatModelConnection.java:265), so under a merged predicate their override collapses into the feasibility test and disappears.

The contract does state that the two are independent (BaseChatModelConnection.java:106-112), but not why they are separate hooks rather than one, which is a fair gap. I will state the asymmetry at the declaration: capability is advisory, so a policy is allowed to override it, while feasibility bounds what the connection can encode at all.

If you would rather have one hook, the shape that keeps both behaviors is a three-valued answer rather than a boolean, so that "cannot encode this" and "this model probably will not honor it" stay distinguishable at a single call site. That is more API surface than the split rather than less, which is why I have not taken it, but it is a real option if the single call site is worth it.

[weiqingy]

Pushed as 7f6417d7. Both declarations now state the asymmetry in each language: capability is advisory and a configured policy may overrule it in either direction, while feasibility is binding, since a request whose schema the connection cannot encode has no native form to send.

Writing it turned up a related error in four places, all claiming that policy and capability are combined at request-build time. That is wrong on both halves: policy is parsed and stored but combined nowhere yet, and what a connection combines while building a request is capability with feasibility. Each site now names resolvesToNative as where policy meets capability, which is a declared method rather than a call path, so it stays accurate while nothing calls it.

One of the four is a pydantic field description rather than a comment, so that one change is a runtime string. Nothing reads it today.

[wenjin272]

Following up on my earlier concern: I agree that effective-model resolution, model capability, and request/schema feasibility are all necessary concepts. My concern is not about removing those checks, but about exposing all three as independent hooks on BaseChatModelConnection. That makes the internal steps of one structured-output decision part of every connection’s extension contract.

Comparing this with LangChain and AgentScope Java suggests a simpler boundary. LangChain uses a request-shaped capability check only for AutoStrategy; an explicit ProviderStrategy bypasses auto-detection and lets native schema binding or request construction fail when it cannot be applied. AgentScope keeps the check coarse at the model level (supportsNativeStructuredOutput and its WithTools variant), with runtime fallback when native execution fails.

Could Flink expose only one request-shaped method on the base connection?

• AUTO calls supportsNativeStructuredOutput(schema, tools, modelParams) to choose native or prompt.
• NATIVE directly enters the native request path.
• PROMPT uses the existing prompt path.

The native request builder can still fail fast on deterministic incompatibilities—such as an unencodable schema, an unsupported API version, or conflicting request configuration—before sending anything to the provider. Effective-model resolution and the separate capability/feasibility checks could then remain provider-internal helpers, rather than three hooks on the common base class.

[weiqingy]

Yes, one method on the base connection makes sense to me. effectiveModelFor, capability and feasibility can become provider-internal helpers. vLLM can still override capability alone through a protected helper on OpenAICompletionsConnection, so my inheritance concern goes away. None of this has shipped, so renaming is free.

One question on NATIVE though. In the follow-up wiring, the schema is only sent on a final call after the tool rounds. If NATIVE skips the check and lets the builder fail, the failure shows up after every tool has already run. It would also go through the generic retry catch (ChatModelInvoker.java:211), so a failure that can never succeed gets retried. LangChain doesn't hit this because it binds the schema on every call, so it fails on the first one.

What do you think about having the method return three answers instead of a boolean? Something like nativeStructuredOutputSupport(schema, tools, modelParams) returning INFEASIBLE, FEASIBLE or NATIVE_RECOMMENDED. AUTO goes native only on NATIVE_RECOMMENDED, NATIVE fails up front on INFEASIBLE, and PROMPT stays as is. It also means the builders keep skipping a schema they can't apply, instead of all 13 of them starting to throw. It needs a new name either way, since Python can't overload supports_native_structured_output.

If that shape works for you, I'll make the change on this PR in place of the two predicates.

[wenjin272]

That makes sense. I missed the timing difference from LangChain: because Flink only sends the schema on the finalization call, deferring deterministic validation to the request builder would discover the failure after the tool rounds and would also send it through the generic retry path.

The three-state result sounds like a good compromise. It preserves early failure for NATIVE + INFEASIBLE, while still reducing the base-class surface to one request-level method and keeping effective-model resolution, capability, and feasibility as provider-internal details.

For naming, I would prefer to keep the unified base method as:

supportsNativeStructuredOutput(schema, tools, modelParams)

and rename the internal model-capability helper to:

modelSupportsNativeStructuredOutput(effectiveModel)

with _model_supports_native_structured_output in Python. Since none of this has shipped, we do not need to retain the old capability-helper name, and this also avoids the Python overloading conflict.

[weiqingy]

Done in b860b4cf (Java) and 6af3e898 (Python). The base connection now has only supportsNativeStructuredOutput(schema, tools, modelParams), returning NativeStructuredOutputSupport (INFEASIBLE / FEASIBLE / NATIVE_RECOMMENDED). The capability helper uses your names, and effective-model resolution and feasibility are private to each connection. vLLM still overrides the capability helper alone. resolvesToNative takes the three-state answer and raises on NATIVE with INFEASIBLE. I updated the PR description to match.

[wenjin272]

Non-blocking follow-up: RowTypeInfo does not necessarily have to remain on the prompt fallback permanently. Since OutputSchema already restricts it to basic field types, we could render a flat RowTypeInfo directly as JSON Schema in Java and Python, while leaving nested rows unsupported initially. This would allow it to use provider-native structured output without generating runtime POJO/Pydantic classes, and the result could still be parsed into a Flink Row.

[weiqingy]

Agreed that a flat RowTypeInfo is renderable without a runtime POJO or Pydantic class, since OutputSchema already restricts the fields to basic types, and the response still parses back into a Row. Nested rows are the part that would need a decision, so leaving those unsupported initially sounds right.

Worth doing as its own change rather than widening this one.

[purushah]

Thanks for the careful work here. I read both sides, ran the suites, and checked each override against its branch. It all lines up, and I couldn't find a valid request that behaves differently than before. I left a few comments on the code. None of them block this one.

What I verified

• Java on JDK 17: api 428, openai 131, anthropic 103, bedrock 90, gemini 71, ollama 18, watsonx 51. All green, matching the description.
• Python: 322 passed, 1 skipped across the seven changed test files. Pinned ruff is clean.
• Every override reads without consuming, is null-safe, and matches its branch's conditions. The only load-bearing raw-parameters pass is Java Anthropic, and it is wired correctly.

On the open thread

On the two-predicates question: I think the asymmetry argument is right, and vLLM only overriding capability is a real reason not to merge them. If a combining method comes later it has to take the strategy into account too, something like feasible && strategy.resolvesToNative(capable), otherwise NATIVE still can't override capability. That keeps the distinction you two already worked out in the thread. Happy either way, that's for @wenjin272 to decide.

---

Looks good to me overall. Everything below is non-blocking for this extraction.

[purushah]

Could we retain one TODO for #912 here noting that strategy resolution is not wired into production yet? When it is wired, the native branches need to honor the resolved policy without independently vetoing NATIVE through another capability check. Non-blocking for this extraction, it's unfinished #912 integration rather than something this PR changed.

Suggested change

	* #resolvesToNative(boolean)} combines the two.
	* #resolvesToNative(boolean)} combines the two.
	*
	* TODO(#912): strategy resolution is not wired into production yet. Once it is, the
	* native branches must honor the resolved policy rather than vetoing NATIVE through
	* their own capability check.

[purushah]

Ollama's capability check is always true, so this test can't tell if someone folds capability into the override later, both sides move together and it still passes. The Python test gets around that with an _IncapableConnection subclass that returns false for capability and checks the hook still says yes for a Pydantic model. Worth doing the same here with a POJO.

[purushah]

Same as Ollama, capability is always true here so there's no way to test that it isn't folded into the hook. These two are the ones the "dedicated test per connection" line in the description doesn't cover yet.

[purushah]

Small thing on the description. Tongyi and Azure use plain set membership for capability, so a non-string model like 123 was already returning False quietly before this change, only unhashable values raised. OpenAI did raise, at its containment check, and Anthropic at .startswith. Might be worth softening "four connections raised" and adding one test for the new fallback so it's pinned. Not a real risk in practice since the setup declares model: str.

[weiqingy]

Thanks for the review, and for running both suites. All six are addressed in dc8bfb03, along with a description fix.

One correction worth surfacing: on the non-string model it's three connections, not four. vLLM raises too. It inherits the query from OpenAI, so it didn't show up in the first sweep.

The one I left alone is the raw_kwargs copy. It keeps all six connections reading the same way, though it's a no-op on that path today. Does that trade seem worth it to you?

[purushah]

This is now the same function in anthropic, ollama and openai, and azure, tongyi and watsonx have their own versions too. If it lived next to render_output_schema in types.py, the schema-form check would have one home when the flat RowTypeInfo follow-up comes. Optional, fine as a follow-up.

[purushah]

Nothing gets popped before the hook on this path, so this copy is the same map as kwargs. The other connections do strip keys before an overridable hook, so their snapshots make sense as a contract for subclasses. Here it could just pass kwargs. Tiny, non-blocking.

[wenjin272]

Thanks for the update! This addresses my concern about the base API surface: a single request-level supportsNativeStructuredOutput(...) method, with model resolution and capability/feasibility checks kept as provider-internal helpers, looks good to me. The three-state result also preserves the distinction needed for AUTO and forced NATIVE.

No further concerns from my side on this API design. We can handle RowTypeInfo support separately. Please also resolve the current merge conflicts before merging.

[weiqingy]

Thanks for the review, @wenjin272 !