[template] support chat_template_kwargs

[Jintao-Huang]

No description provided.

[gemini-code-assist]

Code Review

This pull request introduces chat_template_kwargs to the RowPreprocessor and StdTemplateInputs classes to support dataset-specific template configurations. A review comment points out that reordering existing fields in the StdTemplateInputs dataclass could break positional argument initialization and suggests maintaining the original field order.

[gemini-code-assist]

The current implementation reorders the fields in the StdTemplateInputs dataclass, which changes the signature of the generated __init__ method. This could potentially break existing code that instantiates StdTemplateInputs using positional arguments. It is safer to maintain the relative order of the existing fields.

Suggested change

	chat_template_kwargs: Dict[str, Any] = field(default_factory=dict) # from dataset
	extra_kwargs: Dict[str, Any] = field(default_factory=dict)
	mm_processor_kwargs: Dict[str, Any] = field(default_factory=dict)
	chat_template_kwargs: Dict[str, Any] = field(default_factory=dict) # from dataset
	mm_processor_kwargs: Dict[str, Any] = field(default_factory=dict)
	extra_kwargs: Dict[str, Any] = field(default_factory=dict)

[Jintao-Huang]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces the chat_template_kwargs field to allow sample-level control over multimodal parameters (e.g., max_pixels, fps) and inference settings (e.g., enable_thinking, response_prefix). The implementation includes updates to documentation, dataset preprocessing, inference engines, and the base template class. However, the review identifies that the _get_max_pixels overrides in the Qwen-specific templates ignore the inputs argument, which breaks the sample-level max_pixels functionality for those models.

[Jintao-Huang]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces sample-level control for template parameters such as max_pixels, fps, and enable_thinking through a new chat_template_kwargs field in datasets and inference requests. The changes span documentation, inference engines, and template logic to ensure these parameters are correctly propagated and applied. Feedback indicates a breaking change regarding the response_prefix property, a regression in the removal of thinking_budget logic, and an implementation error in the Qwen template that ignores the new sample-level overrides for max_pixels.

[gemini-code-assist]

Renaming _response_prefix to response_prefix as a simple attribute and removing the property of the same name is a breaking change. Any external code relying on template.response_prefix to return the computed prefix (including thinking/non-thinking prefixes) will now only get the raw value passed during initialization. Please keep the property and have it call _get_response_prefix() to maintain backward compatibility.

Suggested change

	self.response_prefix = response_prefix
	self._response_prefix = response_prefix

[gemini-code-assist]

To maintain backward compatibility after renaming the internal attribute, please re-add the response_prefix property that calls the new helper method.

    @property
    def response_prefix(self):
        return self._get_response_prefix()

    def _get_enable_thinking(self, inputs=None):
        enable_thinking = None if inputs is None else inputs.chat_template_kwargs.get('enable_thinking')
        if enable_thinking is None:
            enable_thinking = self.enable_thinking
        return enable_thinking

    def _get_response_prefix(self, inputs=None):
        response_prefix = None if inputs is None else inputs.chat_template_kwargs.get('response_prefix')
        if response_prefix is None:
            response_prefix = self._response_prefix
        if response_prefix is not None:
            return response_prefix
        elif not self.use_chat_template:
            return ''
        enable_thinking = self._get_enable_thinking(inputs)
        if enable_thinking:
            return self.template_meta.thinking_prefix
        else:
            return self.template_meta.non_thinking_prefix

[gemini-code-assist]

The removal of the thinking_budget logic from _jinja_encode might break existing datasets that provide this parameter in extra_kwargs. While chat_template_kwargs is a more structured way to pass such parameters, removing support for the previous location without a migration path or explanation is a regression.

Suggested change

	enable_thinking = self._get_enable_thinking(inputs)
	if self.template_meta.is_thinking or enable_thinking:
	if 'thinking_budget' in inputs.extra_kwargs:
	kwargs['thinking_budget'] = inputs.extra_kwargs.get('thinking_budget', 0)
	enable_thinking = self._get_enable_thinking(inputs)

[gemini-code-assist]

This override in Qwen2VLTemplate explicitly ignores the inputs parameter, which means sample-level max_pixels overrides in chat_template_kwargs will be ignored during the initial preprocessing in base.py. This contradicts the goal of the PR and the documentation added. If the intention is to let qwen_vl_utils handle resizing, base.py's generic rescaling should be avoided entirely for this template, but returning self.max_pixels here still triggers it for the default value while blocking the sample-level override.