[template] support chat_template_kwargs

docs/source/Customization/Custom-dataset.md

@@ -94,6 +94,13 @@ print(inputs['loss_scale'])
 {"messages": [..., {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"北京\"}}", "loss": false}, {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"上海\"}}"}, ...]}
 ```
 
+- "chat_template_kwargs"字段，（需ms-swift>=4.3.0），你可以通过在数据集中传入该字段**样本级别**控制template的min_pixels, max_pixels, fps等多模态参数，以及enable_thinking（推理时）等参数。以下为不同模型支持的参数：
+  - 其中"enable_thinking", "response_prefix"支持所有模型（推理时生效）；"max_pixels"参数支持所有多模态模型。
+  - Qwen系列多模态模型：min_pixels, max_pixels, fps等qwen_vl_utils/qwen_omni_utils支持的参数。
+```jsonl
+{"messages": [{"role": "user", "content": "<image>这是什么"}, {"role": "assistant", "content": "这是一只兔子", "loss": false}], "chat_template_kwargs": {"max_pixels": 1048576}}
+{"messages": [{"role": "user", "content": "who are you?"}], "chat_template_kwargs": {"enable_thinking": false}}
+```
 
 #### channel loss
 如果你要使用channel loss，你需要设置`--enable_channel_loss true`，并在数据集中增加"channel"字段。channel loss兼容packing/padding_free/loss_scale等技术。

docs/source_en/Customization/Custom-dataset.md

@@ -96,6 +96,15 @@ Note: If you set "loss"/"loss_scale" on consecutive "tool_call" messages in the
 {"messages": [..., {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"Beijing\"}}", "loss": false}, {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"Shanghai\"}}"}, ...]}
 ```
 
+- The "chat_template_kwargs" field (requires ms-swift>=4.3.0) allows you to control template multimodal parameters such as min_pixels, max_pixels, fps, as well as parameters like enable_thinking (during inference) at the **sample level** by passing this field in the dataset. The following parameters are supported by different models:
+  - Among them, "enable_thinking" and "response_prefix" are supported by all models (takes effect during inference); the "max_pixels" parameter is supported by all multimodal models.
+  - Qwen series multimodal models: parameters supported by qwen_vl_utils/qwen_omni_utils such as min_pixels, max_pixels, fps, etc.
+
+```jsonl
+{"messages": [{"role": "user", "content": "<image>What is this"}, {"role": "assistant", "content": "This is a rabbit", "loss": false}], "chat_template_kwargs": {"max_pixels": 1048576}}
+{"messages": [{"role": "user", "content": "who are you?"}], "chat_template_kwargs": {"enable_thinking": false}}
+```
+
 #### Channel Loss
 If you want to use channel loss, you need to set `--enable_channel_loss true` and add a "channel" field to your dataset. Channel loss is compatible with techniques such as packing, padding-free, and loss scaling.
 

examples/deploy/client/llm/chat/openai_client.py

@@ -6,7 +6,16 @@
 
 
 def infer(client, model: str, messages):
-    resp = client.chat.completions.create(model=model, messages=messages, max_tokens=512, temperature=0)
+    resp = client.chat.completions.create(
+        model=model,
+        messages=messages,
+        max_tokens=512,
+        temperature=0,
+        extra_body={
+            'chat_template_kwargs': {
+                'enable_thinking': False
+            },
+        })
     query = messages[0]['content']
     response = resp.choices[0].message.content
     print(f'query: {query}')
@@ -16,7 +25,16 @@ def infer(client, model: str, messages):
 
 # streaming
 def infer_stream(client, model: str, messages):
-    gen = client.chat.completions.create(model=model, messages=messages, stream=True, temperature=0)
+    gen = client.chat.completions.create(
+        model=model,
+        messages=messages,
+        stream=True,
+        temperature=0,
+        extra_body={
+            'chat_template_kwargs': {
+                'enable_thinking': False
+            },
+        })
     print(f'messages: {messages}\nresponse: ', end='')
     for chunk in gen:
         if chunk is None:
@@ -43,5 +61,5 @@ def run_client(host: str = '127.0.0.1', port: int = 8000):
 
 if __name__ == '__main__':
     from swift import DeployArguments, run_deploy
-    with run_deploy(DeployArguments(model='Qwen/Qwen2.5-1.5B-Instruct', verbose=False, log_interval=-1)) as port:
+    with run_deploy(DeployArguments(model='Qwen/Qwen3.5-4B', verbose=False, log_interval=-1)) as port:
         run_client(port=port)

examples/deploy/client/llm/chat/swift_client.py

@@ -46,15 +46,14 @@ def run_client(host: str = '127.0.0.1', port: int = 8000):
     infer_batch(engine, infer_requests)
 
     messages = [{'role': 'user', 'content': 'who are you?'}]
-    infer_stream(engine, InferRequest(messages=messages))
+    infer_stream(engine, InferRequest(messages=messages, chat_template_kwargs={'enable_thinking': False}))
 
 
 if __name__ == '__main__':
     from swift import (DeployArguments, InferClient, InferEngine, InferRequest, InferStats, RequestConfig, load_dataset,
                        run_deploy)
 
     # NOTE: In a real deployment scenario, please comment out the context of run_deploy.
-    with run_deploy(
-            DeployArguments(model='Qwen/Qwen2.5-1.5B-Instruct', verbose=False, log_interval=-1,
-                            infer_backend='vllm')) as port:
+    with run_deploy(DeployArguments(model='Qwen/Qwen3.5-4B', verbose=False, log_interval=-1,
+                                    infer_backend='vllm')) as port:
         run_client(port=port)

swift/dataset/preprocessor/core.py

@@ -33,6 +33,7 @@ class RowPreprocessor:
                                 'channel',
                                 'margin',
                                 'teacher_prompt',
+                                'chat_template_kwargs',
                             ]
 
     def __init__(self,
@@ -260,6 +261,7 @@ def _new_init(_self, schema=None, features=None, *args, **kwargs):
                         features[key] = messages_feature
                     features['images'] = List({'bytes': Value(dtype='binary'), 'path': Value(dtype='string')})
                     features['objects'] = Json()
+                    features['chat_template_kwargs'] = Json()
                 else:
                     messages_feature = [{
                         'role': Value(dtype='string'),

swift/infer_engine/grpo_vllm_engine.py

@@ -108,7 +108,7 @@ def _create_chat_completion_response(self, result, inputs, request_config, reque
         choices = []
         for output in result.outputs:
             output.token_ids = list(output.token_ids)
-            response = self.template.decode(output.token_ids)
+            response = self.template.decode(output.token_ids, template_inputs=inputs['template_inputs'])
             logprobs = self._get_logprobs(output.logprobs, output.token_ids, request_config.top_logprobs)
             toolcall = self._get_toolcall(response)
 

swift/infer_engine/lmdeploy_engine.py

@@ -202,7 +202,7 @@ async def _infer_stream_async(
         else:
             context = self.engine.safe_run(session_id)
 
-        infer_streamer = InferStreamer(self.template)
+        infer_streamer = InferStreamer(self.template, template_inputs=inputs['template_inputs'])
         token_idx = 0
         async with context as gen:
             if version.parse(lmdeploy.__version__) < version.parse('0.6.5'):
@@ -225,7 +225,8 @@ async def _infer_stream_async(
                 usage_info = self._get_usage_info(len(inputs['input_ids']), output.num_token)
                 toolcall = None
                 if is_finished:
-                    toolcall = self._get_toolcall(self.template.decode(output.token_ids))
+                    toolcall = self._get_toolcall(
+                        self.template.decode(output.token_ids, template_inputs=inputs['template_inputs']))
                 finish_reason = self._get_finish_reason(generation_config.max_new_tokens, output.num_token,
                                                         output.status.name == 'FINISH')
                 choices = [
@@ -260,7 +261,7 @@ async def _infer_full_async(
                 async for output in generator.async_stream_infer(session_id=session_id, **inputs, **kwargs):
                     pass
 
-        response = self.template.decode(output.token_ids)
+        response = self.template.decode(output.token_ids, template_inputs=inputs['template_inputs'])
         logprobs = self._get_logprobs(output.logprobs, output.token_ids, request_config.top_logprobs)
 
         usage_info = self._get_usage_info(len(inputs['input_ids']), output.num_token)

swift/infer_engine/protocol.py

@@ -91,6 +91,7 @@ class InferRequest:
 
     tools: Optional[List[Tool]] = None
     objects: Dict[str, Any] = field(default_factory=dict)
+    chat_template_kwargs: Dict[str, Any] = field(default_factory=dict)
 
     def __post_init__(self):
         for key in ['images', 'audios', 'videos']:
@@ -233,6 +234,7 @@ class ChatCompletionRequestMixin:
     messages: Messages
     tools: Optional[List[Tool]] = None
     tool_choice: Optional[Union[str, Dict]] = None
+    chat_template_kwargs: Dict[str, Any] = field(default_factory=dict)
 
     def __post_init__(self):
         if self.tool_choice is None:

swift/infer_engine/sglang_engine.py

@@ -185,7 +185,7 @@ def _create_chat_completion_response(self, output, inputs, return_details: bool
         assert output is not None
         meta_info = output['meta_info']
         usage_info = self._get_usage_info(meta_info['prompt_tokens'], meta_info['completion_tokens'])
-        response = self.template.decode(output['output_ids'])
+        response = self.template.decode(output['output_ids'], template_inputs=inputs['template_inputs'])
         toolcall = self._get_toolcall(response)
         token_ids = output['output_ids'] if return_details else None
         choice = ChatCompletionResponseChoice(
@@ -271,7 +271,7 @@ async def _infer_stream_async(self, inputs: Dict[str, Any], generation_config: D
         engine_inputs = {k: v for k, v in inputs.items() if k != 'template_inputs'}
         result_generator = await self.engine.async_generate(
             **engine_inputs, sampling_params=generation_config, stream=True)
-        infer_streamer = InferStreamer(self.template)
+        infer_streamer = InferStreamer(self.template, template_inputs=inputs['template_inputs'])
         async for output in result_generator:
             res = self._create_chat_completion_stream_response(output, infer_streamer)
             if res is None:
@@ -289,7 +289,7 @@ def _create_chat_completion_stream_response(self, output, infer_streamer) -> Opt
         toolcall = None
         if is_finished:
             finish_reason = finish_reason['type']
-            toolcall = self._get_toolcall(self.template.decode(output['output_ids']))
+            toolcall = self._get_toolcall(self.template.decode(output['output_ids'], **infer_streamer.decode_kwargs))
         meta_info = output['meta_info']
         usage_info = self._get_usage_info(meta_info['prompt_tokens'], meta_info['completion_tokens'])
         # TODO: logprobs

swift/infer_engine/transformers_engine.py

@@ -217,8 +217,7 @@ def _update_batched_logprobs(batched_logprobs: List[torch.Tensor], logits_stream
 
     def _infer_stream(self, inputs: Dict[str, Any], *, generation_config: GenerationConfig,
                       adapter_request: Optional[AdapterRequest], request_config: RequestConfig,
-                      **kwargs) -> Iterator[List[Optional[ChatCompletionStreamResponse]]]:
-
+                      template_inputs) -> Iterator[List[Optional[ChatCompletionStreamResponse]]]:
         if generation_config.num_beams != 1:
             error_msg = 'Streaming generation does not support beam search.'
             raise ValueError(error_msg)
@@ -248,7 +247,7 @@ def _model_generate(**kwargs):
         batch_size = inputs['attention_mask'].shape[0]
         all_is_finished = False
         is_finished = [False] * batch_size
-        infer_streamers = [InferStreamer(self.template) for _ in range(batch_size)]
+        infer_streamers = [InferStreamer(self.template, template_inputs=template_inputs[i]) for i in range(batch_size)]
         request_id_list = [f'chatcmpl-{random_uuid()}' for _ in range(batch_size)]
         token_idxs = [0] * batch_size
 
@@ -298,7 +297,8 @@ def _model_generate(**kwargs):
                 usage_info = self._get_usage_info(num_prompt_tokens, len(generate_ids))
                 toolcall = None
                 if is_finished[i]:
-                    toolcall = self._get_toolcall(self.template.decode(generate_ids))
+                    toolcall = self._get_toolcall(
+                        self.template.decode(generate_ids, template_inputs=template_inputs[i]))
                 finish_reason = self._get_finish_reason(generation_config.max_new_tokens, usage_info.completion_tokens,
                                                         is_finished[i])
 

swift/infer_engine/vllm_engine.py

@@ -494,7 +494,9 @@ async def _infer_stream_async(
     ) -> AsyncIterator[ChatCompletionStreamResponse]:
         request_id = random_uuid()
         result_generator = self._add_request(inputs, generation_config, request_id, adapter_request=adapter_request)
-        infer_streamers = [InferStreamer(self.template) for _ in range(generation_config.n)]
+        infer_streamers = [
+            InferStreamer(self.template, template_inputs=inputs['template_inputs']) for _ in range(generation_config.n)
+        ]
         token_idxs = [0 for _ in range(generation_config.n)]
         async for result in result_generator:
             res = self._create_chat_completion_stream_response(result, request_config, request_id, infer_streamers,
@@ -559,7 +561,8 @@ def _create_chat_completion_stream_response(self, result, request_config, reques
 
             toolcall = None
             if output.is_finished:
-                toolcall = self._get_toolcall(self.template.decode(output.token_ids))
+                toolcall = self._get_toolcall(
+                    self.template.decode(output.token_ids, **infer_streamers[i].decode_kwargs))
 
             choice = ChatCompletionResponseStreamChoice(
                 index=i,
@@ -593,7 +596,7 @@ def _create_chat_completion_response(
         choices = []
         for output in result.outputs:
             output.token_ids = list(output.token_ids)
-            response = self.template.decode(output.token_ids)
+            response = self.template.decode(output.token_ids, template_inputs=inputs['template_inputs'])
 
             # Extract reasoning content if reasoning_parser is enabled
             reasoning_content = None
@@ -743,7 +746,10 @@ def infer(
             if request_config.stream:
 
                 def _gen_wrapper():
-                    infer_streamers = [InferStreamer(self.template) for _ in range(generation_config.n)]
+                    infer_streamers = [
+                        InferStreamer(self.template, template_inputs=inputs['template_inputs'])
+                        for _ in range(generation_config.n)
+                    ]
                     token_idxs = [0 for _ in range(generation_config.n)]
                     while self.engine.has_unfinished_requests():
                         result = self.engine.step()