Skip to content

Releases: modelscope/ms-swift

Release list

v4.5.3

v4.5.3 Latest
Latest

Choose a tag to compare

View all tags
@hjh0119 hjh0119 released this 08 Sep 13:00
v4.5.3
faed594

New Models

New Features

  • Megatron-SWIFT Muon optimizer supports different learning rates for different parameters.
  • OPD distillation (GRPO) now supports separate vision inputs for teacher and student, enabling multimodal distillation. Reference doc: https://swift.readthedocs.io/en/latest/Instruction/Distillation.html
  • A last-checkpoint link is maintained when saving checkpoints, making resume easier.
  • dataloader_persistent_workers now defaults to True to avoid dataloader cold start on each eval.
  • Performance: vectorized padding-free log-prob restore and sequence reductions; Megatron loss all-reduce deferred to logging time; vectorized ring attention LSE extraction on NPU.
  • Added the vllm_kunlun inference backend for rollout/deploy.
  • Removed the gradio dependency.

Bug Fixes

  • Training: resume_from_checkpoint with transformers 4.x, DPO/IPO log-prob normalization, reward model margins aligned with pairs, predict_with_generate DDP evaluation, loss scaling with sequence parallelism, vocab loss & external files in multi-processing, template pickling, py3.14 compatibility, etc.
  • RL: Megatron GRPO vocab-parallel log-prob gradients, vLLM weight sync in colocate mode (LoRA IndexError, sender device alignment), colocated vLLM cleanup ordering on rollout failures, token-in/token-out consistency in agentic multi-step rollouts.
  • Megatron: router replay CP partition and expert indices, MTP shared weights forward, reference adapter loading, zero multimodal learning rates, Megatron-FSDP DTensor checkpoint save/load, etc.
  • Model fixes: Qwen3.5 (linear attention FP32 weights, multimodal packing kwargs), Qwen-Omni video fps, MiniCPM-o 4.5 video inputs, mplug_owl2_1, etc.

What's Changed

Read more

Contributors

sliedes, kiscad, and 39 other contributors
Assets 2
Loading

v4.5.2

v4.5.2

Choose a tag to compare

View all tags
@hjh0119 hjh0119 released this 16 Aug 02:44
v4.5.2
ff51287

Bug Fixes

  • Fixed: Corrected the incorrect size configuration for Qwen3.8-35B-A3B, which was mistakenly copied from the previous Qwen3.5/3.6 series during development. We apologize for the confusion and inconvenience this caused. Thanks to the community for reporting this issue.
Loading

v4.5.0

v4.5.0

Choose a tag to compare

View all tags
@hjh0119 hjh0119 released this 14 Aug 17:07
v4.5.0
074cacc

New Models

Qwen3.8

Qwen3.8, the strongest open Qwen generation so far. Same Gated DeltaNet hybrid backbone as Qwen3.5/3.6, native vision/video, hybrid thinking on by default, plus tunable reasoning effort and preserved historical thinking. (#9914)

NVIDIA Nemotron 3.5 Lightning

Nemotron 3.5 Lightning 30B-A3B: hybrid Mamba-2 + MoE + Attention, 30B total / 3B active, 1M context, MTP. (#9894)

Muse Glimmer

Muse-Glimmer-30B, Meta’s 30B dense multimodal model distilled from Muse Spark for local agents. (#9887)

Other changes

What's Changed

New Contributors

Full Changelog: v4.4.3...v4.5.0

Contributors

vincenthcui, tomzw11, and 16 other contributors
Loading

Patch release v4.4.3

Patch release v4.4.3

Choose a tag to compare

View all tags
@hjh0119 hjh0119 released this 12 Aug 03:39
v4.4.3
e128792
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

What's Changed

New Contributors

Full Changelog: v4.4.2...v4.4.3

Contributors

yszhli, zzong2006, and 21 other contributors
Loading

Patch release v4.4.2

Patch release v4.4.2

Choose a tag to compare

View all tags
@Jintao-Huang Jintao-Huang released this 21 Jul 15:47
v4.4.2
f48847d

Full Changelog: v4.4.1...v4.4.2

Loading

Patch release v4.4.1

Patch release v4.4.1

Choose a tag to compare

View all tags
@Jintao-Huang Jintao-Huang released this 13 Jul 02:52
v4.4.1
98a09c1

Full Changelog: v4.4.0...v4.4.1

Loading

v4.4.0

v4.4.0

Choose a tag to compare

View all tags
@Jintao-Huang Jintao-Huang released this 06 Jul 06:14
v4.4.0
0bf4560

中文版

新特性

  1. RL
    a. GRPO 支持 OPD RL 训练。参考文档：https://swift.readthedocs.io/zh-cn/latest/Instruction/Distillation.html
    b. GRPO/GKD 支持 Multi-teacher 训练。参考文档：https://swift.readthedocs.io/zh-cn/latest/Instruction/Distillation.html
    c. 新增多模态 vLLM 对齐测试。参考 issue：#9668
    d. GRPO/GKD Trainer 完成重构。
  2. Megatron-SWIFT
    a. 新增 model_type 支持：GLM-5.2（glm_moe_dsa 新增 indexer_type = 'shared' 支持）、minicpmv4_6。
    b. NPU Megatron-Core 版本升级至 0.16。（感谢 @addsubmuldiv 的贡献）
    c. moe_router_load_balancing_type 支持同时指定多个类型。
    d. 修复 DeepSeek-V4 FP8 相关问题。
  3. 训练
    a. 新增 DLLM 训练支持（DiffusionGemma）。训练脚本参考：https://github.com/modelscope/ms-swift/blob/main/examples/models/gemma4/diffusion_gemma.sh
    b. 新增 Qwen3-TTS 训练支持。训练脚本参考：https://github.com/modelscope/ms-swift/tree/main/examples/models/qwen3_tts
    c. Embedding 模型新增 MRL 支持，引入 mrl_dims 参数。
    d. Loss Scale 支持 ConcatLossScale，可将多种 loss scale 策略叠加使用，例如：last_round+ignore_empty_think+hermes。
    e. Qwen3.5、minicpmv4.6 支持 padding_free/packing。

新模型

  1. 纯文本模型
    a. ZhipuAI/GLM-5.2
  2. 多模态模型
    a. moonshotai/Kimi-K2.7-Code
    b. MiniMaxAI/MiniMax-M3
    c. PaddlePaddle/Unlimited-OCR
    d. google/diffusiongemma-26B-A4B-it

English Version

New Features

  1. RL
    a. GRPO now supports OPD RL training. Reference doc: https://swift.readthedocs.io/zh-cn/latest/Instruction/Distillation.html
    b. GRPO/GKD now supports Multi-teacher training. Reference doc: https://swift.readthedocs.io/zh-cn/latest/Instruction/Distillation.html
    c. Added multimodal vLLM alignment tests. Reference issue: #9668
    d. GRPO/GKD Trainer has been refactored.
  2. Megatron-SWIFT
    a. Added new model_type support: GLM-5.2 (glm_moe_dsa adds indexer_type = 'shared' support) and minicpmv4_6.
    b. NPU Megatron-Core upgraded to version 0.16. (Thanks to @addsubmuldiv for the contribution.)
    c. moe_router_load_balancing_type now supports specifying multiple types simultaneously.
    d. Fixed DeepSeek-V4 FP8 related issues.
  3. Training
    a. Added DLLM training support (DiffusionGemma). Training script reference: https://github.com/modelscope/ms-swift/blob/main/examples/models/gemma4/diffusion_gemma.sh
    b. Added Qwen3-TTS training support. Training script reference: https://github.com/modelscope/ms-swift/tree/main/examples/models/qwen3_tts
    c. Added MRL support for Embedding models, introducing the mrl_dims parameter.
    d. Loss Scale now supports ConcatLossScale, enabling multiple loss scale strategies to be stacked, e.g.: last_round+ignore_empty_think+hermes.
    e. Qwen3.5 and minicpmv4.6 now support padding_free/packing.

New Models

  1. Text-Only Models
    a. ZhipuAI/GLM-5.2
  2. Multimodal Models
    a. moonshotai/Kimi-K2.7-Code
    b. MiniMaxAI/MiniMax-M3
    c. PaddlePaddle/Unlimited-OCR
    d. google/diffusiongemma-26B-A4B-it

What's Changed

Read more

Contributors

rivergod, YingchaoX, and 13 other contributors
Loading

Patch release v4.3.2

Patch release v4.3.2

Choose a tag to compare

View all tags
@Jintao-Huang Jintao-Huang released this 29 Jun 03:20
v4.3.2
d639d2f

Full Changelog: v4.3.1...v4.3.2

Loading

Patch release v4.3.1

Patch release v4.3.1

Choose a tag to compare

View all tags
@Jintao-Huang Jintao-Huang released this 17 Jun 07:39
v4.3.1
332bb8f

Full Changelog: v4.3.0...v4.3.1

Loading

v4.3.0

v4.3.0

Choose a tag to compare

View all tags
@Jintao-Huang Jintao-Huang released this 10 Jun 06:56
v4.3.0
2d3694f

中文版

新特性

  1. Megatron-SWIFT
    a. 新增 model_type 支持：deepseek_v4, gemma4, gemma4_unified, bailing_hybrid, bailing_moe, qwen3_asr。DeepSeek-V4 微调实践：https://swift.readthedocs.io/zh-cn/latest/BestPractices/deepseek-v4.html 。Gemma4 训练脚本：https://github.com/modelscope/ms-swift/blob/main/examples/models/gemma4
    b. 新增 language_model_only 参数，支持只加载、训练和保存多模态模型的语言模型部分。
    c. 更多任务类型支持上下文并行（长文本训练）：embedding、generative_reranker、seq_cls 和 reward_model 任务。
    d. Qwen3-Next 支持 Mcore-GDN 方式运行（默认），支持序列 packing、FP8 及 CP能力。
    e. Mcore-Bridge 新增极简 Forward 示例：创建模型、执行 forward并计算损失，方便接入其他项目：https://github.com/modelscope/mcore-bridge?tab=readme-ov-file#minimal-forward-example
    f. test_convert_precision 功能增强：新增对 Attention Backend（fused / flash）、上下文并行（CP）及 padding_free 的支持。
    g. generative_reranker 任务训练 lm_head 部分显存占用优化，只提取 positive / negative token 位置的 logits 而不是完整的 logits。
    h. 训练设置 --merge_lora true 时，会将 lora 增量权重和 merged 权重都进行存储。
    i. 新增 megatron_extra_kwargs 参数，支持将额外参数直接透传至 Megatron，提升配置灵活性。
    j. 支持通过 --attention_backend flash_2 / flash_3 / flash_4 手动指定 Attention 后端。
    k. 新增 Megatron FP4 训练参数：fp4_format、fp4_recipe 和 fp4_param_gather。
    l. 新增 FP8 与 LoRA 结合使用的训练示例脚本：https://github.com/modelscope/ms-swift/blob/main/examples/megatron/fp8/lora.sh
    m. embedding 与 generative_reranker 任务支持非 padding_free 方式训练。
    n. batch_p2p_comm 参数支持。如遇流水线并行（PP）训练卡住的情况，可将其设置为 False。
    o. mlp_padding_free 参数兼容上下文并行。
  2. RL
    a. 新增基于 Megatron + Ray 的 GRPO 和 GKD 训练支持，适用于超大规模分布式强化学习场景，参考文档 https://swift.readthedocs.io/zh-cn/latest/Instruction/Ray.html
    b. Megatron GRPO 支持多轮对话场景下的训练。
    c. Megatron GRPO 新增 REAL 算法支持。（感谢@li2zhi 贡献）
    d. GRPO 新增 FIPO 算法支持。（感谢 @li2zhi 贡献）
    e. GKD 的 teacher_server 重构为基于 swift deploy 的部署方式。
    f. 新增基于 FrozenLake 环境的多轮训练完整示例，参考文档 https://swift.readthedocs.io/zh-cn/latest/Instruction/GRPO/DeveloperGuide/gym_env.html#frozenlake
    g. rollout / deploy 支持 vllm 0.16+ 的 dense model 数据并行。
    h. gym 模块重构，参考文档 https://swift.readthedocs.io/zh-cn/latest/Instruction/GRPO/DeveloperGuide/gym_env.html
  3. 训练
    a. 新增 preserve_thinking 参数：可在推理与训练时控制是否保留历史思考内容。
    b. 新增 chat_template_kwargs：训练/推理支持数据样本粒度配置 max_pixels、enable_thinking 等参数，部署侧支持通过 OpenAI 格式传入 enable_thinking / preserve_thinking。
    c. 支持在昇腾 NPU 上使用 Zigzag Ring 序列并行策略。（感谢 @addsubmuldiv 的贡献）
    d. 音频数据支持从视频文件读取。（感谢 @Tohrusky 的贡献）
    e. 兼容python 3.13。

新模型

  1. 纯文本模型
    a. deepseek-ai/DeepSeek-V4-Flash 系列
    b. OpenBMB/MiniCPM5-1B 系列
    c. inclusionAI/Ling-2.6-1T, inclusionAI/Ring-2.6-1T 系列
  2. 多模态模型
    a. OpenBMB/MiniCPM-V-4.6（感谢 @tsjyma 的贡献）
    b. PaddlePaddle/PaddleOCR-VL-1.6
    c. google/gemma-4-12B-it 系列
    d. moonshotai/Kimi-K2.5, moonshotai/Kimi-K2.6 支持多模态

English Version

New Features

  1. Megatron-SWIFT
    a. New model_type support: deepseek_v4, gemma4, gemma4_unified, bailing_hybrid, bailing_moe, qwen3_asr. DeepSeek-V4 fine-tuning best practices: https://swift.readthedocs.io/en/latest/BestPractices/deepseek-v4.html. Gemma4 training scripts: https://github.com/modelscope/ms-swift/blob/main/examples/models/gemma4
    b. Added language_model_only parameter to support loading, training, and saving only the language model component of multimodal models.
    c. Context Parallelism (CP) extended to more task types (long-sequence training): embedding, generative_reranker, seq_cls, and reward_model.
    d. Qwen3-Next now defaults to Mcore-GDN, enabling sequence packing, FP8 training, and Context Parallelism.
    e. Mcore-Bridge adds a minimal Forward example covering model creation, forward execution, and loss computation, making it straightforward to integrate into other projects: https://github.com/modelscope/mcore-bridge?tab=readme-ov-file#minimal-forward-example
    f. test_convert_precision enhancements: added support for Attention Backend (fused / flash), Context Parallelism (CP), and padding_free mode.
    g. Memory optimization for lm_head in generative_reranker training: only the logits at positive / negative token positions are extracted instead of materializing the full logits matrix, significantly reducing peak memory usage in large-vocabulary scenarios.
    h. When --merge_lora true is set, both the LoRA delta weights and the merged weights are now saved simultaneously.
    i. Added megatron_extra_kwargs parameter to allow arbitrary extra arguments to be passed directly to Megatron, improving configuration flexibility.
    j. Support for manually specifying the Attention backend via --attention_backend flash_2 / flash_3 / flash_4.
    k. Added Megatron FP4 training parameters: fp4_format, fp4_recipe, and fp4_param_gather.
    l. Added a training example script for combined FP8 + LoRA usage: https://github.com/modelscope/ms-swift/blob/main/examples/megatron/fp8/lora.sh
    m. embedding and generative_reranker tasks now support training without padding_free mode.
    n. Added batch_p2p_comm parameter. If pipeline parallelism (PP) training hangs, setting this to False can resolve the communication blocking issue.
    o. mlp_padding_free is now compatible with Context Parallelism (CP).
  2. RL
    a. Added GRPO and GKD training support based on Megatron + Ray, targeting large-scale distributed reinforcement learning scenarios. Refer to: https://swift.readthedocs.io/en/latest/Instruction/Ray.html
    b. Megatron GRPO now supports multi-turn dialogue training.
    c. Megatron GRPO adds support for the REAL algorithm. (Thanks to @li2zhi for the contribution.)
    d. GRPO adds support for the FIPO algorithm. (Thanks to @li2zhi for the contribution.)
    e. GKD teacher_server has been refactored to use swift deploy as the serving backend.
    f. Added a complete multi-turn training example based on the FrozenLake environment. Refer to: https://swift.readthedocs.io/en/latest/Instruction/GRPO/DeveloperGuide/gym_env.html#frozenlake
    g. rollout / deploy now supports data parallelism for dense models with vLLM 0.16+.
    h. gym module refactored for a cleaner interface and improved extensibility. Refer to: https://swift.readthedocs.io/en/latest/Instruction/GRPO/DeveloperGuide/gym_env.html
  3. Training
    a. Added preserve_thinking parameter to control whether historical chain-of-thought content is retained during inference and training.
    b. Added chat_template_kwargs support: training and inference now support sample-level configuration of parameters such as max_pixels and enable_thinking; on the serving side, enable_thinking / preserve_thinking can be passed via the OpenAI-compatible API format.
    c. Added support for the Zigzag Ring sequence parallelism strategy on Ascend NPU. (Thanks to @addsubmuldiv for the contribution.)
    d. Audio data can now be read directly from video files. (Thanks to @Tohrusky for the contribution.)
    e. Python 3.13 compatibility.

New Models

  1. Language Models
    a. deepseek-ai/DeepSeek-V4-Flash series
    b. OpenBMB/MiniCPM5-1B series
    c. inclusionAI/Ling-2.6-1T, inclusionAI/Ring-2.6-1T series
  2. Multimodal Models
    a. OpenBMB/MiniCPM-V-4.6 (Thanks to @tsjyma for the contribution.)
    b. PaddlePaddle/PaddleOCR-VL-1.6
    c. google/gemma-4-12B-it series
    d. moonshotai/Kimi-K2.5, moonshotai/Kimi-K2.6 with multimodal support

What's Changed

Read more

Contributors

addsubmuldiv, randydl, and 13 other contributors
Loading