Support MOSS-VL

[SSSSuperC]

PR type

• More Models or Datasets Support

PR information

Add MOSS-VL (8B, model_type=moss_vl) as a first-class multimodal model for the Transformers backend. The model ships as remote code (48-layer LLM with vision cross-attention every 4 layers, 27-layer ViT with deep-stack features, merger, and a bare nn.Parameter separator token); this PR integrates it without copying the model implementation.

Supported

• swift infer (image/video) and swift sft: LoRA (target_modules=all-linear) and full-parameter
• Single/multi-image, single/multi-video, mixed media, text-only (native blank-image contract), multi-turn, batch size > 1
• freeze_vit / freeze_aligner / freeze_llm on the real module boundaries; LoRA target scanning skips the bare separator_token, LoRA-LLM saves it by exact name (and the loader exposes it through a BC property so the generic unfreeze path can reach it)
• Non-reentrant gradient checkpointing, language + vision tower (the reentrant mode is not supported: it double-backwards through the vision deep-stack/shared compute graph)
• Supervision via the native processor's labels_spans (binary strategies only; non-binary loss-scale raises explicitly)
• Controlled truncation: text-only truncation updates all text-axis fields; truncation that breaks the vision token/media metadata contract raises MaxLengthError
• Full/merged checkpoints reload standalone (remote code + processor + chat template saved); LoRA checkpoints load as base model + adapter and support merge/resume

Non-goals for this PR: sequence parallel (--sequence_parallel_size > 1 raises explicitly; support is planned as a follow-up PR), packing=true / padding-free, Megatron/TP/PP/CP, RLHF, vLLM/lmdeploy/SGLang, quantization, streaming video.

Generic changes, and why each is required by this model

Both are behavior-preserving for existing models and carry regression tests.

• find_all_linears skips non-Module paths when scanning LoRA targets: MOSS-VL's separator_token is a bare nn.Parameter (not an nn.Module); the previous scan crashed on it.
• lora_llm matches aligner parameters whose names terminate at the prefix (f'.{name}'.endswith(f'.{prefix}')): required to save the trainable bare separator_token in LoRA-LLM checkpoints — the old f'.{prefix}.' in name match never hits leaf parameters, silently dropping them from vit.safetensors. Strictly additive: parameters under module-type prefixes always extend past the prefix, so existing models match exactly as before.

Tests: 16 lightweight tests (no 8B weight download): registration, template/processor golden alignment, collator padding, mixed-media batches, truncation contract, LoRA scanning, lora_llm name matching, SP rejection. pre-commit run --all-files passes.

Requires: transformers>=4.57.1,<5, torchcodec (0.7.x for torch 2.8), joblib.

Experiment results

• Field-level golden alignment (text/tokens/labels/media order/grid/cross-attention mask) with the native processor on text/image/video/mixed/multi-turn samples
• H200 GPU smoke: LoRA and full SFT converge with finite loss/gradients; checkpoint save/resume/merge verified (resume loss bit-exact vs continuous run)
• --freeze_aligner false with LoRA trains cleanly; lora_llm trains with ViT/aligner unfrozen, saves separator_token into vit.safetensors, and a zero-LR resume reproduces it bit-exactly
• Full SFT ZeRO-3 on 8×H200: finite loss/grad norm
• Examples use public IDs (OpenMOSS-Team/MOSS-VL-Instruct-0708, lmms-lab/VideoChatGPT)

[SSSSuperC]

PR #9944 更新说明

一、关于本次 push：SP 支持撤出本 PR

本次更新将 sequence parallel 支持整体撤出本 PR，改动范围严格限定为：

• swift/sequence_parallel/sequence_parallel.py、swift/sequence_parallel/ulysses.py：全部还原为上游实现。
• MossVLTemplate 增加显式护栏：--sequence_parallel_size > 1 时抛出 NotImplementedError: MOSS-VL does not support sequence parallel yet.，避免用户误入未支持路径产生静默错误

SP 支持后续将整理为独立 PR 提交。

撤回后本 PR 的范围：模型/模板接入 + 两个 MOSS-VL 必需的通用小修复（find_all_linears 跳过裸 nn.Parameter、lora_llm 精确名匹配），公共代码不含任何模型特判。

---

二、回复 tastelikefeet 的 review（swift/tuner_plugin/lora_llm.py）

review 原文：1. parameter_name == module_prefix 是否有意义呢 2. --freeze_aligner false 及 --tuner_type lora_llm 是否会运行报错需要测试下

感谢指正，两点分别回复：
1. parameter_name == module_prefix 确实没有意义，已删除。

现已统一为：

if f'.{module_prefix}.' in parameter_name or f'.{parameter_name}'.endswith(f'.{module_prefix}'):

2. 两个组合已实测（H200，MOSS-VL-0708-Instruct），其中 lora_llm 初版确实会报错，已修复：

• --tuner_type lora --freeze_aligner false：正常训练，无报错。
• --tuner_type lora_llm --freeze_aligner false：初版在 prepare_model 的解冻循环中崩溃，deep_getattr(model, 'model.separator_token') 返回 None。根因：MossVLForConditionalGeneration 为 visual/language_model 提供了 BC property（与 Qwen3VL 相同模式），但唯独没有 separator_token 的，属性路径解析中断。修复方式是在 MossVLLoader 中为模型类补上该 property（模型私有改动，未动公共代码）。
• 修复后验证：训练 2 步 → vit.safetensors 包含 base_model.model.model.separator_token 且值为训练后的结果→ 以 lr=0 从 checkpoint resume 后该参数取值逐位一致，保存/加载链路完整正确。
• 另验证 lora_llm + FA3 + 视频样本训练正常。