Skip to content

[Question] What is MODEL1? #155

New issue
New issue
Open
Open
[Question] What is MODEL1?#155

Description

@gary-wjc
gary-wjc
opened on Jan 21, 2026

I found the term "MODEL1" is mentioned several times in this repository:

FlashMLA/csrc/sm100/decode/head64/kernel.cuh

Line 554 in 48c6dc4

// MODEL1: RoPE is the last 64 dims within the full 512 dim, which couples with the last 64 dim from the NoPE part when performing dual GEMM. i.e.

FlashMLA/csrc/sm90/decode/sparse_fp8/splitkv_mla.cuh

Line 161 in 48c6dc4

// The following fields are only valid for MODEL1

I have some questions about this novel term. @interestingLSY Could you please take a look?

  1. Are there some publicly available models belonging to the category of "MODEL1"?

  2. What are the major differences of attention mechanisms between "MODEL1" and V3.2 ?

Update: My preliminary guess is the following for the 2nd question:

  1. The topk length becomes dynamic in MODEL1 while constant(2048) in V3.2
  2. MODEL1 introduces additional KV states which are not generated from history tokens in the current sequence
  3. The number of features in KV cache without position encoding for MODEL1 is 448, which is reduced from 512 as in V3.2
  4. Position encoding is also included when recovering V cache

Activity

  1. Cipheeer commented on Jan 21, 2026

    @Cipheeer
    Cipheeer
    on Jan 21, 2026

    MODEL1 might be the engineering version of V4, because they released R1 on this day a year ago

  2. DeepSpaceLLM commented on Jan 21, 2026

    @DeepSpaceLLM
    DeepSpaceLLM
    on Jan 21, 2026

    656B (for V32) or 576B (for MODEL1) different parameters means use a new architecture? The changes do seem huge, so it should really be called v4. Looking forward to the release of the new model more and more.

  3. ayokings202 commented on Jan 21, 2026

    @ayokings202
    ayokings202
    on Jan 21, 2026

    You and I know its V4

  4. junjundaidai commented on Jan 21, 2026

    @junjundaidai
    junjundaidai
    on Jan 21, 2026

    @interestingLSY 你们自己提交上来的代码，我出于好奇问了问题，你们不解释就算了，还要放一个"踩"的表情，什么意思？以为自己很了不起吗？

    估计作者点错了？

  5. Cipheeer commented on Jan 21, 2026

    @Cipheeer
    Cipheeer
    on Jan 21, 2026

    @interestingLSY 你们自己提交上来的代码，我出于好奇问了问题，你们不解释就算了，还要放一个"踩"的表情，什么意思？以为自己很了不起吗？

    这不是大拇哥吗

  6. junjundaidai commented on Jan 21, 2026

    @junjundaidai
    junjundaidai
    on Jan 21, 2026

    @interestingLSY 你们自己提交上来的代码，我出于好奇问了问题，你们不解释就算了，还要放一个"踩"的表情，什么意思？以为自己很了不起吗？

    这不是大拇哥吗

    刚变成大拇指了，果然是点错了（狗头）

  7. SamCheng0717 commented on Jan 27, 2026

    @SamCheng0717
    SamCheng0717
    on Jan 27, 2026

    Perhaps this is the R2 model~👿

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions