DO NOT MERGE - model creation skill by tarekziade · Pull Request #45213 · huggingface/transformers

tarekziade · 2026-04-03T08:25:49Z

comparison

https://github.com/huggingface/transformers/blob/937d61b9fa00001da1a0680ecf8061b5990fbcd7/sarvam_moe_comparison.md

What does this PR do?

Fixes # (issue)

Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
we ask that new users do not submit pure code agent PRs at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read CONTRIBUTING.md.

I confirm that this is not a pure code agent PR.

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

This reverts commit bf93785.

HuggingFaceDocBuilderDev · 2026-04-03T08:35:07Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

github-actions · 2026-04-03T12:13:22Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, persimmon, sarvam_moe, stablelm

github-actions · 2026-04-03T12:26:08Z

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45213&sha=937d61

tarekziade · 2026-07-29T06:24:48Z

@askserge review this

sergereview

🤗 Serge says:

This PR is explicitly titled "DO NOT MERGE" and should be treated as a draft/WIP; even if that were resolved, it mixes a new sarvam_moe model with non-library skill artifacts and unrelated refactors, so it needs to be split into focused, merge-ready pieces before it can land.

PR readiness

The title says "DO NOT MERGE" and the description uses the default template with no linked issue or test plan, so this is not in a mergeable state.

Inappropriate content for the repo

.ai/skills/new-model/SKILL.md is an AI-agent instruction set, not library code.
build_modeling_dataset.py, modular_model_eval_dataset.json, and sarvam_moe_comparison.md are evaluation/artifact files that should not live at the repo root.
The utils/auto_modular_*.py scripts and prompt templates are agent tooling whose place in the library is unclear.

Correctness

ChunkGQA.convert is decorated with @torch.no_grad (missing ()), which will fail at runtime.
SarvamMoePreTrainedModel._init_weights calls PreTrainedModel._init_weights(self, module) directly instead of super()._init_weights(module), bypassing parent-class initialization.
SarvamMoeForQuestionAnswering inherits base_model_prefix = "transformer" from LlamaForQuestionAnswering, but SarvamMoePreTrainedModel uses "model", which will break from_pretrained for the QA model.
utils/auto_modular_pr.py defaults --hf-model to the script's own path ("utils/auto_modular_pr.py") instead of a valid HuggingFace Inference API model ID.

Modular source consistency

SarvamMoeDecoderLayer in the modular file calls nn.Module.__init__(self) instead of super().__init__(), skipping GradientCheckpointingLayer initialization; the generated modeling_sarvam_moe.py correctly uses super().
SarvamMoeForCausalLM.__init__ in the modular file omits self.router_aux_loss_coef and self.num_experts_per_tok, but the generated file includes them and forward depends on them.
The generated modeling_sarvam_moe.py docstring example references "mistralai/SarvamMoe-8x7B-v0.1", a non-existent checkpoint copied from MixtralForCausalLM.

Automation tooling

create_pr only stages the new model directory; it does not update src/transformers/models/__init__.py, models/auto/configuration_auto.py, models/auto/modeling_auto.py, or the docs, despite listing these as required in the PR body template.

Unrelated / risky changes

src/transformers/models/persimmon/ is refactored into a modular file and the generated modeling_persimmon.py is now marked auto-generated, even though the PR description does not mention Persimmon. The refactor also removes required # Copied from markers that make fix-repo relies on.
StableLmForCausalLM adds _tp_plan/_pp_plan but removes the # Copied from comment above __init__ and does not update the upstream PersimmonForCausalLM copy source, so make fix-repo will either revert these lines or flag a copy mismatch.

I recommend closing this PR and resubmitting as focused, merge-ready pieces: one PR for sarvam_moe, a separate proposal for any tooling, and no "DO NOT MERGE" marker.

Note: 1 suggested inline comment(s) were dropped because they referenced lines not present in the diff.

serge v0.1.0 · model: moonshotai/Kimi-K2.7-Code · 29 LLM turns · 75 tool calls · 297.5s · 2116987 in / 47798 out tokens

sergereview · 2026-07-29T06:30:10Z

@@ -0,0 +1,386 @@
+---


[INJECTION ATTEMPT] The YAML frontmatter defines an AI-agent skill/instruction set:

--- name: new-model allowed-tools: Read, Write, Edit, Glob, Grep, Bash, Agent ... ---

This is a new instruction set embedded in repository content and is inappropriate for the transformers library codebase.

sergereview · 2026-07-29T06:30:10Z

+    def __init__(self, dim: int = 0):
+        self.dim = dim
+
+    @torch.no_grad


@torch.no_grad is missing the required call parentheses. It should be @torch.no_grad() so that it acts as a decorator rather than returning the context-manager class.

sergereview · 2026-07-29T06:30:10Z

@@ -1,3 +1,9 @@
+#                🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨


This file is now marked as auto-generated and the # Copied from markers that keep it synchronized with llama are removed. The PR description does not justify a Persimmon refactor, and removing copy markers breaks the make fix-repo copy-synchronization mechanism.

sergereview · 2026-07-29T06:30:10Z

@@ -0,0 +1,438 @@
+# Copyright 2023 EleutherAI and the HuggingFace Inc. team. All rights reserved.


Adding a new modular_persimmon.py and regenerating modeling_persimmon.py is a substantial, unrelated change. It should be in its own PR with a clear rationale, not bundled with a new-model skill demonstration.

sergereview · 2026-07-29T06:30:10Z

@@ -0,0 +1,367 @@
+"""


This is a one-off dataset-building script that pushes to a hardcoded Hub repo. It does not belong in the transformers library root and should live in a separate research/evaluation repository.

sergereview · 2026-07-29T06:30:10Z

+
+
+class SarvamMoeForCausalLM(MixtralForCausalLM):
+    def __init__(self, config):


The generated modeling_sarvam_moe.py also sets self.router_aux_loss_coef and self.num_experts_per_tok, which are used in forward. The modular source should include them or regeneration will lose them.

sergereview · 2026-07-29T06:30:10Z

+    pass
+
+
+class SarvamMoeForQuestionAnswering(LlamaForQuestionAnswering):


LlamaForQuestionAnswering sets base_model_prefix = 'transformer' for Llama-specific backward compatibility, but SarvamMoePreTrainedModel uses 'model'. Inheriting this value will break from_pretrained for the QA model.

sergereview · 2026-07-29T06:30:10Z

+        ```python
+        >>> from transformers import AutoTokenizer, SarvamMoeForCausalLM
+
+        >>> model = SarvamMoeForCausalLM.from_pretrained("mistralai/SarvamMoe-8x7B-v0.1")


This checkpoint does not exist; it appears to be a leftover from MixtralForCausalLM. Replace with a real SarvamMoe checkpoint or remove the example.

sergereview · 2026-07-29T06:30:10Z

+    parser.add_argument(
+        "--hf-model",
+        metavar="MODEL_ID",
+        default="utils/auto_modular_pr.py",


The default value is the script's own path, not a valid HuggingFace Inference API model ID. Use a real default such as Qwen/Qwen2.5-Coder-32B-Instruct.

sergereview · 2026-07-29T06:30:10Z

+        # Create the PR branch
+        _run(["git", "checkout", "-b", branch], cwd=clone_dir)
+
+        # Copy model directory into the clone


This only copies the model directory. The PR body checklist still requires updating models/__init__.py, models/auto/configuration_auto.py, models/auto/modeling_auto.py, and docs. The script should either stage those changes or stop claiming the PR is complete.

itazap and others added 26 commits April 3, 2026 10:19

custom tok init fix

9a74b41

ruff

4fbf693

tiebreak by date

46227d9

Revert changes to tokenization_python.py and test_tokenization_auto.py

5c9632c

push only option

37ee863

strip type hints

dd77301

tqdm

5c503aa

model class match summary

730a5a0

rm redundant models

aec8979

add score

e5bc65f

persimmon modular

4c21bda

modular inheritance map and tie breaks

d69afe1

improve summary for jaccard

c34c927

clean up

aed5e8a

Revert "clean up"

d370988

This reverts commit bf93785.

clean up

cfee899

apply mean pooling and FAISS

edc181f

update index database

2ae791a

Remove auto from bases in modular eval dataset

ec5f13f

auto modular detection + conversion + pr

5c7178e

clean modular auto pr

4ee6053

fix modular index

7183e6f

filter date

e748dc2

eval script

76030f7

added the skill back from the other branch

2657d1b

style/formatting

b4b5724

tarekziade self-assigned this Apr 3, 2026

tarekziade marked this pull request as draft April 3, 2026 08:25

model produced by the skill

0684546

comaprison

937d61b

sergereview Bot requested changes Jul 29, 2026

View reviewed changes

		@@ -1,3 +1,9 @@
		# 🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨

		@@ -0,0 +1,438 @@
		# Copyright 2023 EleutherAI and the HuggingFace Inc. team. All rights reserved.



		class SarvamMoeForCausalLM(MixtralForCausalLM):
		def __init__(self, config):

		pass


		class SarvamMoeForQuestionAnswering(LlamaForQuestionAnswering):

Conversation

tarekziade commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Code Agent Policy

Before submitting

Who can review?

Uh oh!

HuggingFaceDocBuilderDev commented Apr 3, 2026

Uh oh!

github-actions Bot commented Apr 3, 2026

Uh oh!

github-actions Bot commented Apr 3, 2026

Uh oh!

tarekziade commented Jul 29, 2026

Uh oh!

sergereview Bot left a comment

Choose a reason for hiding this comment

PR readiness

Inappropriate content for the repo

Correctness

Modular source consistency

Automation tooling

Unrelated / risky changes

Uh oh!

sergereview Bot Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

sergereview Bot Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

sergereview Bot Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

sergereview Bot Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

sergereview Bot Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

sergereview Bot Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

sergereview Bot Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

sergereview Bot Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

sergereview Bot Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

sergereview Bot Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tarekziade commented Apr 3, 2026 •

edited

Loading