Skip to content

[LoRA] apply .alpha to Qwen-Image lora_A/lora_B LoRAs - #15016

Open
dcaayushd wants to merge 1 commit into
huggingface:mainfrom
dcaayushd:fix-qwen-lora-alpha
Open

dcaayushd wants to merge 1 commit into
huggingface:mainfrom
dcaayushd:fix-qwen-lora-alpha

Conversation

@dcaayushd

Copy link
Copy Markdown

What does this PR do?

Fixes #14938

Qwen-Image LoRAs using lora_A/lora_B weights could lose their per-module .alpha values during conversion. When alpha differed from the LoRA rank, the adapter was loaded at the wrong strength鈥攆or example, a rank-32 LoRA with alpha 83 loaded at approximately 1/2.6 of its intended strength.

This change:

  • Applies .alpha / rank during lora_A/lora_B conversion.
  • Raises a clear ValueError when a matching lora_A or lora_B weight is missing.
  • Removes leftover .alpha metadata.
  • Preserves tensor dtype and device without casting.
  • Keeps the existing lora_down/lora_up conversion behavior unchanged.

Added tests cover alpha values of None, 2, 4, and 8, unmatched A/B weights, stray alpha keys, and bfloat16 dtype preservation.

Real-checkpoint verification with e-n-v-y/Qwen-Image-2.1-Fix-v2.0 showed:

  • 30 .alpha keys in the raw checkpoint.
  • 0 .alpha keys after conversion.
  • The converted weights match (alpha / rank) * (lora_B @ lora_A) with maximum absolute differences between 5.6e-09 and 3.1e-05 across three sampled modules.

The existing lora_down/lora_up conversion output was identical to the upstream output for a synthetic Qwen-Image-Lightning checkpoint.

Before submitting

  • Did you use an AI agent (Claude Code, Codex, Cursor, etc.) to help with this PR? If so:
    • Did you read the Coding with AI agents guide?
    • Did you run the self-review skill? The skill was not installed in the environment; its checked-in rubric was applied manually instead.
    • Did you share the final self-review notes in the PR description or a comment?
  • Did you read the contributor guideline?
  • Did you read our philosophy doc? This is a focused LoRA conversion fix.
  • Was this discussed/approved via a GitHub issue or the forum?
  • Did you make sure to update the documentation with your changes? No user-facing documentation changes are needed for this converter fix.
  • Did you write any new necessary tests?
  • Are you the author (or part of the team) of the model/pipeline?

Validation

  • Focused new tests: 8 passed.
  • Qwen-Image LoRA tests: 36 passed, 7 skipped; 2 unrelated MPS group-offloading failures.
  • LoRA regression tests: 81 passed, 56 skipped; 4 unrelated MPS group-offloading failures.
  • ruff check: passed.
  • ruff format --check: passed.
  • git diff --check: passed.

Self-review notes

The checked-in self-review rubric was applied manually because the self-review CLI skill was unavailable. No blocking correctness, dead-code, copied-code, or documentation issues were found.

AI assistance was used for implementation, testing, verification, and drafting this description. Please review and edit the wording before submitting.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Qwen-Image LoRAs in lora_A/lora_B format lose their .alpha and load at the wrong strength

1 participant